Skip to content

Fix missing wait_on_cuda_event_record_corr_id in Event Synchronize activities#1318

Open
jiannanWang wants to merge 8 commits intopytorch:mainfrom
jiannanWang:missingcorrid
Open

Fix missing wait_on_cuda_event_record_corr_id in Event Synchronize activities#1318
jiannanWang wants to merge 8 commits intopytorch:mainfrom
jiannanWang:missingcorrid

Conversation

@jiannanWang
Copy link
Contributor

CUPTI provides no ordering guarantees for activity records in its buffers. SYNCHRONIZATION records (Event Synchronize, Stream Wait Event) can appear before the CUDA_EVENT record that populates the waitEventMap() with the source stream and correlation ID.

The existing code performed the waitEventMap() lookup eagerly at processing time. Stream Wait Event was already deferred (for stream-filtering purposes), so it happened to work — but Event Synchronize was logged immediately. When its CUDA_EVENT record hadn't been seen yet, wait_on_cuda_event_record_corr_id and wait_on_stream were left as -1.

Fix:

  • Move the waitEventMap() lookup inside the deferred lambda so it runs after all CUDA_EVENT records have been processed.
  • Defer both Event Synchronize and Stream Wait Event (change isWaitEventSync → isEventSync for the defer condition).
  • Add an isWaitEvent flag to DeferredLogEntry so that only Stream Wait Event entries are filtered out on unseen streams, while Event Synchronize entries are always logged.

Test

SyncEventCorrIdOutOfOrder — places SYNCHRONIZATION records before their CUDA_EVENT record in the mock buffer and verifies that wait_on_cuda_event_record_corr_id and wait_on_stream are correctly populated for both sync types.

@meta-codesync
Copy link

meta-codesync bot commented Mar 20, 2026

@jiannanWang has imported this pull request. If you are a Meta employee, you can view this in D97553834.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant