Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix wrong tracing location of fetch data #238

Merged
merged 1 commit into from
Oct 30, 2024

Conversation

rayandrew
Copy link
Contributor

@rayandrew rayandrew commented Oct 23, 2024

Hi @zhenghh04 and @hariharan-devarajan,

If we use dlp.iter inside main.py, this does not reflect the real fetch data since the data is already fetched inside DataLoader.next(). Because of this, we ended up tracing the fetching of batch data that resides in the memory instead of tracing its fetching duration from filesystem or other sources.

This PR addresses the problem where we put dlp.iter inside the dataloader, as close as possible to the dataset itself.

For dali dataloader, I haven't implemented the fix since there is no visible iteration there (DALI uses share_outputs)

Note: @hariharan-devarajan when we bump the dftracer version, we should provide the name of the dlp.iter.
I left a note to change dlp.iter to dlp.iter(..., name=self.next.__qualname__)

Copy link
Collaborator

@hariharan-devarajan hariharan-devarajan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good thanks

@zhenghh04 zhenghh04 merged commit cc5abbc into argonne-lcf:main Oct 30, 2024
6 checks passed
@rayandrew rayandrew deleted the fix/wrong-fetch-data-tracing branch November 4, 2024 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants