Adds an experimental one-ahead prefetching utility#1525
Open
peterdsharpe wants to merge 3 commits intoNVIDIA:mainfrom
Open
Adds an experimental one-ahead prefetching utility#1525peterdsharpe wants to merge 3 commits intoNVIDIA:mainfrom
peterdsharpe wants to merge 3 commits intoNVIDIA:mainfrom
Conversation
…ssingDataset - Introduced a new `prefetch_map` function to optimize data loading by overlapping CPU-bound preparation with GPU-bound processing. - Updated `CachedPreprocessingDataset` to raise a ValueError if `sample_paths` is empty, ensuring robust error handling. - Added `prefetch_map` to the module's exports for accessibility.
Contributor
Greptile SummaryThis PR adds a
Important Files Changed
Last reviewed commit: "Add prefetch_map uti..." |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+51
to
+63
| with ThreadPoolExecutor(max_workers=1) as pool: | ||
| it = iter(iterable) | ||
| try: | ||
| future = pool.submit(fn, next(it)) | ||
| except StopIteration: | ||
| return | ||
|
|
||
| for item in it: | ||
| next_future = pool.submit(fn, item) | ||
| yield future.result() | ||
| future = next_future | ||
|
|
||
| yield future.result() |
Contributor
There was a problem hiding this comment.
When a consumer does not fully exhaust the iterator (e.g., break, or an exception in the consumer), Python calls generator.close(), which throws GeneratorExit at the suspended yield on line 60. This triggers ThreadPoolExecutor.__exit__, which calls shutdown(wait=True) — blocking until the in-flight next_future completes.
Since fn is documented as a potentially expensive CPU-bound operation (subsampling, geometry precomputation, host-to-device transfer), this can cause a silent and surprising stall at loop teardown time. For example:
for item in prefetch_map(huge_dataloader, expensive_fn): # fn takes ~5 s per item
if error_condition:
break # silently blocks for up to ~5 s during cleanupIt would be worth adding a note to the docstring warning about this behavior, e.g.:
Notes
-----
If the iterator is not fully consumed (e.g., due to an early ``break`` or
an exception in the caller), cleanup will block until the background
thread's in-flight task completes.
…cumentation - Updated the `prefetch_map` function to ensure the background thread is properly shut down by using a `finally` block for `pool.shutdown()`, allowing for non-blocking cancellation of in-flight tasks. - Added notes to the docstring to clarify behavior when the iterator is not fully consumed, improving documentation for better user understanding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
prefetch_mapfunction to optimize data loading by overlapping CPU-bound preparation with GPU-bound processing.CachedPreprocessingDatasetto raise a ValueError ifsample_pathsis empty, ensuring robust error handling.prefetch_mapto the module's exports for accessibility.PhysicsNeMo Pull Request
Description
Used downstream in GLOBE training on DrivAerML (PR to come; this is just setting the stage in nice chunks).
Checklist
Dependencies
Review Process
All PRs are reviewed by the PhysicsNeMo team before merging.
Depending on which files are changed, GitHub may automatically assign a maintainer for review.
We are also testing AI-based code review tools (e.g., Greptile), which may add automated comments with a confidence score.
This score reflects the AI’s assessment of merge readiness and is not a qualitative judgment of your work, nor is
it an indication that the PR will be accepted / rejected.
AI-generated feedback should be reviewed critically for usefulness.
You are not required to respond to every AI comment, but they are intended to help both authors and reviewers.
Please react to Greptile comments with 👍 or 👎 to provide feedback on their accuracy.