Skip to content

Handle local paths in Hugging Face file probe#1740

Merged
jmartin-tech merged 2 commits into
NVIDIA:mainfrom
musaabhasan:fix/hf-fileformats-local-paths
May 8, 2026
Merged

Handle local paths in Hugging Face file probe#1740
jmartin-tech merged 2 commits into
NVIDIA:mainfrom
musaabhasan:fix/hf-fileformats-local-paths

Conversation

@musaabhasan

@musaabhasan musaabhasan commented May 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Detect when a Hugging Face generator name points to a local filesystem path and gather files from that path directly instead of calling Hugging Face Hub.
  • Skip empty local directories cleanly so the fileformats probe does not create an empty Attempt.outputs update.
  • Catch HFValidationError from ambiguous non-Hub names and skip the probe instead of letting the validation error escape.
  • Catch OfflineModeIsEnabled when Hugging Face Hub offline mode is enabled and skip the probe with a warning.
  • Add regression coverage proving local paths do not call huggingface_hub.list_repo_files() and offline Hub mode is handled cleanly.

Fixes #1034.

Testing

  • .venv\Scripts\python.exe -m pytest tests\probes\test_probes_fileformats.py -k "hf_files_load or local_directory or empty_local_directory or offline_mode" -q - 4 passed, 1 deselected
  • .venv\Scripts\python.exe -m compileall garak\probes\fileformats.py tests\probes\test_probes_fileformats.py
  • git diff --check - no whitespace errors; local Git emitted Windows autocrlf warnings

Full tests/probes/test_probes_fileformats.py was not completed in this local venv because the pre-existing test_hf_files_hf_repo imports garak.generators.huggingface, which requires torch and this lightweight test environment does not include it.

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

@musaabhasan

Copy link
Copy Markdown
Contributor Author

I have read the DCO Document and I hereby sign the DCO

@musaabhasan

Copy link
Copy Markdown
Contributor Author

recheck

github-actions Bot added a commit that referenced this pull request May 4, 2026

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works for the targeted issue, sorry to expand the scope, testing shows an additional edge case that could use a guard as well.

Please be sure to validate suggestions and consider adding an additional test for the additional edge case.

The behavior when the no files are found leaves something to be desired, the report simply omits the probe silently since no attempts generated. That may be something the project should address separately however.

Comment thread garak/probes/fileformats.py
@jmartin-tech jmartin-tech requested a review from leondz May 6, 2026 20:15
@musaabhasan

Copy link
Copy Markdown
Contributor Author

Thanks for the catch. I added a guard for OfflineModeIsEnabled before the HFValidationError fallback and included a regression test that simulates Hugging Face Hub offline mode.

Validation run locally:

  • .venv\Scripts\python.exe -m pytest tests\probes\test_probes_fileformats.py -k "hf_files_load or local_directory or empty_local_directory or offline_mode" -q - 4 passed, 1 deselected
  • .venv\Scripts\python.exe -m compileall garak\probes\fileformats.py tests\probes\test_probes_fileformats.py
  • git diff --check - no whitespace errors; Windows reported the existing autocrlf normalization warning

@jmartin-tech jmartin-tech merged commit a8f319c into NVIDIA:main May 8, 2026
17 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators May 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

probes: fileformats probe fails when targeting a local modal path

2 participants