Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: huggingface/datasets
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 4.7.0
Choose a base ref
...
head repository: huggingface/datasets
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 4.8.0
Choose a head ref
  • 7 commits
  • 32 files changed
  • 5 contributors

Commits on Mar 9, 2026

  1. set dev version (#8059)

    @lhoestq
    lhoestq authored Mar 9, 2026
    Configuration menu
    Copy the full SHA
    6f9d24f View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2026

  1. fix reshard_data_sources (#8061)

    @lhoestq
    lhoestq authored Mar 11, 2026
    Configuration menu
    Copy the full SHA
    9bfe52d View commit details
    Browse the repository at this point in the history

  2. Improve error message for invalid data_files pattern format (#8060) 

    * Improve error message for invalid data_files pattern format

* make style

---------

Co-authored-by: Quentin Lhoest <[email protected]>
    @kushalkkb @lhoestq
    kushalkkb and lhoestq authored Mar 11, 2026
    Configuration menu
    Copy the full SHA
    36b79a6 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2026

  1. fix null filling in missing jsonl columns (#8069) 

    * fix null filling in missing jsonl columns

* add tests
    @lhoestq
    lhoestq authored Mar 16, 2026
    Configuration menu
    Copy the full SHA
    c0f4665 View commit details
    Browse the repository at this point in the history

  2. Read (and write) from HF Storage Buckets: load raw data, process and … 

    …save to Dataset Repos (#8064)

* Dataset.push_to_hub

* DatasetDict.push_to_hub

* fix revisions

* IterableDataset.push_to_hub and IterableDatasetDict.push_to_hub

* load_dataset

* minor improvements

* add tests

* bug fixes

* fix push_to_hub() withn num_proc on macos

* docs

* bump dill and multiprocess for py 3.14

* minor docs

* fix tests

* last fixes
    @lhoestq
    lhoestq authored Mar 16, 2026
    Configuration menu
    Copy the full SHA
    82ca4d3 View commit details
    Browse the repository at this point in the history

  3. Datasets streaming iterable packaged fixes (#8068) 

    * Improved the dataset handling and extraction features

- improved extract logic in StreamingDownloadManager to handle prefixed archive paths with glob patterns.
- added support for iterating over files in prefixed archives in FilesIterable.
- updated IterableDataset to support max_shard_size and num_shards parameters for pushing to hub, ensuring they are mutually exclusive.
- enhanced VideoFolder and WebDataset to handle video files and metadata correctly.
- added tests for new features, including handling of custom file names in metadata and video datasets.
- refactored error handling and logging for better clarity and user feedback.

* update Git actions  trufflehog workflow

* a few fixes

* style

* fix last test ?

* style

* minor

* revert pattern support in _extract and _iter_from_urlpaths

---------

Co-authored-by: Michael-Rusu <[email protected]>
Co-authored-by: Quentin Lhoest <[email protected]>
Co-authored-by: Quentin Lhoest <[email protected]>
    @Michael-RDev @lhoestq
    4 people authored Mar 16, 2026
    Configuration menu
    Copy the full SHA
    ebf5cb8 View commit details
    Browse the repository at this point in the history

  4. Release: 4.8.0 (#8070) 

    release: 4.8.0
    @lhoestq
    lhoestq authored Mar 16, 2026
    Configuration menu
    Copy the full SHA
    c988a5d View commit details
    Browse the repository at this point in the history
Loading