Multimodal Universe: Enabling Large-Scale Machine Learning with 100TBs of Astronomical Scientific Data

The Multimodal Universe dataset is a large scale collection of multimodal astronomical data, including images, spectra, and light curves, which aims to enable research into foundation models for astrophysics and beyond.

All datasets can be previewed directly from our HuggingFace hub and accessed via load_dataset('MultimodalUniverse/dataset_name')! Preview datasets include ~1k examples from each survey.

from datasets import load_dataset

dset = load_dataset('MultimodalUniverse/plasticc', 
                    split='train', streaming=True)

example = next(iter(dset))

You can try this out with our getting started notebook!

To access the full dataset, we recommend downloading the data locally. This is necessary for using the provided cross-matching utilities.

The full dataset content is hosted at the Flatiron Institute and available either through HTTPS or through GLOBUS:

• https://users.flatironinstitute.org/~polymathic/data/MultimodalUniverse
• https://app.globus.org/file-manager?origin_id=57136152-fc1d-418e-b74e-75ca52bddd21

GLOBUS is much preferable when downloading large amounts of data, or a large number of files. Local download of the full data in its native HDF5 format is necessary for using the provided cross-matching utilities.

After downloading the data, you can use Hugging Face's datasets library to load the data directly from your local copy. For example, to load the PLAsTiCC dataset:

from datasets import load_dataset

dset = load_dataset('path/to/downloaded/plasticc', 
                    split='train', streaming=True)
dset = dset.with_format('numpy')

example = next(iter(dset))

The Multimodal Universe currently contains data from the following surveys/modalities:

We are accepting new datasets! Check out our contribution guidelines for more details.

We openly distribute the Multimodal Universe dataset under the Creative Commons Attribution (CC BY) 4.0 license, noting however that when using specific subsets, the license and conditions of utilisation should be respected.

Illustration of the methodology behind the Multimodal Universe. Domain scientists with expertise in a given astronomical survey provide data download and formatting scripts through Pull Requests. All datasets are then downloaded from their original source and made available as Hugging Face datasets sharing a common data schema for each modality and associated metadata. End-users can then generate any combination of subsets using provided cross-matching utilities to generate multimodal datasets.

Please see the Design Document for more context about the project.

If you make use of all or part of the Multimodal Universe dataset, please cite the individual datasets accordingly. The relevant BibTeX citations and text acknowledgement instructions for datasets can be generated through the info.py file (python scripts/info.py --help for details).

It allows you to retrieve all of the dataset information, or just acknowledgement and citation information for some or all datasets. By not specifying a dataset, it will return all datasets. By not specifying at least one of --citation or --acknowledge, it will return all of the information (including license, homepage, etc.).

python scripts/info.py --cite --data <datasets>
python scripts/info.py --acknowledge --data <datasets>

For example, to get the citations for the APOGEE and SDSS datasets and save them to info_citation.bib, run:

python scripts/info.py --cite --data apogee sdss -o info_citation.bib
To get all citations and acknowledgements, run:
```sh
python scripts/info.py --cite --acknowledge

You can always specify an output file for easy transcription to your bibliography or acknowledgements section with the --output flag:

python scripts/info.py --cite --output full_citations.txt
python scripts/info.py --acknowledge --output full_acknowledgements.txt

Acknowledgement instructions are returned alongside citations to encourage attribution. The acknowledgement lines are commented with % to make the citations easy to add to your bibliography.

Francois Lanusse 📆 💡 💻	Liam Parker 📆 💡 💻	Micah Bowles 📆 💡 💻	mhuertascompany 📆 💡 💻	Mike Smith 📆 💡 💻	Helen Qu 📆 💡 💻	Aaron 💡 💻
Ben Boyd 💡 💻	Brian Cherinka 💻	Connor Stone, PhD 💡	David Chemaly 💡 💻	Erin Hayes 💡 💻	Henry Leung 💻	Ioana Ciucă 🖋
Jeff Shen 💻	jeraud 💡 💻	John F. Wu 🖋	CambridgeAstroStat 🧑‍🏫	Kartheik Iyer 💻	Lucas Meyer 💻	Matthew Grayling 💡 💻
Maja Jabłońska 💻	Mike Walmsley 💡 💻	Miles Cranmer 🖋	Peter Melchior 💻	Rafael Martínez-Galarza 💻	Tom Hehir 💡 💻	Shirley Ho 🔍 🖋