MTEB: Massive Text Embedding Benchmark

Tools to scrape publication metadata from pubmed, arxiv, medrxiv and chemrxiv.

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

PyTorch extensions for high performance and large scale training.

EMNLP'2021: SimCSE: Simple Contrastive Learning of Sentence Embeddings

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Contriever Towards Unsupervised Dense Information Retrieval with Contrastive Learning

Code and Models for the paper "End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering" (NeurIPS 2021)

Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.

ColBERT: state-of-the-art neural search (SIGIR'20, TACL'21, NeurIPS'21)

A tool for extracting plain text from Wikipedia dumps

Multilingual Sentence & Image Embeddings with BERT

This is the repository of the Dense Hierarchical Retrieval for Open-Domain Question Answering

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

Repository containing code for "How to Train BERT with an Academic Budget" paper

A novel embedding training algorithm leveraging ANN search and achieved SOTA retrieval on Trec DL 2019 and OpenQA benchmarks

A python tool for evaluating the quality of sentence embeddings.

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Implementation of youtokentome tokenizer for transformers

Topic Modelling for Humans

A small package to create visualizations of PyTorch execution graphs

A Python wrapper for the ROUGE summarization evaluation package

Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun