This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks, no training data required.

• Serializable llm component to integrate prompts into your spaCy pipeline
• Modular functions to define the task (prompting and parsing) and model
• Interfaces with the APIs of
  • OpenAI
  • Cohere
  • Anthropic
  • Google PaLM
  • Microsoft Azure AI
• Supports open-source LLMs hosted on Hugging Face 🤗:
  • Falcon
  • Dolly
  • Llama 2
  • OpenLLaMA
  • StableLM
  • Mistral
• Integration with LangChain 🦜️🔗 - all langchain models and features can be used in spacy-llm
• Tasks available out of the box:
  • Named Entity Recognition
  • Text classification
  • Lemmatization
  • Relationship extraction
  • Sentiment analysis
  • Span categorization
  • Summarization
  • Entity linking
  • Translation
  • Raw prompt execution for maximum flexibility
  • Soon:
    • Semantic role labeling
• Easy implementation of your own functions via spaCy's registry for custom prompting, parsing and model integrations. For an example, see here.
• Map-reduce approach for splitting prompts too long for LLM's context window and fusing the results back together

Large Language Models (LLMs) feature powerful natural language understanding capabilities. With only a few (and sometimes no) examples, an LLM can be prompted to perform custom NLP tasks such as text categorization, named entity recognition, coreference resolution, information extraction and more.

spaCy is a well-established library for building systems that need to work with language in various ways. spaCy's built-in components are generally powered by supervised learning or rule-based approaches.

Supervised learning is much worse than LLM prompting for prototyping, but for many tasks it's much better for production. A transformer model that runs comfortably on a single GPU is extremely powerful, and it's likely to be a better choice for any task for which you have a well-defined output. You train the model with anything from a few hundred to a few thousand labelled examples, and it will learn to do exactly that. Efficiency, reliability and control are all better with supervised learning, and accuracy will generally be higher than LLM prompting as well.

spacy-llm lets you have the best of both worlds. You can quickly initialize a pipeline with components powered by LLM prompts, and freely mix in components powered by other approaches. As your project progresses, you can look at replacing some or all of the LLM-powered components as you require.

Of course, there can be components in your system for which the power of an LLM is fully justified. If you want a system that can synthesize information from multiple documents in subtle ways and generate a nuanced summary for you, bigger is better. However, even if your production system needs an LLM for some of the task, that doesn't mean you need an LLM for all of it. Maybe you want to use a cheap text classification model to help you find the texts to summarize, or maybe you want to add a rule-based system to sanity check the output of the summary. These before-and-after tasks are much easier with a mature and well-thought-out library, which is exactly what spaCy provides.

spacy-llm will be installed automatically in future spaCy versions. For now, you can run the following in the same virtual environment where you already have spacy installed.

python -m pip install spacy-llm

⚠️ This package is still experimental and it is possible that changes made to the interface will be breaking in minor version updates.

Let's run some text classification using a GPT model from OpenAI.

Create a new API key from openai.com or fetch an existing one, and ensure the keys are set as environmental variables. For more background information, see the documentation around setting API keys.

To do some quick experiments, from 0.5.0 onwards you can run:

import spacy

nlp = spacy.blank("en")
llm = nlp.add_pipe("llm_textcat")
llm.add_label("INSULT")
llm.add_label("COMPLIMENT")
doc = nlp("You look gorgeous!")
print(doc.cats)
# {"COMPLIMENT": 1.0, "INSULT": 0.0}

By using the llm_textcat factory, the latest version of the built-in textcat task is used, as well as the default GPT-3-5 model from OpenAI.

To control the various parameters of the llm pipeline, we can use spaCy's config system. To start, create a config file config.cfg containing at least the following (or see the full example here):

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v3"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.model]
@llm_models = "spacy.GPT-4.v2"

Now run:

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("You look gorgeous!")
print(doc.cats)
# {"COMPLIMENT": 1.0, "INSULT": 0.0}

That's it! There's a lot of other features - prompt templating, more tasks, logging etc. For more information on how to use those, check out https://spacy.io/api/large-language-models.

In the near future, we will

• Add more example tasks
• Support a broader range of models
• Provide more example use-cases and tutorials

PRs are always welcome!

If you have questions regarding the usage of spacy-llm, or want to give us feedback after giving it a spin, please use the discussion board. Bug reports can be filed on the spaCy issue tracker. Thank you!

Please refer to our migration guide.