The Oshepherd guiding the Ollama(s) inference orchestration.

A centralized FastAPI service, using Celery and Redis to orchestrate multiple Ollama servers as workers.

pip install oshepherd

1. Setup Redis:

   Celery uses Redis as message broker and backend. You'll need a Redis instance, which you can provision for free in redislabs.com.

2. Setup FastAPI Server:

   # define configuration env file
   # use credentials for redis as broker and backend
   cp .api.env.template .api.env
   
   # start api
   oshepherd start-api --env-file .api.env

3. Setup Celery/Ollama Worker(s):

   # install ollama https://ollama.com/download
   # optionally pull the model
   ollama pull mistral
   
   # define configuration env file
   # use credentials for redis as broker and backend
   cp .worker.env.template .worker.env
   
   # start worker
   oshepherd start-worker --env-file .worker.env

4. Now you're ready to execute Ollama completions remotely. You can point your Ollama client to your oshepherd api server by setting the host, and it will return your requested completions from any of the workers:

   • ollama-python client:

   import ollama
   
   client = ollama.Client(host="http://127.0.0.1:5001")
   ollama_response = client.generate({"model": "mistral", "prompt": "Why is the sky blue?"})

   • ollama-js client:

   import { Ollama } from "ollama/browser";
   
   const ollama = new Ollama({ host: "http://127.0.0.1:5001" });
   const ollamaResponse = await ollama.generate({
       model: "mistral",
       prompt: "Why is the sky blue?",
   });

   • Raw http request:

   curl -X POST -H "Content-Type: application/json" -L http://127.0.0.1:5001/api/generate/ -d '{
       "model": "mistral",
       "prompt":"Why is the sky blue?"
   }'

This package is in alpha, its architecture and api might change in the near future. Currently this is getting tested in a controlled environment by real users, but haven't been audited, nor tested thorugly. Use it at your own risk.

As this is an alpha version, support and responses might be limited. We'll do our best to address questions and issues as quickly as possible.

• Generate a completion: POST /api/generate
• Generate a chat completion: POST /api/chat
• Generate Embeddings: POST /api/embeddings
• List Local Models: GET /api/tags (pending)
• Show Model Information: POST /api/show (pending)
• List Running Models: GET /api/ps (pending)

Oshepherd API server has been designed to maintain compatibility with the endpoints defined by Ollama, ensuring that any official client (i.e.: ollama-python, ollama-js) can use this server as host and receive expected responses. For more details on the full API specifications, refer to the official Ollama API documentation.

We welcome contributions! If you find a bug or have suggestions for improvements, please open an issue or submit a pull request pointing to development branch. Before creating a new issue/pull request, take a moment to search through the existing issues/pull requests to avoid duplicates.

To run and build locally you can use conda:

conda create -n oshepherd python=3.8
conda activate oshepherd
pip install -r requirements.txt

# install oshepherd
pip install -e .

Follow usage instructions to start api server and celery worker using a local ollama, and then run the tests:

pytest -s tests/

This is a project developed and maintained by mnemonica.ai.

MIT