This repo contains a handful of utilities for benchmarking the response latency of popular AI services, including:
Large Language Models (LLMs):
- OpenAI GPT-3.5, GPT-4 (from OpenAI or Azure OpenAI service)
- Anthropic Claude 3, Claude 2, Claude Instant
- Google Gemini Pro and PaLM 2 Bison
- Llama2 and 3 from several different providers, including
- Anyscale
- Azure
- Cloudflare
- Groq
- OctoAI
- Perplexity
- Together
- Mixtral 8x7B from several different providers, including
- Anyscale
- Azure
- Groq
- OctoAI
- Perplexity
Embedding Models:
- Ada-002
- Cohere
Text-to-Speech Models (TTS):
- ElevenLabs
- PlayHT
Snapshot below, click it to jump to the latest spreadsheet.
- Tests are run from a Google Cloud console in us-west1.
- Input requests are short, typically a single message (~20 tokens), and typically ask for a brief output response.
- Max output tokens is set to 100, to avoid distortion of TPS values from long outputs.
- A warmup connection is made to remove any connection setup latency.
- The TTFT clock starts when the HTTP request is made and stops when the first token result is received in the response stream.
- For each provider, three separate inferences are done, and the best result is kept (to remove any outliers due to queuing etc).
- A best result is selected on 3 different days, and the median of these values is displayed.
This repo uses Poetry for dependency management. To install the dependencies, run:
pip install poetry
poetry install --sync
To run a benchmark, first set the appropriate environment variable (e.g., OPENAI_API_KEY, ELEVEN_API_KEY) etc, and then run the appropriate benchmark script.
To generate LLM benchmarks, use the llm_benchmark.py script. For most providers, you can just pass the model name and the script will figure out what API endpoint to invoke. e.g.,
poetry run python llm_benchmark.py -m gpt-3.5-turbo "Write me a haiku."
However, when invoking generic models like Llama2, you'll need to pass in the base_url and api_key via the -b and -k parameters, e.g.,
poetry run python llm_benchmark.py -k $OCTOML_API_KEY -b https://text.octoai.run/v1 \
-m llama-2-70b-chat-fp16 "Write me a haiku."
Similarly, when invoking Azure OpenAI, you'll need to specify your Azure API key and the base URL of your Azure deployment, e.g.,
poetry run python llm_benchmark.py -b https://fixie-westus.openai.azure.com \
-m gpt-4-1106-preview "Write me a haiku."
See this script for more examples of how to invoke various providers.
usage: llm_benchmark.py [-h] [--model MODEL] [--temperature TEMPERATURE] [--max-tokens MAX_TOKENS] [--base-url BASE_URL]
[--api-key API_KEY] [--no-warmup] [--num-requests NUM_REQUESTS] [--print] [--verbose]
[prompt]
positional arguments:
prompt Prompt to send to the API
optional arguments:
-h, --help show this help message and exit
--model MODEL, -m MODEL Model to benchmark
--temperature TEMPERATURE, -t TEMPERATURE Temperature for the response
--max-tokens, -T MAX_TOKEN Max tokens for the response
--base-url BASE_URL, -b BASE_URL Base URL for the LLM API endpoint
--api-key API_KEY, -k API_KEY API key for the LLM API endpoint
--no-warmup Don't do a warmup call to the API
--num-requests NUM_REQUESTS, -n NUM_REQUESTS Number of requests to make
--print, -p Print the response
--verbose, -v Print verbose output
By default a summary of the requests is printed:
Latency saved: 0.01 seconds <---- Difference between first response time and fastest reponse time
Optimized response time: 0.14 seconds <---- fastest(http_response_time - http_start_time) of N requests
Median response time: 0.15 seconds <---- median(http_response_time - http_start_time) of N requests
Time to first token: 0.34 seconds <---- first_token_time - http_start_time
Tokens: 147 (211 tokens/sec) <---- num_generated_tokens / (last_token_time - first_token_time)
Total time: 1.03 seconds <---- last_token_time - http_start_time
You can specify -p to print the output of the LLM, or -v to see detailed timing for each request.
To generate TTS benchmarks, there are various scripts for the individual providers, e.g.,
python elevenlabs_stream_benchmark.py "Haikus I find tricky, With a 5-7-5 count, But I'll give it a go"
-
Ensure you have Poetry installed.
-
Set up the following environment variables or provide them as command-line arguments:
- ELEVEN_API_KEY
- CARTESIA_API_KEY
- PLAYHT_API_KEY
- PLAYHT_USER_ID
-
Run the benchmark using:
poetry run python tts_benchmark_suite.py "Your text here" [--eleven-api-key KEY] [--cartesia-api-key KEY] [--playht-api-key KEY] [--playht-user-id ID]
Example:
poetry run python tts_benchmark_suite.py "It's simple: Overspecialize, and you breed in weakness. It's slow death." --eleven-api-key YOUR_ELEVEN_KEY --cartesia-api-key YOUR_CARTESIA_KEY --playht-api-key YOUR_PLAYHT_KEY --playht-user-id YOUR_PLAYHT_USER_ID
or
poetry run python tts_benchmark_suite.py "It's simple: Overspecialize, and you breed in weakness. It's slow death."
Note: If you provide the API keys and user ID as command-line arguments, they will override any existing environment variables for that run.
(TTFU) time to first utterance: 316.48ms <---- Time from the outbound request start to first audio chunk received
Average chunk latency: 234.13ms <---- Average time between receiving consecutive audio chunks
Total chunks received: 12 <---- Number of audio chunks received for the entire request
Total processing time: 2809.58ms <---- Total time from request start of request to the final audio packet being received
By default, only timing information for TTS is emitted. Follow the steps below to actually play out the received audio.
First, install mpv via
brew install mpv
Then, just pass the -p argument when generating text, e.g.,
python playht_benchmark.py -p "Well, basically I have intuition."
You can use the -v parameter to select which voice to use for generation.