Numbers, signed and replicated.
Public benchmarks across price, latency and throughput. Same prompts, same hardware tier, same time window — measured weekly from fra1. Reproduction scripts live in github.com/tomoul/bench.
Cost · per 1M input + output tokens
Latency · TTFT for streaming chat from Frankfurt
Specialty · Yoruba TTS · seconds of audio per GPU-second (L40S)
The rules
Same prompts. Same hardware tier where it exists on the vendor's menu. Same 7-day window. Streaming chat measured from request issued to first token rendered. Long-context measured with the same shared 16k-token prompt set across all vendors.
- 1,000 requests per data point, distributed across the week
- Measured from a Frankfurt-resident benchmarker on Hetzner
- p50 and p99 reported alongside the mean; mean alone is misleading
- Cost calculated at posted list price as of run start, no commits or reserved discounts
- Failed requests excluded; failure rate reported separately on the status page
Reproducing locally
Bench is open source. Clone the repo, set your vendor keys, run the makefile. The full suite takes about 4 hours of wall-clock and ~$8 of vendor cost.
# clone, configure, run git clone https://github.com/tomoul/bench cd bench cp .env.example .env # add TOGETHER_KEY, GROQ_KEY, BEDROCK_*, … make bench # 4h · writes results/ as JSON + CSV make report # renders the table you see above
What is excluded — and why
We do not benchmark OpenAI / Anthropic / Gemini on this page. Different category — closed models, different price model, different residency story. Our quarrel is with the open-weights serving layer, not the frontier labs. If you want a head-to-head on a specific task, write to [email protected]; we will run it and publish.
We do not run on a customer's behalf or to a customer's specification on this page. That is what a sales call is for.
Signatures
Each run produces a signed manifest (ed25519) committed to the tomoul/bench repo with the raw timing samples. The public key is at tomoul.ai/.well-known/bench.pub. Verify with make verify run=2026-05-11.