Aether Gateway

Measured, not estimated

Every number here is reproducible via make bench against a real running stack – full methodology and raw data in BENCHMARKS.md.

8.9ms

Cache-hit latency, p95

target: 50ms

67.92ms

Failover latency, p99 (100 trials)

target: 500ms

500/500

Concurrent streams survived a real rolling update

run twice, zero broken

76.8%

Line coverage, 4 core modules

JaCoCo, unit + integration

What actually happens on a request

The response headers tell you exactly what the gateway did – no guessing whether a request was served from cache or which provider actually answered it.

$ curl -X POST http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"mock","messages":[{"role":"user","content":"What is 2+2?"}]}' -D -

HTTP/1.1 200 OK
X-Aether-Cache: MISS
X-Aether-Cost-USD: 0.0000195
X-Aether-Attempts: 1
X-Aether-Provider: mock-primary

Same prompt, sent again:

X-Aether-Cache: EXACT_HIT – served from the semantic cache, X-Aether-Cost-USD: 0

Force the primary provider to fail, then send a new prompt:

X-Aether-Provider: mock-fallback – failover happened, same request shape, no client change

This is a real transcript from a real docker compose up, not a mockup – see the Getting Started guide to reproduce it yourself in under a minute.

What it does

The engineering value isn't in calling models – it's in everything around the call. Six things it actually handles for you:

Routing and failover

Circuit breakers, retries, and bulkheads across multiple providers (Ollama, Groq, Gemini, or any OpenAI-compatible endpoint) – a failing provider doesn't take your application down with it. See why breaker state is local, not shared.

Semantic caching

Near-duplicate prompts are served from a vector cache (Postgres + pgvector) instead of hitting a provider again, with a guard against false hits. See how the false-hit rate is actually measured.

Quota enforcement

Per-API-key rate limits, concurrency caps, and monthly token budgets, checked and reserved atomically before a request is ever dispatched – not after the fact.

Cost accounting

Every request’s real (or avoided, on a cache hit) cost, computed from a configurable per-model price sheet and exposed in response headers, the request log, and a Grafana dashboard.

Versioned prompt registry

Name a prompt, version it, and roll a production alias back to a previous version on the next request – no gateway restart, no redeploy.

Kubernetes-proven

Graceful SSE drain under a real rolling update, in-flight-stream-based autoscaling, not CPU. See the graceful-drain chain in detail.

Bring up the whole stack in one command

Gateway, Postgres+pgvector, Redis, two mock providers, Prometheus, and a pre-provisioned Grafana dashboard – seeded and ready, no manual setup.

Getting Started guide →Browse the docs