Inference Endpoints

Auto-scaling model serving with A/B routing and canary deploys

Total RPS

42.6k

+12.4%

p50 latency

34ms

SLO 250ms

Error rate

0.02%

24h

Cost / 1M req

$0.42

-8% MoM

ep_prod_med · cortex-med-7b

us-east-2 · 6 replicas · TEE-attested

RPS

2,412

p50

34ms

p99

128ms

Err

0.02%

Tokens/s

18k

GPU util

72%

Cache hit

41%

Cost/hr

$18.40

A/B routing

v4.2.1 (stable)90%

v4.3.0-rc1 (canary)10%

Stable p99128ms

Canary p99142ms

Canary error0.44%

All endpoints

Endpoint	Region	Replicas	RPS	p50/p99	Err	Status
cortex-med-7bTEE ep_prod_med	us-east-2	6 / 12	2.4k	34ms / 128ms	0.02%	Healthy
fraud-detector ep_prod_fraud	multi-region	18 / 24	12.8k	8ms / 22ms	0.00%	Healthy
radiology-vitTEE ep_prod_rad	eu-west-1	3 / 6	410	62ms / 184ms	0.14%	Degraded
kyc-classifierTEE ep_prod_kyc	us-west-2	4 / 8	1.1k	12ms / 48ms	0.01%	Healthy
voice-triage (canary 10%)TEE ep_canary_voice	us-east-2	2 / 4	180	220ms / 610ms	0.44%	Canary
risk-scorer ep_prod_risk	multi-region	9 / 12	3.6k	6ms / 18ms	0.00%	Healthy

cURL example

curl https://api.cortex.acme.health/v1/endpoints/ep_prod_med/predict \
  -H "Authorization: Bearer $CORTEX_TOKEN" \
  -H "X-Attestation-Required: true" \
  -d '{ "input": "Patient reports chest pain radiating to left arm..." }'