Every time I put a model behind an endpoint I make the same lazy decision. I pick whatever I used last time, or whatever I read about most recently, and I tell myself I'll benchmark it properly later, and later never arrives because there is always something with an actual deadline on it and comparing model latencies feels like procrastination even when it isn't. I never do it. Not once.
So I built the thing that would make me do it. One prompt, fired at six models at once, streaming side by side in columns, with time to first token and cost per run underneath each one. About 390 lines of Python. Code's here, MIT, take it.
Every model below goes through that one client. Llama, DeepSeek, Mistral, Qwen, OpenAI's open-weight gpt-oss line. Only the model string changes.
That is the pitch, and it's real, and I'll move past it quickly because you already knew an OpenAI-compatible endpoint would work like an OpenAI- compatible endpoint. What I didn't know is everything that follows.
One footnote before you paste that snippet. The credential is a model access key, created under the Gradient AI Platform. It is not the API token from Settings, API. Different thing, different page. (Although, as I found out later, the endpoint doesn't care nearly as much about that distinction as the docs do.)
I wanted the columns to fill simultaneously. Real racing, not six sequential progress bars pretending.
The tidy way to do that is one endpoint that fans out server side and multiplexes everything back down a single connection. I didn't do the tidy way. The browser opens one EventSource per model instead:
Six models, six connections, six independent lifetimes. Nothing merges anything. Flask stays synchronous, no async, no orchestration layer, and the entire streaming path is about forty lines.
I did it that way because it's simpler, and I stand by that. But the reason I'm glad I did it turned out to be different from the reason I chose it, which I'll get to.
I knew gunicorn's default sync worker would be a problem. It handles one connection per worker process and holds it until the response is done. Fine for requests that last 40 milliseconds. Streaming responses stay open for seconds, so six concurrent streams need six workers or they queue.
I predicted the page would hang. It doesn't hang. Here's what six concurrent streams actually look like against one sync worker:
| model | first token | | --- | --- | | mistral-3-14B | 1250 ms | | openai-gpt-oss-120b | 5326 ms | | openai-gpt-oss-20b | 7278 ms | | deepseek-3.2 | 8347 ms | | llama-4-maverick | 9147 ms |
Look at the spacing. Each stream's first token shows up right about when the previous stream finished. That's not slow models, that's a queue. Six requests, one at a time, 10.7 seconds to get through all of them.
Now four of the six first tokens land inside a 1.4 second window instead of marching across a ten second one, and the whole thing takes 6.4 seconds.
But go back and look at that first table again, because the interesting part isn't the fix. Every one of those requests succeeded. Correct responses, no timeouts, no errors, nothing in the logs. If I'd shipped the broken version I would not have filed a bug against myself, I'd have watched the columns fill in one after another, concluded the models were slow, and gone off to write a caching layer for a problem that was sitting in my Procfile the entire time. A hang would have been kinder. A hang makes you look at your server.
The model picker is built from GET /v1/models, because hardcoding a model list is how you end up shipping a dead one.
There is nothing in the /v1/models response that tells you which is which. No availability flag, no tier field, no hint. You find out by calling it and reading the 403.
Which is how I shipped a broken default. My preselected list had anthropic-claude-haiku-4.5 sitting right there in it, because it's in the published catalog and it's on the pricing page with a real per-token rate beside it, and at no point between reading those two documents and writing that list did anything suggest I ought to check whether my own account could call the thing. First real run, that column went red in front of me.