// HACKER NEWS — CYBERSECURITY
So you want to use OpenRouter?
Might seem simple on the face of it, but unfortunately it's pain all the way down.
I run Olly, an AI assistant that lives in iMessage, on open source models through OpenRouter. To date Olly's transacted over 18 million messages, roughly a third of those on open models via OpenRouter. That's enough volume to hit every edge case at least once. So here's a list of things I wish I'd known going in.
But first quick vocab: the model is the weights. The provider is who OpenRouter routes you to, they host the model on their GPUs, at their chosen precision, and their "proprietary" optimizations, with their own XML/tool parsers, which means each has a "proprietary" list of bugs too. When you ask for deepseek/deepseek-v4-flash you get one of ~20 companies you've mostly never heard of. They're the same model on paper, but very different models in real life.
Ok, here's a few of the pitfalls you should watch out for.
OpenRouter runs per-provider benchmarks on the same model: GPQA Diamond and TAU-Bench Airline (a tool-calling task). Here is today's board for DeepSeek V4 Flash 0731, every provider serving the exact same weights:
First-party DeepSeek: 90% GPQA, 81% TAU. DigitalOcean, same weights: 75% and 58%. Most hosts cluster 5 to 7 points below first-party on tool calling, and four of them fall off a cliff on knowledge. For an agent TAU is the score that matters and a 20 point swing is not noise. (In July it was worse: Fireworks scored 46% on TAU, a 30 point gap)
Check the board for the benchmark closest to your workload before you trust a provider. And recheck when you switch models, the same providers looked completely different on GLM-5.3.
I noticed some strange non-deterministic behavior on image tasks so I ran the same three tiny images (a letter, a solid color, a word on a background) through every host of two open vision models:
DeepInfra's Qwen endpoint read a K as an R, called red blue, and described the word "umbrella" as "funny", while four other hosts of the same weights got everything right. Venice and Together didn't see the MiniMax images at all. The model page says it supports image input, but two of its providers don't and even worse they'll pretend everything is 200 OK.
reasoning.effort is accepted everywhere. Whether it does anything depends on the model and the provider. I pinned every provider serving DeepSeek V4 Flash 0731 and sent the same prompt at low, high and max, three times each, from a prod machine, here's the reasoning tokens output: