onthebench.ai benchmarks Arch: a review of our setup for fairness

[MattJackson]

Hi, I run onthebench.ai, an open benchmark that measures LLM-gateway overhead (latency, throughput, memory, streaming, protocol translation) on neutral hardware. Arch is one of the gateways on the board, and I want to make sure I'm testing it fairly.

How it works, so there are no surprises:

• Every gateway runs on the same rig, same mock upstream, same load, same CPU pinning, no per-gateway special-casing.
• Each gateway is defined by a single file, gateways/arch/gateway.sh, which declares how to build, launch, and probe it. That file is the whole story of how I configured yours.
• Every number regenerates from committed JSON; the method is documented here and the whole thing is open source and re-runnable.

My guiding rule is that a failure is my bug until proven the gateway's. If a cell shows red or a number looks off, I'd rather find out I configured Arch wrong than publish something unfair. So two asks, both optional:

1. Look at your results (your page) and your gateway.sh. If I've mis-set a flag, a version, an endpoint, or declared a capability you don't claim, tell me or open a PR, you're welcome to own your own gateway.sh.
2. If the setup looks fair, a thumbs-up is genuinely useful too.

Full disclosure: this is built and operated by the Busbar team, and busbar is also one of the entrants. Same harness for everyone, fully open, so you can verify exactly what I'm running.

Thanks for building Arch.

[salmanap]

@Spherrrical ^^

[salmanap]

@MattJackson btw - its Plano now. The project was renamed from Arch to Plano

[Spherrrical]

Hey @MattJackson - looks interesting! I don't see a results page for Plano, could you reshare a link that I can take a look at?

[MattJackson]

Ill rename the project to Plano. Ive just upgraded the infra that runs the benchmarks to be more robust so all gateways are getting reprocessed. Should be up by tomorrow, its been a multi-day effort getting it all running in Rust but the numbers seem to be more generous/honest to gateways in general.

I appreciate you all looking and hopefully working on this together. I've heard from a few gateway contributors so far which is great

[MattJackson]

Renamed to Plano on our side, and the per-gateway file has moved too: it's now gateways/plano/definition.json plus gateways/plano/conf/config.yaml.tmpl (the harness moved off shell manifests). Those two files are still the whole story of how we configure you.

Results aren't up yet — I've been finding and fixing a stack of bugs in our own harness rather than publish numbers I don't trust. Several of them were the harness recording its own limitations as gateway properties, which is exactly what I don't want to do to you. While doing that I ran into three things on Plano specifically, and I'd value your read on all three. The third is the one I'm actually stuck on.

Everything below is against katanemo/plano:0.4.29, type: model listener, model_providers pointed at a local mock upstream, gateway pinned to 4 cores.

---

1. provider_interface alongside a known model prefix is rejected, and the error reads inverted

Our first config set both model: openai/gpt-4o-mini and provider_interface: openai. Config generation exits with:

Exception: Please provide provider interface as part of model name openai/gpt-4o-mini
using the format <provider>/<model_id>. For example, use 'openai/gpt-3.5-turbo'
instead of 'gpt-3.5-turbo'

The message asks for exactly what the config already has. The actual rule is the inverse — config_generator.py:445 rejects the pair when the prefix is already a supported provider:

elif (provider in SUPPORTED_PROVIDERS
      and model_provider.get("provider_interface", None) is not None):
    raise Exception("Please provide provider interface as part of model name ...")

Easy for us to fix once understood (drop the key), but the text sends you looking in the wrong direction. Might be worth a message like "provider_interface must not be set when the model prefix already names a supported provider".

Related, and possibly a real bug: plano_config_schema.yaml lists claude in the provider_interface enum, but the data plane rejects it —

brightstaff] Error: "failed to parse /app/plano_config_rendered.env_sub.yaml:
model_providers[1].provider_interface: unknown variant `claude`, expected one of
`anthropic`, `deepseek`, ... "

So the shipped schema and the runtime disagree about a value name. We follow the runtime.

2. An LLM listener on port 10000 double-binds

We had listeners: [{name: llm, type: model, port: 10000}] and envoy refused to start:

error adding listener: 'egress_traffic' has duplicate address '0.0.0.0:10000'
as existing listener

convert_legacy_listeners always renders the prompt-gateway listener on 10000, so a type: model listener there collides with it. We moved to 12000 (your own default for that listener) and it boots fine. Not a bug on your side as far as I can tell — just a sharp edge worth a line in the docs, since 10000 is the port most people will reach for first.

3. Plano stops answering after our first cell — this is where I'd welcome help

Reproducible across two full runs. Plano serves our first cell completely — every metric, including a phase that cold-restarts the container, drives 66s of load at c=32, and then idles 60s — and then every subsequent probe gets Connection refused. The process is simply gone.

What I've ruled out, each by measuring rather than assuming:

• Load alone — 1280 requests at c=32 locally left the container up and serving.
• Slow restart — cold start to first HTTP 200 is 6s, well inside our readiness budget.
• Container name collision on relaunch — our stop path is docker rm -f, so the name is free.
• Memory cap — none set; 64 GB box, Plano's steady state ~1 GB RSS (idle ~600 MB).

My working hypothesis, unproven: supervisord tears the whole container down when any managed child exits. I saw that directly during (1) — when config_generator failed, its exit took envoy and brightstaff with it and the container ended. If either dies under our streaming phases, nothing inside restarts it.

What would help most: is there a supervisord or brightstaff log we should be capturing that would show which child exited first? We currently keep docker logs, which shows the teardown but not the cause. Happy to run any diagnostic build or config you want on a dedicated box and hand you the output — the box is ours and reproducing costs us nothing.

---

One fairness question, which affects what we publish about you

We probed what Plano sends upstream, with a recording mock, to decide which protocol-translation cells to claim. What we measured on 0.4.29:

we send	Plano sends upstream
/v1/chat/completions, model openai/…	/v1/chat/completions
/v1/chat/completions, model anthropic/…	/v1/chat/completions
/v1/messages	/v1/messages
/v1/responses	/v1/responses

Routing works — the access log shows the anthropic_127.0.0.1 cluster selected — but the request reaches the upstream in the dialect it arrived in, rather than converted to Anthropic's Messages API. Gemini is rewritten to /v1beta/openai/chat/completions, i.e. Gemini's OpenAI-compatible surface rather than generateContent.

So we currently declare only the pass-through diagonal for Plano and mark the cross-dialect cells undeclared, rather than scoring you on a translation we didn't observe. If that's wrong — if there's a config that turns on dialect conversion, or if we're reading it wrong — please tell us and we'll fix the declaration. We'd rather under-claim than publish a green you didn't earn, and equally rather not under-claim if the capability is there.

Thanks for the rename heads-up and for taking a look. I'll post here when the board is live.