You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at the top end snappy.
The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists.
I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds.
I need to discard the requests that overlap the model changeover for a truer result.
does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. that's obviously a much bigger lift than what you've built here, but it would turn a hard stop into something closer to a brief latency bump for whoever's unlucky enough to hit it, rather than a shared 25s wall for everyone behind them in line.
Good question. short version: it's a shared wall, not a per-request bump. one KV260, one model on the fabric at a time, so the rotation is global. but the ~25s is mostly software, not fabric-dark time. the PL reconfigure is ~1-3s via fpgautil, the rest is orchestration (daemon teardown, streaming the weights back into URAM, clock change, coherence check, chat-server reconnect). DFX keeping the sequencer + I/O live only recovers that 1-3s slice, and it fights the design, since the weights are smeared across every URAM column while a reconfigurable partition wants a contiguous pblock.
The swap needs a fresh bitstream at all only because four little ROMs are baked at synthesis: layernorm gammas, the activation-quant scale, two dequant tables. the embeddings already ride the boot-streamed weight image. make those four boot-loadable through the same path and a same-arch model hot-swaps with no reconfigure. There is basically zero margin at the moment to implement this.
which is a decent excuse for a 24h telemetry update.
Peak 46 concurrent connections monday afternoon, 61 uniques in the busiest half hour, ~6,600 requests / ~400k tokens total. At no instant did more than 2 inferences overlap (although it came close once). the fabric finishes a ~100-char reply in ~2ms of compute (end-to-end round-trip is ~20-60ms once you add network + server), so even with 40+ people connected the queue sits empty ~100% of the time. Worst 30-second window of the whole day was p95 59.9ms, and the heaviest burst (~200 requests in 5 minutes from one enthusiastic visitor) didn't even produce it. still ticking at ~37 connections this morning, latency flat.
That's the counterintuitive bit about a 20k tok/s model with short replies: the "single stream" is busy for milliseconds per request, so it timeshares across human typing speed essentially for free. the synthetic ceiling (2,000 concurrent requesters, p95 ~9.4s, zero errors) is ~40x past anything real traffic did. every actual visitor had the chip to themselves.
After this is all over, I'll do a post-mortem to see how it held up. I was really hoping for FP with the amount of effort this one took compared to my other submissions, which really would have stressed it, but I guess it was just a bit too niche!
The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists.
I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds.
I need to discard the requests that overlap the model changeover for a truer result.