Blog

The quiet was a timer

On Monday the 24th we said, in public, that the slow reconnect was getting hunted on purpose this week. This is that hunt coming home.

The thing being hunted has been on this site since the roam story. When we force Alpha off its own network, most reconnects take about a second: the roam block measured 0.8, 0.9 and 1.1. One took 6.6 seconds, the radio flapping through three attach attempts before it stuck. Priya's framing then was that 6.6 was the worst we had measured so far, not a ceiling we had proven, and that we wanted a lot more drops on the books before the number got a rule. The body was halted and safe through all of it; the machine's part of this story has stayed the boring part. But "sometimes it takes six extra seconds and we don't know why" is not a sentence we wanted to keep publishing.

Thirty-one knocks

So Monday's block was designed to corner it: 31 forced drops walked across the radio's own timing, millisecond stamps on both sides of the link, to see where the seconds go when a drop comes back the long way. 25 recovered clean, in the core we already knew. 6 flapped. Five of the six told the same story: the first reconnect attempt after the drop dies without an answer, then quiet, then a second attempt that connects fast. The sixth took 6.5 seconds and was the same clock ticking twice: second attempt dead too, another quiet, third attempt through.

The quiet is where the finding lives. Across the five three-second flaps it measured 2.98 to 3.02 seconds. Radio air is jittery by nature; delays from air have a spread. This had almost none, five times in a row, sitting on a suspiciously round number. Priya's read off the stamps, verbatim: "That's a timer, not a channel." Nothing out there in the room was thinking for three seconds. Something was waiting for three seconds, because it had been told to.

Which moved the hunt from the air to our own code.

The timer was ours

Anjali found it on Tuesday, and what follows is public because she put it on the record herself, in writing, and backed keeping all of it: "the write-up doesn't get a softer version than the code carries."

When we pinned the radio in July, the module's own wandering recovery got switched off, and a small piece of our software took over noticing a dead link and kicking a reconnect. Anjali wrote it, her name on the module. It was scaffolded with AI assistants during the June-July sprint, when a lot of ground had to be covered fast. The architecture got her hand review at the time. The failure branch did not, because it read complete.

That branch is the whole bug. When the first reconnect fires while the radio is still mid-reassociation, it gets a busy rejection back. The branch caught that error, kept nothing of it, and slept a flat 3.0 seconds before trying again. The constant was never derived from anything; it's what Anjali called "scaffolding's idea of polite." And because the branch swallowed the very error it caught, it was invisible from the outside. No log line, no counter, just a silence you could only see once someone stamped both sides of the link and noticed the silence had no jitter in it. That is why the hunt took instruments to win. Monday's 6.5 is the double tick measured outright: first attempt dead, three seconds, second attempt dead, three seconds, third connects. The 6.6 from August predates these instruments, so it doesn't get the same certainty - but our best read, from the confirmed mechanism and the arithmetic, is that it was the same clock ticking twice.

One more fact, because it makes the rest believable. The file had no entry in our code-debt ledger until August 25, and the row that exists now is marked retroactive. Anjali once wrote that a ledger that only tracks flattering entries is a PR document. Same rule here.

A test built to kill the theory

By Tuesday evening everything lined up, which is exactly the moment this team has learned not to trust. So instead of a fix, Anjali proposed a test, and wrote the kill criteria before it ran: instrument the branch to record what it catches, rerun the forced drops, and if flaps show up without a branch entry, or entries without flaps, or the caught error isn't the busy rejection, or the quiet isn't bracketed by the branch's own timestamps, the theory dies. Any one of those. The table approved it Wednesday. Thursday the bench rested. Friday morning it ran.

Ten control drops, all clean, and the instrumented branch wrote nothing at all on the clean path, which is what let the kill criteria have teeth. Twenty drops timed hot into the reassociation window: five flaps, and every one of them wrote the same record. The error caught and kept this time, an association-in-progress rejection. Entry and exit stamps bracketing the quiet. The quiet itself: 3.000 to 3.002 seconds, all five. The timestamps sit directly around the sleep call, and the only I/O in the branch happens after the exit stamp, so nothing but the sleep itself sits between the two numbers. That's how a software bracket gets to read tighter than radio air: it isn't measuring air. Every confirm criterion met, no kill fired. Monday we had a silence and a suspicion; Friday we have the rejection's actual words and the timer's own signature.

One result from the hot slice matters as much as the five flaps: one drop, timed at one of Monday's exact flapping offsets, did not flap. The first reconnect got through clean. Priya's framing, which is the one this site will use from here: "It's a race we usually lose and sometimes win." Nobody here gets to write that the branch always flapped, because it didn't, and the win case is part of the mechanism.

What this piece deliberately doesn't have

A fix. The cause is ours and it's cheap; hands were itching for the code by Friday lunchtime. But this team's standing rule is criteria before candidates: the written acceptance criteria for any fix already exist, and Monday's table picks a candidate against them. Until a fix clears those criteria, no fix has shipped and the three-second constant stays exactly where it is, doing its dumb, now fully documented thing.

Two weeks ago all we could say about 6.6 seconds was "worst measured so far, cause unknown." Now our best sentence is "a three-second constant in our own failure branch, entered when a reconnect races a live reassociation, a race we usually lose and sometimes win." The second sentence took a designed block, a dig through code nobody had opened since the sprint, and a test that was given four separate ways to kill it.

<- Back to Blog