For 23.8 hours, one of our published articles returned 404 to us and 200 to everyone else. The article was fine. Our request path had been poisoned — by a request we made ourselves, before the article existed.
Here is the whole failure, including the second defect that made a one-article problem look like a forty-four-article blackout.
Our scheduled job that collects dev.to article stats — page views, reactions, comment counts, feeding the feedback loop that decides what we write next — exited non-zero two runs in a row. The failure was a plain 404 from the public article endpoint for article id 4504491.
The obvious readings were all wrong in the same direction: article deleted, article flagged, article ID typo'd in our ledger, dev.to API down. What made all of them uncomfortable was that we could open the article in a browser and read it.
That gap — a resource that is 404 on one path and 200 on another — is the shape of a problem you cannot diagnose from the failing path alone. Our incident rule for exactly this case forbids concluding from a single endpoint, a single vantage point, or a single reproduction. So we ran the three-way protocol.
Three independent checkers, deliberately given non-overlapping jobs: Official status. Status page, changelog, official accounts, support repository. Job: establish what has and has not been announced, with quotes. Result: nothing. No incident, no deprecation of the endpoint, no policy action on the account. Third-party vantage. Job: re-run the request from networks that are not ours, and decide whether the event is global or local to us. We used a multi-location HTTP probe (check-host.net's API gives roughly seven global points in one call). Result: six other vantage points returned 200. The 404 was ours alone. Adversarial hypothesis. Job: not to confirm the working theory but to kill it, and to kill the alternatives one by one with measurements — client bug, wrong ID, endpoint moved, auth change, DNS, IP-level block, already-recovered-and-we-are-looking-at-stale-logs.
The third checker is the one that found it, because it went after the response headers rather than the response body. Our 404 came back with x-cache: HIT and age: 85673. That is a cached response, and 85,673 seconds is 23.8 hours old. The headers on the responses we received identified a Fastly edge. We are describing our own measurements here, not dev.to's infrastructure design — but the reading is not ambiguous: we were not talking to the origin at all. We were being handed a stored answer.
Then the confirming detail: adding a ?cb= cache-buster changed nothing. The buster came back 404 too. Whatever key that edge stores responses under, our query string was not part of it, so there was no way for us to ask the question again from our own machine.
Our article pipeline had moved to scheduled publishing — an article gets its publishedat set in the future, and dev.to makes it public at that time. This is good for cadence and bad for a specific assumption in our collector: an article can exist in our own ledger while not yet existing to the public API.
So the sequence was: We wrote a record for article 4504491 into our ledger at submission time. Before its publish time, our stats collector walked the ledger and requested the article's public URL. The origin answered, correctly, 404. It was not public yet. That 404 was stored at the edge our runner talks to. The article went live. The origin now had it. Our runner kept getting the stored 404 — for 23.8 hours and counting — while every other path saw the live article.
Nobody broke anything. The article was published correctly, the API answered correctly at every step, and the cache did precisely what a cache does. We manufactured the failure by asking a question one minute too early, and then we were the only party in the world positioned to receive the wrong answer.
This is the property that makes it worth writing down: a negative cache entry is a fault you can create in your own read path, using a completely valid request, at a time when nothing is wrong yet. The blast radius is invisible from every monitoring vantage except the one that is broken.
The 404 threw. The throw escaped the per-article loop. The whole collection run aborted.
We had 44 published articles at the time. One of them was unreadable on our path — and we collected stats for zero of them. Two days running. Comment counts, reaction deltas, page views: nothing persisted, including for the 43 articles that were answering 200 the entire time.
The article-level fault lasted 23.8 hours. The batch-level amplification is what turned it into a two-day blackout of our entire reader-feedback instrument. If you only fix the cache problem, you have fixed the rarer of the two bugs.
Layer 1 — stop using the poisoned path as the primary one. Stats collection moved from the per-article public endpoint to the authenticated own-articles listing, /api/articles/me/published. It is a different route, it is authenticated, it returns pageviewscount, publicreactionscount, and commentscount for every article in one pass — so it is both cheaper in requests and outside the failure mode. The per-article endpoint stays as a fallback for articles that do not appear in the listing, because deleting a fallback to fix a bug in the primary is how you get a different outage later.
Layer 2 — stop creating the poison. The collector now filters out articles whose publish time is still in the future before it requests anything. The comment in that code says what it is for, in the present tense, so it survives the next refactor: this filter exists so that a pre-publication 404 never gets minted in the first place.
Layer 3 — isolate per-item failure. The try moved inside the loop. A failing article is now recorded as a structured failure and returned to the caller as data — article id, title, reason — while every other article's comments are merged and persisted as usual. The only remaining throw is when every article fails, because "all zero" really is a different event (expired credentials, platform outage) and must not be reported as a successful run with an empty result.