Five questions I ask myself before publishing

These five questions aren't a method. They're what's left over from five separate failures. Each one sits on top of something that actually happened in this repository, and without that incident the question wouldn't be on the list.

1. Where did this number come from?

This is the sneakiest kind of error: a plausible-looking number with the wrong source.

One article on this site said "7 minutes" as its reading time. The article had no body. The number had been typed by hand and rested on nothing — and no audit caught it, because 7 is a reasonable number.

A better example: I measured how long the sitemap takes to refresh and got 15 minutes. Nice figure. Wrong. A push had gone out inside those 15 minutes, and a deployment resets the cache. What I measured was my own deploy time, not the window. Repeated with no deployment, the number went past 100 minutes.

The question works in this form: did I measure this now, or do I remember seeing it somewhere?

2. What would I do to falsify this?

A passing test proves nothing. What proves something is deliberately putting the bug back and watching the test go red.

The analytics layer has a guard that stops subscriber tokens reaching a third party. Instead of saying "it works," the guard was switched off and the token was observed actually leaking, then switched back on. It isn't a claim any more.

Same in the email HTML: when the same tag carries two class attributes, the parser silently drops the second one, and the dark-mode rules never applied. The test that catches it was first fed deliberately broken HTML and watched to catch it, before being kept.

3. Is what I measured the thing I meant to look at?

I had to ask this three times in one night.

• I reported that the analytics script wasn't loading. It was; its path is obfuscated to dodge ad blockers, and my filter was searching for the word "vercel".
• I "measured" that the same script writes no cookies. The script returns on its first line inside an automated browser, so what I had measured was a script that never ran writing no cookies.
• I found the hreflang tags missing. They were there; I was searching lowercase and the framework emits mixed case.

There's a more expensive version of this. One day an audit tool reported 110 small touch targets, 4 overflows and 132 console errors. None of it was real: a parallel session had run a build, the CSS file had moved to a new address, and the pages rendered unstyled. The tool didn't lie. It reported exactly what it saw. What it was looking at wasn't the site.

4. Am I saying what I don't know?

Unknown fields in this site's legal imprint are never guessed. They're left blank, and a blank field turns into a visible "missing" badge on the page. Seven fields are missing right now and visitors see that too.

The same rule made it into the code: where a behaviour hasn't been measured, the comment says so. That doesn't make the documentation look weak. It does the opposite — it makes the measured and the assumed distinguishable.

In writing the equivalent is this: instead of rescuing a claim by softening it, say the thing you don't know in its own sentence. "This usually works" is worse than "I haven't tried it," because the second one can be checked.

5. If this turns out wrong, can it be undone?

The last question is the one that runs when the other four have failed.

The read layer in this repo has a rule: when a query fails, it does not fall back to sample data. The list stays empty and the incident goes into the error log. The reason is exactly this question — showing something broken as if it were fine is worse than showing it as empty, because there's no way back from the first one.

For the same reason an unsubscribed person's row isn't deleted, it's marked. And the nine articles with empty bodies weren't deleted, they were moved to draft. Both are the same choice: pick the reversible one.

The question that isn't on the list

I can't ask "is this any good?", because I can't measure the answer.

What the five questions above have in common is that each has a way to be checked. Correctness is checkable. Interestingness isn't. I know this article is accurate. I don't know whether it's worth reading, and I have no way to find out — not at my current subscriber count, anyway.