The Great Test, Part 4: The Topic-to-Publish Machine

Anyone can wire four API calls in a row and call it a pipeline. The real problem with unattended publishing is that every stage depends on the last one, and the world is hostile: research APIs rate limit, extraction occasionally times out, a model call fails mid-article, and a domain you registered yesterday gets indexed at whatever pace Google feels like. A human editor absorbs all of that by checking in every day. A machine has to absorb it structurally, or it stops silently, and the worst failure of an autonomous site is not an error message, it is quiet. So the topic-to-publish chain had to be built as a system that survives its own failures: every step retryable, every state durable, and the whole thing metered to the call.

On August 19, 2026, the first version of the pipeline ran as a single linear script, and the build log preserves the whole run: reconcile facts (6 claims, 2 hypotheses), draft the article (290 words), build the SEO payload, generate a hero image, run a polish pass (readability 58, quality 85), run a guardrail check (plagiarism similarity 35.5, acceptable), publish, regenerate the sitemap, and submit the URL to Search Console. Thirty-four seconds, start to finish. As a proof that the chain could work, it was perfect. As a way to run a website, it was a trap: a linear script either runs or it doesn't, and when it doesn't, nothing tells you where it stopped. The next week's build log shows why. The orchestrator logged jobs failing on a missing environment key, then retrying on a schedule, then failing again. A reconciliation job died because a topic record was created and deleted in a race. Each failure was survivable, but in a linear design, one failure stops the whole line. So the pipeline was rebuilt around a durable job queue: every stage of the chain became a job with attempts, max attempts, priority, a lease with an expiry, a lock token, and a next_run_at timestamp. Nothing runs ad hoc anymore. A heartbeat automation fires every thirty minutes and claims a batch of queued jobs, and a production batch tops up the pipeline nightly at 02:00 UTC. Upstream, a discovery sweep feeds a ranked topic bank, and paid research jobs fetch and ingest sources. A bank system keeps the queues from starving or flooding: a buffer target of 25 ready articles, a floor of 10, and a claim rate of 2 per tick. Downstream, publish is deliberately throttled: a publishing window from 08:00 to 20:00 UTC, a minimum spacing of 45 minutes between posts, and a publish cadence governor that sets the daily target (minimum 2, maximum 10) based on evidence. Its rule is beautiful in hindsight: it steps the target up only when at least 80 percent of recent articles got indexed within a 7-day window, steps down below 50 percent, and watches whether the crawled-but-not-indexed count is rising. The machine is allowed to publish more only when Google proves it is actually absorbing what has already been published.

You can see the design working in the logs. A typical heartbeat claims 12 jobs: six succeed, six defer. Deferral is not failure, it is backpressure, the queue saying the next stage is not ready yet. And when a human finally cleared the blocked publisher on September 24, the queue did exactly what it was built to do: it drained, publishing 27 articles in 23 minutes. The stockpile was never wasted effort, it was inventory waiting for the belt to start moving again. What week one exposed instead were the edges the queue cannot defend by itself: a breaker that tripped on a misconfigured call budget, and a redraft stage that was never triggered. Those are next-turn fixes, and the monitoring loop now catches them within a day instead of never.