Ziv Elbaz 3H AGO

My AI digest looked perfect and was quietly wrong. What actually fixed it. Every run succeeded. Every dashboard was green. And the content was still wrong. My digest invented "AI Moat Brief", a newsletter that does not exist. It reported scan counts nobody measured. It resurfaced week-old stories as fresh headlines. Here is what broke, and what fixed it. The sorting used to happen in my head: skimmed subject lines, unopened tabs, quiet guilt. The Signal is one email at 08:00: a single Make scenario calling Claude Sonnet through OpenRouter. It reads the last 24 hours of my RSS feeds and newsletters, keeps what touches what I am actively building plus the domains I need to stay current in, and arrives in the language I actually think in. Core items end with what it means for my work. Five to ten minutes, and I know where to go deep today. Structurally it looks like this, minus the content, rendered in English for this post (Image 1). No real edition is shown; section names and sample lines are illustrative. The dangerous failures were never pipeline failures. They were content failures, and the cause is structural: an LLM summarizing newsletters that already summarize primary sources is third hand by construction. Every hop strips attribution and adds confidence, and when data goes missing the model fills the gap the way LLMs do: fluently. Valid HTML, confident tone, green pipeline, wrong content. Image 2 is that whole failure class in one frame. Three rules closed the gaps I caught, all live in production: Step-by-step: 1. Verbatim or nothing. A source name is copied character for character, and a link exists only if that exact URL is in the input. The model copies; it never composes. 2. The model never generates metrics. Every count the report shows is injected by the pipeline after the model returns. 3. Recycled news gets demoted. A recap of recaps gets one line at most, and is dropped when the underlying story falls outside the collection window. Rules 1 and 3 lean on the prompt, and that is why the counters exist. The pipeline writes a hidden HTML comment into every email it sends: items, links, urls, cost, finish status. That line caught what I could not see. In one run, the published-links counter and the leftover-urls counter read 45 and 435: the only sign a new cleanup step was a silent no-op. Another morning the model stopped at 15,999 tokens against a 16,000 cap, one token from an email cut off mid-sentence. On the morning I wrote this they agreed, 21 links and 21 urls, and boring is the goal. Image 3 is that morning's actual comment, with the same two counters from the no-op run. The run itself has a dead man's switch on Healthchecks.io, so a missing 08:00 email reaches me before I notice. Honest limits: the $0.31 per report is a fresh measurement I am still validating, and I have not proven these rules hold as the source set scales. There is more behind every part of this; I would rather share it where it is wanted. Ask and I will put it in the comments: the three rules in full, the exact cost and what drives it, what this replaced in my day, how it compares to what is on the market, or the ugliest of the 15 documented bugs. I am sharing this because I doubt I am the only one building fragile things behind the scenes, and monitoring text is harder than monitoring uptime. What content-level checks do you run on LLM output, the kind pipeline monitoring cannot see? Real thresholds and embarrassing failures especially welcome.