Stuck in the queue
In August I was running ANVISA, the Brazilian health regulator's website, through my manager Chris's first iteration of a harness, to see whether it could boldly build a scraper for a source we'd never scraped before, and to make the harness usable for the next 50-something sources we needed to scrape. The agent hit Cloudflare and stopped. It tried a headless browser, a visible one and a stealth one, only to hit the same "Just a moment" page every time from Cloudflare's bouncer. Its conclusion: we'd have to build our own entrance, with extra infrastructure, proxy credentials or a cookie jar.
Part of how I work with AI is simple: if the agent can't do something, I try it myself. So I opened ANVISA in my own Chrome, and it let me straight in. No challenge, no click. My browser was on the VIP list. I told the agent while it was still declaring the source dead, and my screenshot changed its recommendation. Once in, the club seemed pretty dead: a 20-second check from inside showed that the request the agent had been banking on wasn't even the list of acts. It was the list of years. A second check found the real room, a speakeasy with another way in, and the site's own page was already carrying the key: a page-size setting that handed over a whole year's acts in one response, no browser needed.
The agent had declared ANVISA non-viable twice that morning, and I would have had to escalate a non-problem to Chris. Instead, I cut the queue and saved Chris a headache. Turns out the bouncer wasn't checking for humans, as I'd initially thought. It was checking for a real, visible browser that remembers it's already been let in. Mine was one; the agent's weren't. In October we taught our browser service to be that kind of guest, keeping a visible session alive between requests, and the queue mostly disappeared.
That's why the harness I ended up building has a rule for asking a human to look. It's also why the agent isn't allowed to call in the builders until someone has checked the door. That morning set the pattern for everything that follows: build your harness against your hardest source, because every rule worth keeping comes from something going wrong there. Now here's the whole club.
The whole job, end to end
But first, some origin story. Flinn builds AI software that helps MedTech and IVD manufacturers stay compliant, and Regulatory Monitoring, keeping track of the regulations themselves, is one of its four solutions. That's where my scrapers come in. They gather all the regulatory acts our clients need to monitor, from many different sources, and for each act they capture what it is, who issued it where we can establish that, whether it's still in force, each consolidated version of its text, and which other acts it changes or is changed by. Then we serve them all in one place, in a consistent format. We're basically the Ash Ketchum of regulatory acts. Our clients stay informed about which acts exist, and stay updated on the changes to the ones they care about, so regulatory people can sigh with relief instead of hacking their way through yet another website's way of organising information. You'd think regulatory people would have found a way to regulate their websites, but no, we're not at that level of Inception yet.
I don't know why ANVISA decided to play so hard to get, but it is very exclusive. It turns away so many requests you'd think it's trying to outdo Berghain. It isn't even one club: the information is split across a chain of them, and we have to assemble everything from all of them. And it doesn't keep every version of an act in full, so we have to reconstruct them. No single page lists every act, half or more of our requests get a bot challenge, and an act's original text lives only in the official gazette, one day's edition at a time. I'll give the full tour in a future article. So it's all one big, slightly painful, regulatory treasure hunt. Here's the club we built to hunt it.
The map: six rooms
The harness works like a club with six rooms. In each room the agent does one job. Between the rooms are doors, and at every door stands a bouncer: a gate that checks the evidence before the work can move on. Most bouncers are code that checks automatically; at some doors, a human has to sign off too. We call the website we're scraping a source, and our work on it moves from room to room.
- Discovery. A regulatory expert makes sense of the source and matches it to our data model: where the information comes from, what it means, and how it should be stored and served. What comes out is the spec: our guest list, an answer key the finished scraper is checked against. It's what makes everything after it checkable: every bouncer in the club measures against it. At the door: a format check, a match with hand-built examples, and the expert's sign-off.
- Recon. The agent goes to the website and tries to find and open every entrance, to reach the information the expert found, from the best places, in the most efficient way. At the door: proof the agent reached the source, and every entrance accounted for.
- Approach. The agent looks at what already works in our existing scrapers, every option at every step, and makes a plan, choosing what it thinks will work best from reaching the website to serving the client. At the door: three bouncers, all required: buildable, approved by a human, and a valid build plan.
- Build. The agent executes the plan, writing the code and reviewing it, until the scraper gathers what the expert described, assembled exactly the way the expert said it should be. At the door: every task reviewed, every test passing, and the output matching the spec.
- Document. Once the scraper works, the agent writes down what it learned and adds it to the menu of options in Approach, so the next source sees every working option we have. At the door: every claim points to a file that was actually opened.
- Monitor. A scheduled check, not an agent, that the live website still behaves the way the scraper expects. It never changes code: a human reads its drift report and decides whether something is real drift, a legitimate change on the site, or a broken scraper. A check that didn't run is reported as exactly that.
My favourite rooms are Recon and Approach. I like digging into things, cracking a website, and laying foundations that prove strong enough to survive a full build. Build brings its own satisfaction: seeing the work actually work.
How a guest moves through the club
Moving through the club runs on three loops.
Loop 1: one room per visit
Each time the harness runs, it checks how far the work on the source has got, runs exactly one stage and stops; running it again moves the work on to the next room. That loop is Chris's. I added that every stage runs in a fresh session. I'd tried the same documentation task with an agent that had our whole conversation behind it, and with fresh Opus and GPT agents in separate chats, and the fresh ones did better: the long conversation was polluting the context. So each room starts clean, and what it needs from earlier rooms comes from the source's status and an append-only history log of what happened, which I added too, rather than from memory. If a stage stops halfway, a new session can pick up where the last one stopped instead of starting over.
Loop 2: the check before every door
Before every move, a routing check asks whether an earlier question is still open: something a previous room left unsettled, like an entrance nobody tried or a figure nobody confirmed. If there is such an open question, the next move is the back door to the room that owns that question, instead of the next room. A person can trigger it too, in plain words. That loop is mine, and it matters most for the early rooms: before it existed, nothing could undo them, and everything after them was built on whatever they got wrong. More in house rule 4.
Loop 3: review until clean
The first loop I designed was the review loop, though I didn't think of it as a loop then, just as the review process. It replaced writing the code in one shot, which I knew wouldn't work, because that's not how coding with AI works, or, you know, coding in general. In the build room, the agent builds, then a panel of reviewers, each a separate agent on a different model, checks the work. Every finding is tested before anything is fixed, and the round repeats until it finds nothing new. The whole story of that loop is for another article.
House rules
A club wouldn't be a good club without some house rules. Ours has plenty by now, fourteen of them written down as formal decisions. Here are four, and each one came out of something going sideways on ANVISA.
House rule 1: every state means a next action
What went sideways: Recon used to stop at the first good entrance it found on the website, and entrances it had noticed but never followed simply disappeared from the record. So now every entrance the agent finds gets a name and a state, and each state says what happens next: discovered (work out the exact request), parameterized (try it once), probed (Approach can choose it) or rejected (closed, with proof). Nothing can vanish quietly. ANVISA has 284 entrances on record today, and none is left open.
I'll admit the split into states only fully clicked for me while writing this article. I was speedrunning the build under a deadline, and I signed off on it while working at some ungodly hour with my batteries low. It wasn't negligence; it was budgeting: time and mental compute are finite, and I chose to spend most of mine on the foundations. It worked anyway, because the harness enforces the states whether or not I'm thinking about them. These days the agent remembers to check that things are in order more reliably than I do.
House rule 2: a menu, not a cage
Having Chris's blessing to work from the ground up gave me the freedom to question his structure, and some stages didn't add up. What went sideways: in the first version, a feasibility stage asked "can we build a scraper for this source at all?" before anything had looked at what our existing scrapers already do. That was the first lesson of the morning in the queue: the agent judged "can't get in" before checking what we already had, because if it had checked, it would have found a way in. So what was it basing its decision on? And why spend compute on an answer we might already have? It felt like reinventing the wheel, only to discover a nearly finished one in the garage that just needed a new tyre.
So I folded feasibility into Approach and changed the order: the agent now decides whether and how to build only after it has looked at what we already have, and "buildable" means every step has a known option. And it gets not just a menu, but a full-on smorgasbord: the options our working scrapers already use, at every step. It's told to mix and match whatever sugar, spice and everything nice it considers best to Frankenstein a working scraper.
The original order did have one advantage: it let the agent roam freely and come up with whatever it thought best, which might have produced new patterns and ideas we'd have missed if I'd left it feasting only on the existing ones. I'm a firm believer in "if it ain't broke, don't fix it", but I also know the only constant is evolution, and however good the existing human patterns are, even when they beat what an agent would come up with, they can always be improved. So I went for the best of both worlds: the agent sees everything that exists, and it's free to challenge any of it, add to it, or try something completely new. The rule, still in the harness today: challenge each choice before taking it; deviate when the improvement is real and you can name it, not to be novel. And it worked: ANVISA is the first of our scrapers to record what each page of a run returned, and to reconcile page by page. To sum up, if you feed a harness from existing human code, make sure it doesn't amplify human gaps, mistakes or limits.
Food for thought: a harness fed from human code inherits human limits. Show the agent what exists, and give it explicit permission to challenge it.
House rule 3: bouncers check evidence, and sometimes you need someone inside
What went sideways: that same morning held a second lesson. The agent's "non-viable" was a claim, and nothing checked it until a human looked inside. Later, a fix reported itself as applied while the code was still reading the old value: another claim that nobody had checked.
So now every door on the map has a bouncer, and I made sure they're strict ones:
- "Is your name on the list?" is the output checked against the spec.
- The dress code is the format check.
- "I don't care about your story" means every claim needs evidence.
- A fake ID is a check that passed when it shouldn't have.
- A bouncer on a smoke break isn't a yes: a check that didn't run never counts as a pass.
- And a bouncer who lets everyone in isn't a bouncer at all: a check you've never seen fail might not be checking anything, like a smoke alarm you've never heard beep.
The Approach door has three bouncers, all required, and I affectionately call them the club's Cerberus, or Fluffy. Most of the bouncers are small programs that check the agent's work automatically, and they come with tests of their own, to prove they really check (more on that in a future article). When a bouncer says no, we go back, find out why, and fix it. No getting in on fake IDs: we prove we're legit.
Key idea: bouncers that demand evidence. A gate shouldn't ask "did the agent say it's done?" but "show me that it happened". And a check that didn't run counts as a fail, never as a pass.
Some doors need a human too, in two different ways. The difference is who decides. When I approve, I check that the agent is doing what I need: that's a human bouncer at the door, like signing off the plan before any code gets written. When I'm its eyes, I do what the agent tells me it needs: that's the agent stuck at a locked door, asking someone inside to look, the rule that came out of that August morning.
Bouncers can only check the evidence they're given, so someone has to bring it. And I wasn't only a sensor for the website. Inside the rooms, I was a sensor for everything else the operation needed: Slack messages, Notion pages, meeting transcripts, messages to colleagues, the team's work on the data model for how we store and serve the acts. I verified things, told the agent where to find them, and was part of its memory.
House rule 4: every club needs a back door, and you should only have to say "friend" to open it
What went sideways: in October, on my second harness run for the ANVISA scraper, I hit a point where the build started to feel like patching too many holes in a sinking boat instead of sailing smoothly. That feeling was my trigger to stop and reconsider. And I had a clue as to what might have been off: the expert had produced the spec, our guest list, without ever flagging that the information was inordinately hard to reach, so if the agent was struggling this much, we were probably missing something important.
So I said we needed to hit the brakes and rewind, and the agent proposed a long, precise prompt for going back: act counts, file names, forbidden actions, expected output. It felt like reciting an elaborate Elven spell that no one would actually use, so I kept the magic in one sentence: "the current build is getting too complicated. did we miss anything? please back up the steps, find the origin, and see if we missed finding and exploring other options." A lightly edited version of it is now the first of three examples in the harness's instructions for going back. That's the second design decision behind the back door: it opens on a plain sentence, so nobody needs to know a command name.
I kept it plain because that's what a human is more likely to say. I wanted to test whether the agent could figure things out from a problem described vaguely, as a question based on a feeling, rather than being told exactly what was wrong and where. It was also just the truth: I didn't know the exact source of the problem. That's the purpose of the harness, for the agent to do things on its own. It fits the bigger experiment I ran with this harness: give the agent a clear starting point and a measurable goal, let it find the road there, and judge it by whether it arrives rather than by reading every step it took. That experiment is for another article.
Key idea: test the harness with the words a real person would use. If it only works with a perfect prompt, it doesn't work.
But a back door is only as good as what it undoes: once, to fix a single number, ours took a sledgehammer to three rooms and 90 of my rulings. That one gets its own story.
The club is a graph
Not every problem is a wrong turn, though. Sometimes a room itself has no way out. Draw the club as rooms, with the doors between them, and you get a map of every way the work can move: a graph. In a well-built club, every room connects to the rest. In August, while I was building the harness for ANVISA, five bugs were actually missing doors. Take one for example: the harness had a new side room, "waiting for a human to look", for when the agent hits an entrance it can't get through. A stage could pause the work in that room and ask a human to look, and the human could answer. But no stage was allowed to pick the work up again from that room, so everything just stopped. I added that room in my first version of the harness, and didn't connect it to every part of the harness that needs to know about it, like some sadistic Sims pool without a ladder, so the work could walk in and find no way out, turning a funhouse into a haunted one. The other four were the same mistake in different rooms. All five were fixed in one day, and model reviews are on record for catching two of them. The fix that would catch this whole class of bug automatically, a check that every room has a way in and a way out, has been designed but not built yet.
How the club got built
None of this was designed up front. It was built the way the thesis says: on a hard source, one version at a time. Chris built the club. His first harness already had seven stages, from the expert interview to monitoring, resting on one decision: with around 50 sources to onboard, split the work into stages with clear inputs and outputs, and have a human approve the plan before any scraper code gets written.
I was happy to see it, because I'd been itching to start working on a harness for a while. Running ANVISA through it was good groundwork: a chance to learn what the stages should look like, and how a scraper gets assembled, from someone who had already built a few. I was a baby in the world of scrapers, so it was a good starting point. As I ran it, I started changing it, beginning with a small program that checks each stage's report before the next stage starts.
Chris then restarted the harness in our production repo, around a machine-checkable answer key, an expert interview and a plugin so experts could run it themselves. He asked me not to be anchored by his experiments and to argue it from first principles. And so I did, happily hacking away over two more versions. In the first, I rebuilt the stages and added the review panel; in the second, I made the rules enforceable in code. Each version was rewritten by what ANVISA did to the one before.
One big design principle I followed was to have the agent help design the harness, instead of starting from a fixed idea of one. The thinking was that every regulatory source has its own bucketful of challenges, so the agents needed as much independence as possible to figure things out, and my job was to understand how agents work best: where they can and can't be independent, and how to guide them. I also wanted to draw on whatever the models knew about harnesses and coding practices that I didn't know existed. The field is new, I didn't assume I knew the best way, and I didn't want to produce premature answers that I'd have to pay a lot to dismantle later.
Key idea: let the agent help design its own harness. Don't start from a fixed picture of what the harness should be. Watch where the agent struggles, and turn that into structure.
Building the club while the party was already going on was cool, stressful and very interesting. Cool, because I could see the results of my prompts instantly and course-correct in real time. Stressful, because I didn't know when the harness would settle enough to produce one run from end to end that would lead to a working scraper. Very interesting, because I love architecting, and this meant getting to metacogitate: to think about how the agent thinks, and about how I think when I instruct it; to learn the cause-and-effect chains, the loops and the graphs in a harness; and to work out how to combine the freeform nature of natural language with enough structure to produce automation.
What's still being renovated
As of late October, the build is out for review: the scraper, the browser-service changes, the proxy change, the service that ingests the output, and the refreshed harness stages. The verification passes: the spec check, all 11 of 11 documents in the eval set, and the output check except for two findings. Those two aren't ours: Brazil's official gazette decided to quit on us right at the end, and has returned nothing for nine days, so a few original texts can't be linked. The scraper handled it the way an earlier ruling said it should: ship without the link, flag it, retry next run. That ruling was written before the outage happened. What's still ahead is turning the source on, with its weekly schedule and first full backfill, and running the last two rooms, Document and Monitor, for the first time.
Seven steps to steal
Each of these came out of ANVISA pushing back. In the order you'd build them, each with a sign it's working:
- Write the guest list first. A measurable answer key. You can tell pass from fail without reading the code.
- Give the agent a menu, not a cage. It reuses some solutions and improves others.
- Design every room by the action it triggers. No state where nobody knows what happens next.
- Hire bouncers who can say no, and say what they checked. You've seen each bouncer turn someone away at least once, and "didn't run" never reads as a pass.
- Build two human doors: judge and sensor. The agent asks you to look instead of inventing infrastructure.
- Build a back door that opens on plain words, and undoes only what the error touched. "This is getting too complicated" sends it to the right room.
- Check the club as a graph. Every room has a way in and a way out.
A good harness lets you stop micromanaging. Some nights I actually went to sleep while the agents built, knowing I'd be a little less behind on my work. One overnight build ran on its own for nearly five hours and finished eleven minutes after I woke up. Across the whole ANVISA run, three quarters of the agents' working time passed without a single message from me. Checking first thing in the morning came with fingers crossed, of course. It's like Christmas Eve: you hope you built Santa's sleigh well enough that Santa makes it to your house with presents. This time Santa made it, and that was a great marker for a harness doing its job.
But the partnership runs both ways. That was one of my theses going into this experiment, and it was satisfying to see it hold: trusting the agent with independence paid off. The agent found things I would have missed: it caught my spec contradicting my own notes about an archive, its reviewers caught missing doors in rooms I'd built, and it noticed our own menu was hiding a step. I caught things it got wrong: the wall it wanted to knock down with new infrastructure, and the sledgehammer it swung at three rooms to fix one number. So design your harness as a partnership, because that's what it's going to be: neither of us would have got ANVISA built alone.
And build your harness against your hardest source first: the easy ones will tell you it already works. See you at the club.