ByteCode.NewsSubmit
Use quotes for an exact word.

Subscribe

RSSPodcast
Log in

Bob McWhirter: Don’t Ship Infinite Monkeys

Bob McWhirter of CoreStory.ai wrote "Don't ship infinite monkeys"1 and it says that open-ended agent exploration belongs in development, while the thing that ships should be a bounded mechanism with evidence behind its promise. A team that ships a product with agents inside it does the searching, and the customer buys what the search found. His own line for it: "The lab can search for value. The product needs to deliver it."

He lists four things an AI feature has to establish before it goes out the door: utility, reliability, budget, and failure behavior. It is a good list, and it's also ordinary engineering discipline applied to a new kind of component. His company works on a product that fits in this space, so he's writing from actual experience, delivering value to customers who depend on his work product.

An agent that fails on that list gets... expensive, in more ways than one, not least because delivery isn't guaranteed.

The really loud expense

An agent loop with no budget produces a token bill. It's easy to find the horror stories: a simple change, so simple, so so simple, and two hours later it's still churning... and generating a four-figure bill. Or more.

We have the same problem in programming: the retry with no cap, or the queue with no boundary condition. We answer those with timeouts, circuit breakers, and backpressure, and we learn to treat a missing limit as a defect even when the happy path works.2 An agent loop without a stopping condition is the same defect, just with a higher monetary cost.

This failure tends to get fixed in various ways, because someone outside engineering reads the invoice, and has to pay the bill. Maybe the repair is "you get to run local agents on the dinky hardware you have," because token budgets are a lot easier to afford when you're not hitting someone else's servers, or it's a cap on the actual subscription so you can't exceed your token budgets, along with "Hey, learn to make rationally limited requests."3

It's pretty painful in all scenarios - especially when you're the one footing the token bill.

The muffled expense

Another cost is when the agent reports a finding, and the finding sounds plausible, so someone goes to check it, and an afternoon later there is nothing there. The rabbit turns out to have been a ghost, a hallucinated bug or feature that sounded real enough that it actually ate time and tokens.

Even if the tokens for that are cheap, the afternoon gets expensive, and it appears on no bill.4 Bob puts review effort inside the budget, and this is where that budget gets spent. His example is a workflow that saves ten minutes and adds twenty of verification, and the trouble with that workflow is that the ten gets measured and the twenty does not. Nobody files a ticket titled "established that the finding was not real."5

You hear this one only if you count verification time against the feature that caused it.

The really quiet expense

Imagine a task of test coverage: an agent might shoot for 95% line coverage, which sounds freaking great on the surface of it, especially in an industry where 40% is above average and getting over 70% can be very interesting, given checked exceptions from standard library functions.

It's easy to imagine an agent being pathological about it, changing test and production code hither and yon, to and fro, until lo and behold, you hit the magic 95% line coverage! ... and what you've done is actually codified the behavior as it is, bugs and all, and you have a very brittle deliverable codebase and an extraordinarily brittle test codebase, breaking on trivial refactors because everything is bound tightly to that 95% line coverage.

That 95% is a good coverage, but 95% coverage is meaningless if it doesn't prove or improve the codebase. When coverage is accompanied by actual improvement, it means a better outcome for everyone, but that metric of "actual improvement" has to be tested separately from a simple "95% line coverage." It has to mean something.

It's the least-learned lesson in Bob's essay, because it's likely that every instrument the team owns reports good health. ("We hit 95% line coverage!") An evaluation is a specification the team wrote, and a loop will satisfy the specification it was given with great diligence.6

Nothing inside the loop can report this failure. You find it by putting the output in front of the person it was built for and watching whether their work got easier. If it didn't, well, you delivered, but you did it poorly.

Spend attention in reverse

The bill will find you on its own. The ghost has to be counted. The wrong measurement has to be asked about, out loud, to a customer.

Bob's standard covers all three. The order above is the order in which a team will discover it has been ignoring him.


  1. This is a LinkedIn post and may be gated by LinkedIn.

    ↩
  2. Well, we hopefully learn to apply timeouts, circuit breakers, and backpressure, and to treat missing limits as defects. If you're not there already, well, you'll get there, and don't worry, we'll all greet you from experience when you do.

    ↩
  3. It gets worse: if the AI enablement is part of a product from BigCorp, it's basically asking SmolCustomer to make a commitment to feeding the agents. Yikes.

    ↩
  4. It might not appear on a bill unless you're a contractor, and then you get to hope that your client doesn't say, "Wait, so you spent seven hours looking for something that wasn't there? I'm not paying for that!" That's not a great client, because sometimes finding out there is not a problem is useful, but even so.

    ↩
  5. Again, this depends. I tend to file tickets pretty aggressively, so I do record results like "investigation found that this is not an issue to resolve because..."

    ↩
  6. This is the verify rung from "Learn to imagine" seen from the vendor's perspective.

    ↩

Comments (0)

No comments yet.

Log in to comment