No. 118Product strategy

Ironies of Automation

Your agents took the easy part of the job. The part they left is harder than it looks.

Here is a job posting at an invented company:

Senior Engineer, Agent Review

Our engineers' coding agents open about 60 pull requests a day. We're hiring someone to make sure they're right.

You will:

- Review agent pull requests across billing, permissions and onboarding. - Catch changes that conflict with product decisions, including ones that were never written down. - Take over when an agent gets stuck, in any part of the codebase. - Keep the review queue under one day.

You have:

- Five or more years writing production code in our stack. - Deep knowledge of our pricing, permissions and customer contracts.

You won't spend much time writing code. The agents do that.

Read the "You will" list as a set. It is everything the agents couldn't be given: the judgment calls, the stuck cases, and the decisions nobody recorded. The "You have" list asks for years of the work the job no longer includes.

A psychologist described this job in 1983.

The tasks the designer can't automate

Lisanne Bainbridge, of the Department of Psychology at University College London, presented "Ironies of Automation" at an IFAC/IFIP/IFORS/IEA conference on man-machine systems in Baden-Baden in September 1982. A revised version was published in Automatica (vol. 19, no. 6) in 1983. The paper is mainly about control in process industries, with examples from flight decks. The paper defines an irony as a "combination of circumstances, the result of which is the direct opposite of what might be expected."

The first section, "Introduction," starts with the designer. "The designer's view of the human operator may be that the operator is unreliable and inefficient, so should be eliminated from the system." The paper finds two ironies in that attitude. One is that designer errors can themselves be a major source of operating problems. The second is the one this post is about: "the designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate." That approach, Bainbridge writes, "means that the operator can be left with an arbitrary collection of tasks, and little thought may have been given to providing support for them."

The leftover tasks fall into two general categories. The operator may be expected to monitor that the automatic system is working correctly, and if it isn't, to call someone more experienced or take over. Both turn out to be harder than they look. Quoting Bibby and colleagues, the paper notes that "automated systems still are man-machine systems," and draws the irony "that the more advanced a control system is, so the more crucial may be the contribution of the human operator."

The monitor's impossible task

Taking over is hard because skills decay. Bainbridge writes that "physical skills deteriorate when they are not used," so "a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one." Knowledge fades too: "efficient retrieval of knowledge from long-term memory depends on frequency of use," and the knowledge an operator needs for unusual situations "develops only through use and feedback about its effectiveness." The paper adds a worry about the future: automated systems monitored by former manual operators "are riding on their skills, which later generations of operators cannot be expected to have."

Monitoring is hard in a different way. Vigilance studies show that even a highly motivated person can't keep effective visual attention, for more than about half an hour, on a source where very little happens. So monitoring for unlikely problems is "humanly impossible" and has to be left to alarms. And Bainbridge points out that "the operator will not monitor the automatics effectively if they have been operating acceptably for a long period." Even forcing attention fails: operators can be made to keep a log, but "people can write down numbers without noticing what they are."

Then comes what Bainbridge calls "a more serious irony": that "the automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively." Bainbridge names two problems with this. The first is about knowledge: "In complex modes of operation the monitor needs to know what the correct behaviour of the process should be," as in batch processes where variables have to follow a particular path over time, and "such knowledge requires either special training or special displays." And when the computer decides faster and on more dimensions than a person can check, the operator can only judge at "some meta-level" whether its decisions are "acceptable." The section ends with a question and an answer. If the computer is making the decisions because "human judgement and intuitive reasoning are not adequate in this context, then which of the decisions is to be accepted? The human monitor has been given an impossible task."

The final irony

The section "Approaches to solutions" asks what the operator needs. For rare problems that must be noticed quickly, the operator "must be given artificial assistance, if necessary even alarms on alarms." The automation itself should make failure visible: "automatic systems should fail obviously." And if a person has to monitor how the computer decides, then "it is necessary for the computer to make these decisions using methods and criteria, and at a rate, which the operator can follow." Otherwise an operator who doesn't believe or agree with the computer "will be unable to trace back through the system's decision sequence to see how far he does agree."

The paper also takes on written procedures. They "cannot cover all the possibilities, so the operator is expected to monitor them and fill in the gaps." In Bainbridge's words, "it is ironic to train operators in following instructions and then put them in the system to provide intelligence."

The paper's best-known line comes at the end of that section. "Perhaps the final irony is that it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training." The next section opens with a line that sums up the paper: "By taking away the easy parts of his task, automation can make the difficult parts of the human operator's task more difficult."

Coding agents and the leftover job

The mapping that follows is ours, not Bainbridge's. The paper is about process plants and cockpits, and a pull request is not a reactor. But the job posting has the same shape as the operator's job in the paper.

The coding agent is the automatic system. It does the part that could be specified: writing the code, running the tests. What it leaves is the arbitrary collection: the product questions in the ticket that nobody answered, the pricing rule that lives in one person's memory, the change that is technically correct and wrong for a customer contract. Those land on the reviewer, who is asked to monitor a system that writes code faster than they can and to judge, at a meta-level, whether 60 pull requests a day are acceptable.

Bainbridge put the monitor's knowledge problem in "complex modes of operation," with batch processes as the example. Product review is one of those complex modes, in our reading. Whether a change is correct depends on much more than whether its tests pass, and much of that correct behaviour is a set of product decisions: which plans get which limits, who can see what, what was promised to whom. If those decisions are in people's heads, the reviewer has to already be one of those people. The posting asks for exactly that, "deep knowledge of our pricing, permissions and customer contracts," because nothing else would give it to them.

Our essay on the modern product development lifecycle describes where the work now gathers: "The work now piles up at the same review gates, waiting on people." We've also argued that you can't review your way out of a context problem. Bainbridge explains why the gate is so hard to staff. The better the agents get, the rarer the bad pull request, and the harder it is to keep watching for it.

Where Brief fits: giving the monitor something to check against

Brief is the product strategy agent. The homepage says it "captures decisions where they happen, connects them into one graph, and acts on them." In Bainbridge's terms, it is meant to give the person left watching the agents what the paper says a monitor needs: knowledge of what correct looks like, an alarm for the rare problem, and a system that fails obviously instead of quietly.

The correct behaviour is written down. Bainbridge said that in complex modes of operation, a monitor's knowledge of correct behaviour "requires either special training or special displays." For product decisions, a record is our closest equivalent. In the Brief app, recording a decision requires both What was decided? and Why? before Record Decision will save it, and a decision can also carry Tags and a Re-evaluate when condition. "Annual plans get no discounts, because discounting trained customers to wait for the end of the quarter" is something a reviewer can check a pull request against without having been in the room.

The record is read from where decisions are made. The posting asks for someone who already knows the contracts and the pricing history, because that knowledge is scattered. The homepage says Brief "reads decisions from Slack, Jira, Confluence, and GitLab, self-hosted included." A rule settled in a thread or a ticket comment can reach the review without the reviewer having seen the thread.

Agents raise the alarm before review. Bainbridge called monitoring for unlikely problems "humanly impossible" and said a rare event that must be noticed quickly needs "artificial assistance." Coding agents set up with Brief search the recorded decisions before they start work, and ask Brief to check significant changes with brief_ask in check mode, which compares a planned change against existing decisions and flags conflicts. An agent about to add a discount to annual plans can meet the decision before the pull request exists, so the reviewer isn't the first and only check. People can run the same check with brief ask --mode check.

The leftover questions get an owner. Bainbridge's operator inherits whatever the designer couldn't automate, with "little thought" given to support. In the app, Propose turns an open question into a pending decision and can name Who should decide?, which records who the question belongs to. With the user's approval, an agent can record a pending decision too, and agents can call brief_list_pending_decisions to see what is waiting. The question the agent couldn't answer becomes a named item, not a guess buried in a diff.

Agents fail obviously. "Automatic systems should fail obviously." When a decision is pending, an agent can hold instead of guessing. The homepage shows an agent asked to add per-seat pricing to the checkout API learning that D-241, the seat-based pricing decision, is pending, and replying: "Holding the pricing change until D-241 is decided." A hold is a visible stop. A confident guess that passes review is the quiet failure Bainbridge warned about.

Decisions can be traced back. Bainbridge wanted the operator able to "trace back through the system's decision sequence." Each recorded decision keeps its Why?, and when a decision changes, its History shows its status changes. A reviewer who disagrees with an agent's change can see which decision it relied on and whether that decision still stands.

None of this removes Bainbridge's irony, and the paper wouldn't expect it to. Bainbridge asked who notices when the alarm system itself isn't working, and the honest answer here is that Brief only knows the decisions a team records. When nothing has been recorded, check mode finds no conflict and the agent goes ahead. The reviewer is still needed, and still needs to know the product. What Brief changes is how much of that knowledge has to live in one reviewer's head, and how often the agent's guesses reach review unflagged.

Bainbridge suggested, with a "perhaps," that the final irony is that the most successful automated systems may need the greatest investment in training the people who step in. If your agents are getting better every month, what are you investing in the people reviewing them?

Objections

Objection 1

What is "Ironies of Automation"?

"Ironies of Automation" is a 1983 paper by Lisanne Bainbridge, published in Automatica after a 1982 conference presentation. It argues that automating industrial processes "may expand rather than eliminate problems with the human operator," because the operator is left with the tasks the designer couldn't automate, including monitoring the automation and taking over when it fails.

Objection 2

Who wrote "Ironies of Automation"?

Lisanne Bainbridge wrote "Ironies of Automation" while at the Department of Psychology, University College London. The paper draws on research into how human operators control industrial processes and aircraft, including Bainbridge's own earlier work on how operators keep track of a process's current state.

Objection 3

What are the main ironies of automation?

Bainbridge's main ironies are that designers who try to remove the operator still leave the operator the tasks they can't automate, that the operator is asked to monitor a system installed because it does the job better than they can, and that skills fade when they aren't used. The paper's final irony is that "the most successful automated systems, with rare need for manual intervention," may need "the greatest investment in human operator training."

Objection 4

How do the ironies of automation apply to AI coding agents?

Coding agents automate writing code, and the engineers who review their work are left with what the agents can't do: judging whether changes fit product decisions, catching problems in work that is usually fine, and taking over when the agent is stuck. To review well, they need to know what correct looks like, which for product decisions often isn't written down.

Objection 5

How does Brief help reviewers of coding agent work?

Brief records product decisions with their reasons, including decisions it reads from Slack, Jira, Confluence and GitLab. Coding agents set up with Brief search recorded decisions before they start work and use check mode to flag conflicts in significant changes, and open questions can be proposed as pending decisions that name who should decide. Brief only knows the decisions a team has recorded, so reviewers still need to know the product.


Previously
No. 117

Hidden Hierarchy

Your team says anyone can decide. Someone is still deciding, and your agents can’t see who.