Verify the committee example: working can still be wrong
Good to have you back. In the specifications lesson, we wrote down what the committee report must do, and worked out its expected results by hand, before any code existed.
Now we use those answers. In this lesson, we'll:
- read a program written from the spec;
- compare what it shows with the answers we worked out;
- run the same checks on the first program;
- decide how much checking a task needs, and when to do it.
Why it matters: checking is how you decide whether to accept the work, or send it back.
Read the program written from the spec
An AI wrote this program from the committee spec, for this lesson. Let's check it.
# Committee report written from the specification.
# Amounts recorded as received this month.
received_a = 1000
received_b = 1000
received_c = 500
# Each member is due 1000 this month.
due_each = 1000
members = 3
total_received = received_a + received_b + received_c
total_due = due_each * members
still_due = total_due - total_received
print("Received:", total_received)
print("Still due:", still_due)
Most of it uses what you already know. The blank lines are only there to make it easier to read. Two things are new:
-means subtract, sostill_dueis the total due minus the total received.printcan show more than one thing. With a comma between them, it shows the text in quotation marks first, then a space, then the value.
First, check the spec's boundaries by reading. The program uses only sample amounts. It works out totals and prints them. Nothing in it changes or deletes a stored record. You can download this program too.
Do the results match your answers?
Here is what the program shows for each criterion:
| Criterion | Member C paid | Expected (worked out first) | What the program showed | Result |
|---|---|---|---|---|
| 1 | 500 | Received: 2500, Still due: 500 | Received: 2500 / Still due: 500 | Pass |
| 2 | 750 | Received: 2750, Still due: 250 | Received: 2750 / Still due: 250 | Pass |
| 3 | 0 | Received: 2000, Still due: 1000 | Received: 2000 / Still due: 1000 | Pass |
For rows 2 and 3, the only change was the number on the received_c line. Row 3 is the criterion the last lesson asked you to write. If your answer was different, work through it once more: 1,000 + 1,000 + 0 is 2,000 received, and 3,000 − 2,000 leaves 1,000 still due.
The program passes every check we decided in advance. That's real proof, and it's the kind that counts, because the expected answers came from the committee's needs, not from the program.
What happens when the first program faces the same checks
The first program, from the Code Literacy lesson, printed one line: 3000.
Against criterion 1, it fails twice over. The number is the total due, not the total received, and there is no "Still due" line at all. Changing Member C's payment makes no difference to it either, because the program never reads what anyone paid.
In the Code Literacy lesson, reading the lines showed us the hidden decision. Here, comparing the output with the spec's checks shows it too. Reading and checking support each other. Each one catches things the other can miss.
What passing checks don't prove
Three passing checks don't prove the report is right in every situation. They prove it's right in those three.
Try a situation the spec never mentioned. Suppose Member C pays 1,500 by mistake. The program prints:
Received: 3500
Still due: -500
Is "Still due: -500" correct? The program did exactly what its lines say. But should the report show a negative amount, show 0, or warn the organiser about the overpayment? That's a question for the organiser, not for the code. The right move is to add the answer to the spec, add a check for it, and only then change the program.
This is what verifying means: checking that the work does what was asked, both by comparing its results with answers decided in advance and by reading how it does it. For the "never change a stored record" boundary, we read the whole program: it only does arithmetic and printing, with no instruction that writes a file or contacts another computer. For a bigger change, you would also look at which files changed and which commands ran.
Decide how much checking a task needs
Not every task needs the same care. Before you hand a task to an AI, ask three questions:
| Question | For our report | For an app that messages real members about their payments |
|---|---|---|
| How reliably does AI do this kind of task, measured on cases with known answers? | We checked one program on three inputs. That tells us about this program only, not about how often AI gets tasks like this right | Unknown until measured on many similar tasks, and many more parts can go wrong |
| What does checking its work cost? | A few minutes, with three criteria | Much more: records, permissions, messages and recovery |
| How bad is a mistake? | Small: sample amounts on your own computer | Large: wrong amounts sent to real people about real money |
How far a mistake's damage can spread is its blast radius. Your three answers tell you how much to hand over, and how much checking it needs. A small task that's easy to undo may need a quick check. A change that touches real records may need you to lead every important decision. Handing over the writing still leaves you responsible for accepting it.
Check at set points, with checks the AI can't change
Checking doesn't mean watching every step an agent takes. It means stopping at set points, such as after the spec is written, after each change, and before anyone relies on the work, to run checks that the agent can't edit.
Acceptance criteria give you the expected answers. Keep them somewhere the agent can't change, or compare any edits against a copy you control. A test that runs those checks automatically is the same idea, written as code. If the agent can edit the test, check that change too: an edited test may pass and still no longer check what you intended.
An agent may finish a task by reporting, "All checks passed." Treat that sentence as a claim to look into, never as proof. Look at which checks ran, what they expected and what actually came out.
And check during the work, not only at the end. If an agent had said at the start, "I'll show the total due", that was the cheapest moment to catch the mistake.
Safety floor
Whatever the AI writes for you, it is not finished until you have checked it yourself. The AI can test its own work, but its tests can miss things, and it often sounds just as sure when it is wrong. And if something breaks later, "the AI wrote it" is not an answer anyone will accept. The work is yours.
Check what you can now explain
Without looking back, try these:
- Member C pays 250. Work out both lines the report should show, before reading on.
- Explain why the first program fails criterion 2, even though it never shows an error.
- Name the three questions that decide how much checking a task needs.
- An agent says, "All checks passed." What do you look at before you believe it?
For question 1, the program shows Received: 2250 and Still due: 750. Compare it with your own answer only after you've written yours down.
In the next lesson, we'll put specifying, building and verifying together, and give the whole way of working its name.