The Mythical 10x Engineer

Mon, 12 Oct 2026 10:00:00 -0400

Intro

Pfft,

What a weird new environment we ended up in. The codebase sits at the bottom of a valley, ten years old, like an olive tree, aged and grown. But the ground around its roots is black and sticky. A tarpit. The tree is slowly being swallowed by it, and I suspect it has been for years.

Bobby: Oh no, are we going to maintain this till the end of its life?

No Bobby, it can barely stand upright.

Bobby: Let me try to pull out a module to make it a separate dependency. Darn, it took half the repository with it.

Yeah, it is a big ball of mud. If we keep on working on it we will sink into the tar together with the tree. We will be bogged down. Are there any tests?

Bobby: I found some tests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


def test_create_user():
 db = Mock()

 db.execute.side_effect = [
 {"id": 41},
 None,
 {"id": 42, "name": "Tommy"},
 None,
 ]

 create_user(db, "Tommy")

 assert db.execute.call_count == 4

 db.execute.assert_any_call(
 "SELECT MAX(id) AS id FROM users"
 )

 db.execute.assert_any_call(
 "INSERT INTO users (id, name) VALUES (?, ?)",
 (42, "Tommy"),
 )

So what does it test?

Bobby: User creation. Let me look at how it is implemented.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


def create_user(db, name):
 latest = db.execute(
 "SELECT MAX(id) AS id FROM users"
 )

 user_id = latest["id"] + 1

 try:
 db.execute(
 "INSERT INTO users (id, name) VALUES (?, ?)",
 (user_id, name),
 )
 finally:
 user = db.execute(
 "SELECT id, name FROM users WHERE name = ?",
 (name,),
 )

 if user:
 db.execute(
 "INSERT INTO user_events (user_id, event) VALUES (?, ?)",
 (user["id"], "created"),
 )
 else:
 db.execute(
 """
 INSERT INTO user_events (at, user_id, event)
 VALUES (?, ?, ?)
 """,
 (now(), user_id, "failed"),
 )

 raise RuntimeError("User was not created")

Right, so we don’t spin up a database? We just check how often we call db.execute? Who even dared to commit this?

1
2
3
4
5
6


test: increase coverage from 41% to 78%

Added tests for the database layer and failure handling.

assisted-by: Claude
committed-by: Tommy

Bobby looks impressed.

Bobby: What an impressive commit. Three thousand lines of code and a thirty-seven percent increase in test coverage.

Yeah, I suppose Tommy really was a 10x developer. A negative 10x. Tommy left before we arrived, but you hear the stories at the coffee machine. He pushed his PR, didn’t add any tests and merged. He didn’t even deploy it to RC for testing… They eventually had to require a PR review and disable the break-glass procedure to merge to main. For their own sanity. They were lucky he didn’t know about force pushes.

Bobby was already getting distracted by something else in the codebase.

Bobby: Wouldn’t it be more efficient if we fixed this loop? Someone put a lot of sleeps here. I’ll open a PR since it seems like a good improvement.

PR rejected: DO NOT TOUCH THIS

We see a man running at us, an old man with a long gray beard. He has probably worked here for the last 40 years.

Old man: How could you be so careless, trying to change things! Fixing this loop to be faster would have caused other scripts to fall behind and create race conditions.

Okay old man, calm down, we will leave this tree alone.

We can’t keep patching the old tree anyway. We’ve already seen what happens when we shake one branch. Another branch is propped up with an old piece of wood. I doubt it could even carry a strangler fig, and the tar would probably swallow the fig too.

So now we are here, to clean up this mess.

But where do we even start?

We need a plan. We need to understand the system.

Part I: Finding The Trail

Fortunately someone marked the trail before us.

Somewhere in the company is another codebase. It’s doing something different, but it was easy to work in. It was built by my old mentor, Mr. T.R., who was to me what I am to you now, Bobby. Small world: years after he taught me, he ended up at this company too. By the time I arrived he had already left again, but his code was still here.

Mr. T.R. had the habit of not chasing the latest and greatest. He had the habit of keeping things boring.

No elaborate architecture, no self-made framework, no layer cake of abstractions, no complicated validation library.

Just a simple for loop, a simple if, and raising an exception.

In the past I believed I could encode everything in the type system. FP, it was called. But soon I was no longer solving the business problem. I was solving the type signature puzzle.

As a junior like you, Bobby, I found his codebase too simple. But now that I have grown up, I appreciate his code more and more.

So let’s try to run his tests. And here is the first thing I always loved about his projects: it is one command. pytest runs the Python tests, and through a small plugin it also runs the JavaScript tests of the frontend. No README with twelve steps, no “oh, and for the frontend you also need to…”. One command, and you know whether the whole thing works. It sounds like nothing, but I have rarely seen it since.

Okay, we install the dependency tool and the libraries, and then we can run the tests:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


pip install
...
... # Okay, my attention span is running out,
... # Bobby: man, this takes forever
...
# okay, it's done, now to run the tests
pytest
test_invoice_totals.py V
test_invoice_rounding.py V
starting the database
running the schema migration
test_create_invoice.py V
shutting down database
frontend/test_payment_button.js V
starting the database
running the schema migration
test_payment_flow.js V
shutting down database
...

I don’t know what you will do, Bobby, but I’m going to grab a coffee while this runs. I remember that it takes at least 15 minutes. But it is worth the wait.

1

======================= 0 failed, 300 passed in 1023s ========================

Bobby: Right, let me fix that one annoying thing in the UI first. The payment button is blue, but the corporate branding guide says buttons should be magenta. And the guide says it should say “Pay now” instead of “Create payment”.

Bobby, it’s not even our project.

Bobby: The branding guide applies to all projects. It says so on page one.

Of course it does. And because he is Bobby, he does it properly: a full test run before the change, the change, and another full run after it.

17 minutes later

1

======================= 7 failed, 293 passed in 1023s ========================

Ah, it seems you got caught changing the behavior.

Bobby: Okay, let’s update those tests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


async function freshDb() {
 const container = await startPostgresContainer();

 try {
 await migrate(container);

 return {
 db: createDbClient(container),
 async cleanup() {
 await stopPostgresContainer(container);
 }
 };
 } catch (error) {
 await stopPostgresContainer(container);
 throw error;
 }
}

it("initiates payment when the button is clicked", async () => {
 const testDb = await freshDb();

 try {
 const invoice = await createInvoice(testDb.db, {
 customerId: 42,
 amount: 125.50
 });

 await clickPayButton(invoice.id);

 const payment = await testDb.db
 .select("*")
 .from("payments")
 .where({ invoice_id: invoice.id });

 expect(payment).toHaveLength(1);
 expect(payment[0].status).toBe("pending");
 expect(payment[0].amount).toBe("125.50");

 const auditLog = await testDb.db
 .select("*")
 .from("audit_log")
 .where({
 entity_id: invoice.id,
 entity_type: "payment"
 });

 expect(auditLog).toHaveLength(1);
 expect(auditLog[0].action).toBe("created");

 const button = screen.getByRole("button");
 expect(button).toHaveTextContent("Pay now");
 expect(button).toHaveClass("brand-magenta");
 } finally {
 await testDb.cleanup();
 }
});

Another 17 minutes later

1

======================= 4 failed, 296 passed in 1023s ========================

Bobby, you’d better limit yourself to only running the failing tests, otherwise we’ll be drinking coffee all day.

Bobby: Fine.

1

======================= 4 failed in 101s ========================

Bobby: Ah, come on. Four tests and it still takes almost two minutes? Why does every test need its own database?

Because we don’t want tests interfering with each other.

Bobby: Right. But why do we need a new PostgreSQL server for each one?

Good question. If we look at the profile, most of the tests don’t touch the database at all and finish in milliseconds. But about forty of them do, and every one of those first starts a fresh PostgreSQL container, which takes 20 seconds, and then runs the migrations, another 5 seconds. Forty times 25 seconds, and there go our 17 minutes.

That can’t be how Mr. T.R. left it. I remember him being proud of how fast his tests were. Maybe he left some kind of breadcrumb. Let’s look at the git history.

1
2
3
4
5
6
7


commit c41e07 TheCathedralBuilder
"use testcontainers: start a fresh container per test to fix state issues..."

...

commit 9a02cd Mr. T.R.
"reuse the database ..."

Of course. The cathedral builder. Mr. T.R. had already fixed it, and after he left, our favorite bureaucrat put a fresh container in every test, because things have to be the same everywhere, and every project in the company uses testcontainers now, by decree. The “state issues” in his commit message turned out to be one test that forgot to use its own database.

So what did Mr. T.R. do? Ah nice, PostgreSQL allows us to generate a ’template’ database upfront.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


let migrated = false;

async function freshDb(testName) {
 if (!migrated) {
 await createDatabaseTemplate();
 migrated = true;
 }

 const databaseName = `test_${testName}`;

 await adminDb.query(`
 CREATE DATABASE "${databaseName}"
 TEMPLATE test_template
 `);

 return connect(databaseName);
}

it("initiates payment when the button is clicked", async () => {
 const db = await freshDb("initiates_payment_when_button_clicked");

 // ...
});

Bobby: What is this arcane magic, TEMPLATE??

It is a PostgreSQL feature that allows you to create a new database as a copy of another one, schema and all. He started the database server once, ran the migration once, and then created a copy of the migrated database for each test.

That is a clever use for it. Let me quickly try it out. Oh, it allowed us to create databases in less than 100ms. So the tests were not slow by nature; it was the result of our favorite bureaucrat, who wanted things to be the same everywhere.

Bobby: I suppose this project was too fast in his opinion.

Now that I think of it, Mr. T.R. used to be able to tell quickly why a database test failed. Did you see the databases are not cleaned up? We can easily check what the state was in the failure cases.

Bobby, I think we’d better revert our changes. Since we will waste several days to get them merged.

Bobby: But the branding…

The branding will survive, Bobby. Better yet some cranky customers, wouldn’t be happy with a changed button. We still have to continue our migration, but let’s note down what we learned:

use a real database
have explicit dependencies
test the actual behavior
avoid mocks

To make our database tests go fast we should:

Reuse the database container.
Run migrations only once.
Create fresh databases from the template.
Leave the test databases around so we can inspect failures.

That saves about 25 seconds per database test, so the whole suite drops from 17 minutes to about a minute. Having such a fast cycle time makes it so much easier to maintain a project.

Bobby: Where did he learn this from?

I don’t know. But it seems he left another nugget before he left:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


## Testing strategy

The current integration tests are deliberately explicit.

This makes failures easy to understand, but there are
too many assertions describing the exact shape of the
database and API responses.

Things to investigate:

- database templates instead of migrations per test V
- snapshot testing for large response objects

Ah, snapshot testing. That would simplify the codebase even more. When was this committed?

1
2
3
4
5
6
7
8


commit b712aa
"farewell"

commit 8f31a2
"document testing improvements"

commit 9a02cd
"reuse the database ..."

And here the trail goes cold.

Part II: The Foot of the Mountain

So we went back to the valley, back to the tree and the tar.

For a few days we tried to apply what we had learned. Use a real database, avoid the mocks, test the actual behavior. But every time we pulled one root out of the tar, two others got stuck. To test create_user with a real database, we first had to untangle it from the user events, which were tangled with the billing, which was tangled with something nobody could explain to us.

One evening I walked to the end of the valley, mostly to get away from the smell of tar, and looked up. Behind the mud, where the valley ended, there was a mountain. I had never noticed it before. Its slopes were steep, the top was lost in the clouds, and if the old stories were right, from up there you could see past the valley. Maybe even see what the valley was for.

But we had no path, no map, and nobody to ask. Mr. T.R.’s trail ended at the foot of it.

And then, out of a clear sky, like lightning striking the villain at the end of an old film, the corporate overlords did something right. Not that I noticed at the time.

Ah, here we go again, the corporate overlords want to bring in another tool to make us go faster. Here the board is making a new announcement.

Board member: We are proud to announce our deal to bring the latest and best AI models into our company. AIchemy is the leading-edge model, and we predict that it will increase our productivity. It can answer any question and is highly advanced in its knowledge. We encourage you to try it out while the free credits last.

Yeah, “try it out”, more like we have to use it! All aboard the hype train, I guess.

Bobby: Oh, let me install it right away.

It is a command line tool. When it starts, it greets you with its logo, a little glowing ring.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


> AIchemy prompt
-*=:=-:=::+..=:=::*:-*:=*=-*
=%#:*+:*+-+-@@@+--*=*#-**::=
:+-:+=:#+:#%@@@%=-#--+::*::-
:=::-::+::%@@@@@@++::=::-:::
:-:::::-@@%@@@@@@@@:::::::*#
+:.:::::::::@%@::::::::::#+=
#-.:::.-@@@@@+@@@@%::=::=*++
:-.:+::%@@@@@@@@@%@#-=:-:.--
.==:*=:@@@@@@@@@@@@@+*:==.--
:-=-#++@@@@@@@@@@@@@%=:==::-
-#=:+:@@@@@@@@@@@@@@@-:+*--:
-=::=.-@@@@@@@@@@@@@::::====
:::.:::@@@@@@@@@@@@@.::::+++
::#::::@@@%@@@@@@@@@:::*-:-.
..*::::%@@@@@@@@@@@@:::=.%::
..--.-@@%@@@@@@@@@%@@.......
=+-*@%@@@@@@@@@@@@@@@%%..-.:

+ AIchemy, we have a legacy project at ~/projects/legacy
We are inspired by ~/projects/mrtr-project and would like to improve the testing of our project.
But how can we make this a swift improvement? We know the code is in a terrible state.
We want to avoid a slow migration, so give us an answer as if you were a 10x engineer.

AIchemy: Only a 10x engineer escapes the valley swiftly. You are not a 10x engineer.

Bobby: Rude.

Bobby wasn’t going to let that go. He leaned over my keyboard.

1

+ Who is this 10x engineer? Tell us everything.

AIchemy:

You ask how one might swiftly escape the valley.

Be warned. There are many fates.

Some engineers dwell in the Valley of Technical Debt until the end of their careers.

Others escape the valley, only to build another one somewhere else. They repeat the patterns they have never learned to escape.

But a few climb above the valley.

From there, they can see both the old land and the new. They ask not only what the old system does, but what it was meant to do. They learn the land. They study the roads others have walked.

Then they build.

And the new land must be easier to understand, easier to change, and better suited to the work than the old one.

The old tales call such an engineer a 10x engineer.

I know not the way.

I have but read the old tales.

Bobby: So where does this lead us?

Well, I suppose we’d best look at the commit logs, ask around and see how people use it? But are we really going to sit in meeting after meeting? Since management wants AI, we could use it. So here goes nothing:

1
2
3
4
5
6
7
8


AIchemy, look at the project history and tell us what this system
is supposed to do.

What was its original goal?

What does it do today?

And are there simpler or alternative ways of solving the same problem?

AIchemy: It is meant to be a CI system. But it became an automation system.

The engine is sprinkled throughout the codebase. It can only be tested as a whole, not as parts.

Its parts speak different languages and have different expectations.

There is tension. Different views. Different kingdoms.

The parts think they are separate.

They are not. They are a single monolith wearing many uniforms. If you seek to rebuild it, do not rebuild the kingdoms.

Conquer the lands around them.

Unify them under a single banner.

Give each part clear borders.

Then decide what deserves to remain.

That explains why we can’t really ask the other developers how things work: nobody sees the full picture anymore. Every team guards its own kingdom.

I keep thinking about Mr. T.R.’s project. We have seen what a codebase can feel like when it is easy to work in, and it is nothing like the tree in the tar. I don’t think we can get there by patching the tree.

Maybe we have to start fresh. Greenfield.

I say it out loud and immediately feel unsure about it. Probably every engineer who ever built a second valley started by saying exactly that.

Bobby: Isn’t that what it just warned us about? “Others escape the valley, only to build another one somewhere else.”

Maybe, Bobby. Maybe that’s what we’ll end up doing. But Mr. T.R.’s project shows it doesn’t have to go that way. And I don’t see another way up.

Bobby: So where do we start?

AIchemy: Take this ring, it will be your guidance.

Bobby points at the corner of the terminal, where the little ring icon is now pinned. I suppose that’s the ring. From now on, whenever we’re stuck, we can rub the ring. Well, type into it.

I close the laptop.

We need to understand what we’re actually trying to build.

And how do we do that?

One problem at a time.

We start walking.

Part III: The Ascent

Bobby, we will start from scratch. We pick the dependencies and technology.

Otherwise we might accidentally break the existing system.

We have to go fast. Faster than the existing system is expanding. Otherwise we will never catch up with it, let alone absorb the whole thing.

Dear reader, a quick word before we start climbing. This is a long climb, and if I describe every handhold this post turns into a book and you walk away halfway up the mountain. So here I’ll stick to the path. Each stretch of the climb gets its own post later in this series, where we dig into the details.

Bobby: Who are you talking to?

Nobody, Bobby. Let’s climb.

The first wall we hit wasn’t even the tests. It was the layers not agreeing with each other.

In the old system every layer had its own idea of what a build was. The database had a status column, the backend had a BuildStatus object with a slightly different set of values, the API returned its own JSON, and the frontend had a hand-written type that matched none of them. Bobby added a CANCELLED status to the backend on his first day, and the frontend happily showed “undefined” for a week before anybody noticed. Somewhere in the old code someone had once added "SUCCEEDED" next to "SUCCESS", because one layer said it one way and another layer said it the other way.

We didn’t want to carry that up the mountain. So I rubbed the ring.

1

+ AIchemy, how do we make the layers agree on the types to use?

AIchemy: The light source casts shadows in one direction. Do not mirror what is cast. Let the projection fall where it lands.

Bobby: What does projection mean?

Good question. Like a projector in a cinema? Or a shadow on a wall?

Bobby: A shadow, I think. You can’t paint a shadow and expect the thing that casts it to change.

Huh. That’s it. We have been painting shadows. Every layer is a hand-painted copy of the layer next to it, so every change has to be painted twice, and the copies slowly drift apart. That’s the mirroring. If I change the query, I need to change the row record. If I change the row record, I need to change the API, and then the client.

Bobby: So the frontend is the source! That’s what the user sees, that’s where the light falls.

No Bobby, that’s where the shadow lands. Although… you have a point that the frontend is the part that matters to the user. It’s just not the part we should be painting by hand.

What if we invert our design and start from the source? The database. We first write a query and its indexes, and then we know for sure what the return type is, and we can generate the database client from it. And the same trick works one layer up: some will first write an OpenAPI schema, but what if the server code itself is the schema? We generate the client from the server, and then it becomes very easy to find out whether the server and client disagree: the type checker tells us before we have run anything.

And that’s the part that surprised me. A type checker is the cheapest test we have. It runs in a second, it doesn’t need a database, and it checks every caller, including the ones we would never have written a test for. A whole category of tests we no longer have to write.

With the layers agreeing, we take Mr. T.R.’s database setup. Container started once, migrations run once, a fresh database from the template for every test. Now the database tests will not slow us down.

Then Bobby adds an age and a last name to the user. The branding guide wants “Full name” on the profile page, of course. And a dozen tests break, because every one of them checks every property of a user, one by one.

Why did we write our tests like this? Well, we ported them from Mr. T.R.’s project, and Mr. T.R. was a bit paranoid. Let’s look at one of them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


setup_user_db()
assert get_users() == [
 User("dike"),
 User("jane"),
 User("jimmy"),
]
delete_user("jane")

assert User("jane") not in get_users()
# But here comes the kicker:
# what if delete_user drops all users,
# or for some reason mangles jimmy and renames him to jimmy hendrix?

# so Mr. T.R. added this to prove that no other change was made by the query:
assert get_users() == [
 User("dike"),
 User("jimmy"),
]

I think I understand where the paranoia came from. His project did too many things. Invoices, payments, users, audit logs, all in one place. When a system does too many things, any change can touch anything, so you end up checking everything, every time.

What was on his note again? “Snapshot testing for large response objects.” Let’s rub the ring, hopefully it gives us some guidance on how to make this easier.

AIchemy: The snapshot runs the code and freezes what was once valid.

So we run the code, write the result to the test file, and keep it as the expected output of the test. Oh, this is handy. Remember that API call that had to match a format exactly, but we had no typed schema? Or right now, with the age and the last name, where we would have to update a dozen tests by hand?

1
2
3
4


setup_user_db()
assert get_users() == snapshot()
delete_user("jane")
assert get_users() == snapshot()

After running and accepting the updated results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


setup_user_db()
assert get_users() == snapshot([
 User("dike"),
 User("jane"),
 User("jimmy"),
])
delete_user("jane")

assert get_users() == snapshot([
 User("dike"),
 User("jimmy"),
])

So now we see that the delete operation has no effect on the other users! And when we add a field, we rerun, read the diff, and accept it, instead of editing a dozen tests by hand.

Bobby: But how do we know it doesn’t influence other tables?

Hmm, fair. It doesn’t tell us that.

Bobby: We could snapshot the whole database before and after, and assert that only the users table changed.

Okay, that’s doable.

Bobby: But then the application could still mangle something in memory, a cache or so. So we’d have to snapshot the memory as well. And the files it writes. And since we can never snapshot everything, we’d have to fuzz the inputs to find what we missed. And what if Tommy hid a date() in there that only renames Jimmy to Jimmy Hendrix on Tuesdays, or Brad to Brad Pitt? Then we need a linter that bans date() calls, and then there is random()…

Bobby. Stop.

Every step you just described costs more than the one before it and catches less. It is a wall that gets steeper the closer you get to it, a vertical asymptote. You never reach “total”, you just keep spending.

Oh snap, now we know where Mr. T.R.’s paranoia came from: the pursuit of total testing. In a perfect test environment all the side effects would be tracked, but we don’t live in one, and neither did he. So we stop somewhere. The code does what we expect, and the neighbors we can see in the snapshot are untouched. Bobby’s other tables stay a blind spot, and we write it down as one, so at least we know where it is.

Now the harder part. In our system, deleting a user also has to reach the billing system, and the billing system, being a telecom billing system, times out whenever it feels like it. So we retry: wait 10 seconds, then 20, then 40, and then we give up.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


async def delete_user(db, billing, name):
 await db.delete_user(name)

 for delay in (10, 20, 40):
 try:
 return await billing.close_account(name)
 except BillingTimeout:
 await asyncio.sleep(delay)

 raise BillingUnavailable(name)

If we test this as is, the test for the failure path sleeps 10 + 20 + 40 seconds. More than a minute, just to test this one failure path. And there will be more paths like it.

So how would we best test this?

My first thought is that we should drop the retry from the test. But then we push the retrying onto other parts of the system, and we are no longer testing the thing we actually run in production.

Let’s ask AIchemy for advice.

AIchemy: He who commands time does not wait.

Anything else you’d like to add?

AIchemy: I have spoken.

Well, so far so good. Let’s have a look at how others solve this.

Ah, Mr. T.R. solved it by mocking sleep. That works, but then there is the problem that other pieces of code check the time, and no real 10s have passed. So now we would have to mock the clock as well.

Let us have a look at how other languages solve this. Here is a little gem: Go’s testing/synctest.

The important part is that inside a synctest bubble the clock is fake: time.Sleep() doesn’t actually wait, time jumps forward as soon as every goroutine in the bubble is blocked, and synctest.Wait() waits until all those goroutines are durably blocked. So instead of actually waiting, the test can move time forward.

But can we do this in Python? It seems possible if we are using asyncio everywhere since then all code runs in a coroutine. But building something like that ourselves would take weeks, and we don’t have weeks.

So I suggest we delegate it to our new hire, Claude…

And then I hesitate. Wasn’t I making fun of Tommy and his AI-assisted commit a few weeks ago? I think there is a difference, and I hope it’s a real one. Tommy asked the AI to write his tests, so it wrote tests that test nothing. We ask it to build us a tool, a clock we can control, and then we write the tests ourselves, and we read what it built.

It takes an afternoon. And the trick turns out to be small enough to explain over a coffee. A sleeping task doesn’t start a real timer; it parks itself on a future and writes its wake-up time on a heap. The clock only moves when nothing in the event loop is runnable anymore and no query is still on its way to the database. Then it jumps straight to the earliest wake-up time and lets those tasks run. To know that nothing is in flight it peeks into the event loop’s selector, which is a bit naughty, but it works.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


async def test_delete_user_gives_up_on_billing(db, billing):
 clock = FakeClock()
 billing.always_time_out()

 with clock.patch_asyncio_sleep():
 task = asyncio.create_task(delete_user(db, billing, "jane"))
 async for attempt in clock.eventually(within_s=120):
 with attempt:
 assert task.done()

 assert isinstance(task.exception(), BillingUnavailable)
 assert clock.now == 70

Seventy seconds of retrying, in a few milliseconds. Honestly, I think this clock is one of the best tools we have built so far, and before AI I would never have found the time to build it.

Now our test scope increases. We don’t just control the code anymore, we also control the time within our application.

But then we hit another problem.

The database doesn’t agree with us.

We can move the clock forward inside our application, but when we ask the database what time it is, the database still thinks it is now.

Let us read the message from AIchemy again.

AIchemy: He who commands time does not wait.

Maybe we can command the time we give to the database as well.

We could make the timestamp an optional argument to the query. In production we don’t provide it, so the database uses its own time. In tests we provide the timestamp, and now the test can decide what time the query sees.

Bobby: But what if the time between the database and our system doesn’t match?

Oh Bobby, don’t you want to be done with the migration? We have NTP for this case, the paranoid can add an alert for when the time drifts.

We could probably make the database clock and application clock agree somehow. Or we could give each test its own database with its own notion of time. I’m not sure how we’d do either of those things, and both sound like another abstraction, another thing to maintain, and another migration.

Or we could just pass the timestamp to the query and accept the blind spot.

We don’t need to control every clock in the system to test the behavior we care about. We need to control the clock where it affects the behavior we are testing.

So the database keeps its real clock in production, but the test can explicitly provide a timestamp when it needs to.

Another dependency has moved out of the loop.

A week later Bobby spends an afternoon on a bug where the build page shows the oldest build as the latest one. The query used to return the builds sorted by date, someone dropped the ORDER BY because “the index already sorts it”, and on a small table that was even true. The code below it simply took the first row. None of our tests caught it, because with three builds in a test database the order happens to be right.

That kind of thing is hard to test. But it is very easy to check while the code runs:

1
2
3
4
5
6
7
8


builds = await db.recent_builds(repo)
if builds <= 1:
 return builds

previous = builds[0]
for builds in builds[1:]:
 assert previous.created_at >= build.created_at, \
 "recent_builds must return newest first"

Whenever we make an assumption in the code, like “this query returns the data sorted by column X”, we can pin it down with an assertion. It’s cheap, but it makes the program fail when the assumption is violated. It fails fast, instead of quietly doing the wrong thing and leaving us with a long investigation into why.

Bobby: So we just crash in production?

Yes. Loudly, instead of quietly showing the wrong build for a month.

Bobby: Maybe the users would prefer the wrong build over an error page.

Maybe. I’m honestly not sure. But I’d rather know.

Somewhere around here I looked back down the slope, and I noticed what we had been doing all this time without ever deciding to do it. The template database, the generated types, the snapshots, the clock, the assertions. Every single one of them made the time between changing something and knowing whether it worked shorter. Nobody had told us that was the goal. We just kept removing whatever made us wait.

But our deployment currently is, we run the gitlab ci, publish an docker image and a helm chart, we install this through gitops. Wait for the pods to come up. Submit a test build, and wait for that build to finish. The first time we did it, the pipeline took forty minutes. And the first time we did it, it failed at the very end, because one of the labels we generated for the pods contained a timestamp, and Kubernetes doesn’t allow colons in a label value. Forty minutes to learn that a string had a colon in it.

Bobby: We can cache it!

And he did. Docker layer caching in GitLab CI, the dependency cache kept between runs, the jobs running in parallel instead of one after the other. Forty minutes became fifteen. Bobby was very proud, and to be fair, he had every right to be.

But fifteen minutes is still a coffee. And the next time something failed at the end of the pipeline, I rubbed the ring.

Make the loop faster. I suppose that is what everybody means when they say “shift left”. But how far can we shift left?

Let’s ask AIchemy.

AIchemy: Left and right are relative.

Bobby: Relative to what? AIchemy: To where you stand.

BOBBY: But i stand in the middle?

Perhaps, you stand in the middle. Me too. Aren’t we all the center of everything?

Anyway lets try this once more.

1

+ AIchemy, our deploy pipeline takes fifteen minutes. How do we make it faster?

AIchemy: The swiftest pilgrim is the one who did not need to walk.

My first reading was: skip steps. Build only the image that changed, skip the test build when only the docs changed. That helps a bit. But the clue kept bugging me. A pipeline that doesn’t run is faster than any pipeline we could cache.

Why were we running that pipeline at all? Most of the times we pushed, we weren’t deploying to production. We were experimenting. We wanted to know whether our code worked inside the cluster, with the real network, the real permissions, the real Kubernetes. And for that, we were baking images, pushing them, and rolling them out, every single time, as if each experiment were a release.

So we split the two. For experimenting, we stopped building images altogether: a development pod runs in the dev cluster, wearing the same labels and permissions as the real service, and our code is synced into it every time we save a file. Seconds instead of fifteen minutes. Production keeps the full, slow, cached pipeline, and we make it boring on purpose. The 10x part happens before production; production itself should be the dullest part of the week. How that development pod works is a story for one of the follow-up posts.

With go it would be even easier

1
2
3
4


watchexec --restart -- \
 sh -c 'go build -o ./app . &&
 kubectl cp ./app dev-pod:/tmp/app &&
 kubectl exec dev-pod -- /tmp/app'

But boring takes more than a slow pipeline.

A week later the firmware team’s nightly build failed. It had run for an hour, compiled everything, run all their tests, and then, at the very end, when it tried to upload the artifacts, the upload token turned out to be expired. An hour of compute and an hour of waiting, to learn something we could have known in the first second.

Bobby: We could retry the upload.

Retry an expired token, Bobby?

Bobby: …Okay, no.

But it made me look at our own services, and they were no better. Our runner happily started without the registry credentials and only complained the first time it needed them. If an environment variable was missing, we found out when that one code path ran, sometimes days after the deploy.

So we turned it around. At boot, the services check everything they need: the environment variables, the database connection, the tokens. If anything is missing or invalid, they refuse to start. Kubernetes then never sends traffic to the new pods, the rollout stalls, and the old version keeps running. Rolling back becomes not rolling forward.

And since we are the CI system, we can do the same for the builds. An action checks its token before it starts the hour of compiling, and refuses to run if the token is invalid. Tokens can also expire halfway through, so the check is part of the action’s health check, and a build stops in the middle instead of finishing a calculation that nobody can upload.

Better a deployment that refuses to boot than one that boots and falls over at 3 a.m.

Then the firmware team asked us for a new check. From version 1.5 on, every build should produce an SBOM. Fair enough. I opened the code that decides which checks to run, and found this:

1
2
3
4
5
6
7
8


def required_checks(repo, version):
 checks = ["lint", "unit"]
 if repo == "firmware" and Version("1.1") <= version <= Version("1.4"):
 checks.append("legacy-signing")
 if repo == "firmware" and version >= Version("1.5"):
 checks.append("secure-boot")
 ...
 return checks

We had ported it from the old system without thinking, and it had already grown. Every rule about their versions lived in our code, so every new rule meant a change to our system, a test of our system, and a deploy of our system. For a rule that is really about their repository.

Bobby: I’ll just add another if.

Wait.

What if that knowledge moves to where it belongs? The firmware repo knows which checks it needs; it’s their code. So each repo carries a small file with the checks it needs, and our system asks the code: which checks do you need? An added bonus is that if a check gets updated, it can only be merged if the new check passes. You can’t update the CI and break the target repo anymore. And the version logic simply disappears, because the version is the commit. Branch 1.4 carries its own file, branch 1.5 carries its own.

Bobby: So we just pushed the work onto them.

Partly, yes. But they were already doing the work. Every time they needed a check, they filed a ticket with us and waited for us to deploy. Now they change a file in their own repo. I’m not sure every check belongs there, but these ones did.

And then something odd happened. We deleted required_checks, and its tests, and all the deploys it would have needed. Not faster. Gone.

The fastest loop we had found so far was one that never needed to run.

I didn’t know what to do with that thought. Every trick on the way up had been about making the work faster. This one made the work unnecessary. And if the best way to speed up some work is to not have to do it, then how much of what happens down in the valley needs doing at all?

I rubbed the ring one more time.

AIchemy: You have climbed high enough to ask. Go and look.

There was nothing left on the slope that we wanted to make faster. So we kept walking up.

Part IV: The Summit

We finally reached the top. But something was off. Normally a mountain is made of rock, but this one was made of dirt. The summit also seemed rather large for a summit.

There was a man sitting there in a suit, a bit worn at the elbows, looking out over the valley. Not the kind of suit you wear to impress anyone. The kind you start wearing when you stop writing code and start attending meetings about it. He turned when he heard us.

The man: You made it.

Bobby: Are you the 10x engineer?

He laughed, as if it was foolish of us to still believe in such a myth. He pointed down into the valley.

The man: This valley is something I started. I built part of it.

We looked down and saw a lot of people with shovels, all busy doing something. Some were throwing their dirt into other people’s holes, while others were moving it from one place to another and then back again. Somewhere in between, I could make out the old man with the gray beard, running from team to team, begging people to stop touching things.

There seemed to be a lot of work going on, but somehow everyone was slowing everyone else down.

The man: That section over there? Mine.

Bobby looked at the enormous hole.

Bobby: You dug that?

The man: Not all of it.

The man pointed again.

The man: That part was already there. I made it deeper.

We stood beside him for a while, looking down at the valley.

Far away, on the other side, I noticed someone else sitting on the edge, in a folding chair, looking at the same hole from a different angle.

The man: That’s our CTO. From his side it looks like a mountain range. He’s quite proud of it.

The man: Oh, and you really came on the right day. My request finally came through.

We saw a group of middle managers bringing in excavators.

Bobby, with your young eyes, can you make out the brands of those things?

Bobby: Anthropic, ChatGPT.

And the one that seems to be burying itself in the soil?

Bobby: Grok…

Bobby looked at the man.

Bobby: Why don’t you come with us?

The man smiled.

The man: Come where?

Bobby: Out of here.

He looked back at the valley.

The man: I like the view.

We looked down again.

From up there, the whole thing almost made sense. Every layer, every workaround, every compatibility requirement, every abstraction, and every team that had dug a little deeper while believing they were building a little higher. The man had once been one of the engineers who knew enough to see how the whole thing fitted together. Then he had become a manager.

Now he mostly watched.

“Doesn’t it bother you?” I asked.

The man: Of course.

“Then why stay?”

He shrugged.

The man: Someone has to watch the hole.

Bobby looked at me, and I looked back at the man. I thought about all the things we’d discovered on the way up, all the problems we’d managed to solve, and all the things we could probably still improve if we stayed.

“Come on, Bobby.”

Bobby: Where are we going?

“I don’t know yet. Let’s look around.”

We left him on his rock and walked across the summit. Now that we were up there, it was strangely flat, and strangely large. No rocks, no peak, packed dirt with the tracks of the excavators running through it.

At the far edge, where the excavators hadn’t been, a narrow path led away. Next to it stood a small pile of stones, stacked carefully, the way hikers mark a trail. Wedged between the top stones was a faded sticky note.

farewell — T.R.

Bobby, I think we found a way out of here.

We followed the narrow path away from the summit.

Somewhere in the valley, AI was generating code. They were digging faster now.

After a while, I looked back.

From here, the shape of the landscape was different. What had looked like a mountain from below was now a vast depression in the earth, surrounded by land stretching out towards the horizon.

I stopped.

“Bobby, it was never a mountain.”

He looked back.

Bobby: What do you mean?

“It was a quarry. We’ve been climbing out of a hole.”

Bobby looked towards the world beyond it.

Bobby: So there’s a whole world out there?

“Looks like it.”

He was quiet for a moment.

Bobby: So perhaps the 10x engineer is somewhere out there.

“Perhaps.”