This lesson's interactive features are locked, please to keep using them

AI and Test Economics

Writing tests with AI is way cheaper than writing them by hand, which changes the ol' "How many tests are enough tests" debate in an interesting way.

These days, you can be more aggressive about adding targeted tests because the cost of writing them has come down. But "cheap to write" does not mean "generate 1,000 brittle, low-quality tests."

Good tests communicate expected behavior. You still don't want redundant, brittle, or meaningless tests just because someone told you 100% test coverage is a "best practice."

That said, if there's undertested code that's been running correctly and deserves to be covered by tests, it's great to get the agent to help you increase the test coverage.

Should I Keep AI-Written Tests?

I've had people ask me:

"My agent makes a lot of new tests for each change. Should I commit them all? Or let the agent discard them after it ensures the code works?"

Some developers assume that all correct tests are keepers. But I actually have a higher standard:

If I'm confident that the current expected behavior of the code will be around long-term, I keep tests. Good tests help prevent regressions.
If the behavior is likely to change as the code is refactored, extended, or deleted, or as business logic changes, then I'll discard those tests after validating that the code works. Less code to maintain and change later.

I do this because I don't want thousands of tests for specific behavior that happens to be correct now, but that will likely get respecified later. For example:

A function that extracts email addresses from PDF files will likely always do that, and keeping tests for that saves me work later.
A function that sends a specific greeting message to new Discord users is very likely to change as the marketing team experiments with different messaging, so keeping tests for that creates more work later.