If left alone, AI writes endless happy-path tests and is poor at testing access-permission boundaries. The following criteria are for stopping endless happy-path tests and, in order to avoid information leakage, finding critical boundary conditions and testing there. Follow them when writing tests, when deciding whether to add one, when fixing a failing test, and when reviewing.
Principle 1. Decide the line you want to protect, and write minimally only there
A test is both proof that the PR is correct and something that protects it from being broken by other changes. Decide the line you want to protect, predict where it will break in the future, and write minimally only there. If you write without deciding the line, most of the tests will have to be rewritten every time the spec changes, and mistakes made during that work will leak information, so the tests will end up effectively protecting nothing.
- Focus on places where you think "this might be broken by another feature addition." Through paths such as importing and using just one function from another file, depending on file load order or function registration order, updating a dependency library, or libraries conflicting with each other, features you thought were unrelated are in fact related somewhere
- Do not write tests for parts where it is fine if they break
Principle 2. Focus on testing whether information is leaking
Test not that a member can access, but that a non-member is denied access. Even when trimming test targets to prevent writing too many tests, write the test that access is properly denied, that is, the information-leak-prevention test.
A bug where someone with permission cannot access will be reported by that person saying "I can't view it." A bug where someone without permission can access is, in most cases, not reported. They either think they have permission and it just looks normal to them, or reporting is a hassle, or in many cases seeing someone else's private information does no real harm to themselves. Only after critical real harm occurs does the report finally come, and an apology press release gets issued.
- Identify the exact edge of the necessary line, and intensively test the dangerous side of it. Access-permission boundaries are not just one. If it is a page only project members can read, what you want to protect is the member versus non-member boundary, and you test that a person who is logged in but is not a member is denied access and gets a 403. Trying it with a person who is not logged in is wrong, and you have tested a different boundary, namely logged in versus not logged in
- Check the expected status code strictly. If a 404 is returned where a 403 is expected, the test should fail
- A test that only checks that 200 is returned to a person who has permission may still be returning 200 to a person who does not have permission, so it protects almost nothing. If you have the leeway to keep maintaining it and the leeway in test execution time, it is fine to write it
- Do not cover all combinations; write only the minimal set of permission boundaries. For reading a public project that anyone can read, one case where the lowest-privilege person passes is enough
- The implementation side, conversely, confirms that permission exists. The types of permissions will definitely increase in the future, and if the implementation confirms that permission exists, data will not be returned to people with the increased permissions, so it falls on the safe side
Lesson 3. Write data format tests and access permission tests separately
- For access permission tests, send a request to the controller and verify with the status code that is expected to be rejected. If you write it as a table of combinations of path, user type, and expected status code, you can verify it all at once. If maintenance is not too hard, you may enrich it with cases other than the exact boundaries. For APIs where permission is determined by what is included, such as lists and searches, check that things that must not be included are not included
- For data format tests, look at the method's return value. For simple APIs, you may test them all together
- For features that are not super important, do not write tests of the kind that check whether a field exists in the response. Once you start writing them, there is no end. Inconsistencies of the degree that you would notice if you were using it can just be fixed as soon as they are found
Lesson 4. Do not write tests that are a copy of the implementation
If you add normal-case tests endlessly, they become a mere copy of the implementation. When you change the implementation, you end up rewriting them twice, which only raises the cost of change and protects nothing. Before adding a test, consider whether it is proportionate to the amount of implementation and what would happen if there were a bug. If the consequence of the bug is on the dangerous side, such as information leakage or permission escalation, consider a test that detects that anomaly. If it is on the safe side with no real harm, such as less being displayed or a process not being called, do not write it. The following tests should not be written if they are on the safe side and are a copy of the implementation.
- Constant arrays, or,
arr.map(fn)unit tests of thin wrappers with no logic, like - A test that passes a fixture to a formatting function and checks a substring of the output
- A test that, when a schema validator is fixed, only checks that the validator is written correctly
- A test that, in a fix that merely removes a process call, checks that the removed process is not executed
- A normal-case test for "just-in-case regression prevention" to confirm that a bug fix is fixed. "If someone reintroduces it in the future, it can be detected" is a weak motive; confirmation can be done by running it locally
Lesson 5. Do not modify the implementation or the test mechanism for the sole purpose of making tests pass
When a test fails, do not fix the implementation or modify the fundamental policy or mechanism of the test for the sole purpose of making the test pass. When migrating the test runner, verify that the total number of tests does not change before and after.
Tip 6: For fixture pseudonyms, use the user's own handle
When writing placeholder email addresses or usernames in test fixtures or configuration examples, use the handle of the user who is instructing the work for the local part.taro Generic personal names like can cause problems if they match the account of a real person. If it is your own handle, then even if it exists in reality, it is you. For the domain, use .example or match an existing test.