Who Guards the Guards? - Finding Code Bugs in Your Tests with Static Analysis

Testing is a critical component of software development, yet even test code itself can harbor subtle bugs that undermine its effectiveness. Igor Blanchi, a static analysis expert at SonarSource, presented three common pitfalls in JavaScript unit testing that developers frequently overlook. These issues range from incomplete assertions to confusing error messages, and they can leave test suites vulnerable to undetected failures. Understanding and addressing these pitfalls is essential for maintaining robust test coverage.

The Problem of Incomplete Assertions

The first and most critical pitfall is the incomplete assertion, where developers write expectations without actually asserting anything. In this scenario, a test might contain expect(someFunction()) but fail to chain an assertion method like .toBe(), .toEqual(), or similar. This creates a test that appears valid but never actually fails, regardless of the function's output. Blanchi demonstrated this issue using real-world examples from major projects like Kibana and Angular, showing how surprisingly common this problem is across codebases. To fix this, developers must ensure every expectation includes a complete assertion statement, such as expect(result).toEqual(expectedValue).

Argument Order and Error Message Clarity

The second pitfall concerns the order of arguments in assertion methods. While seemingly cosmetic, providing the actual computed value first, followed by the expected value, significantly impacts debugging efficiency. When arguments are reversed, error messages become confusing—displaying "expected 2 to equal 3" instead of the correct "3 expected to equal 2." This small mistake can waste considerable time during debugging, especially in complex codebases. Blanchi illustrated this issue with examples from real projects and emphasized that correct argument ordering provides clearer feedback when tests fail, making it easier to identify whether the test expectation or the implementation is at fault.

Missing Assertions and Exception Handling

The third pitfall involves tests with no assertions at all, often occurring when testing exception handling. A test that only executes code without asserting outcomes provides zero meaningful validation. This commonly happens when developers forget to assert that an exception was thrown or that specific behavior occurred. The proper approach is to wrap calls in expectations and explicitly assert whether exceptions should be thrown or caught, using framework-specific methods like Chai's assertion documentation suggests. Without these assertions, tests pass vacuously, creating false confidence in code quality.

Static Analysis as a Prevention Tool

Rather than relying on manual code review to catch these issues, developers can leverage static analysis tools integrated into their development workflow. SonarLint, available as an extension for Visual Studio Code and other IDEs, provides real-time feedback on test code issues with squiggly line indicators. For teams using continuous integration pipelines, SonarQube (on-premise) or SonarCloud (cloud-based) can automatically analyze and report these problems during builds. These tools scan for incomplete assertions, incorrect argument ordering, and missing assertions, allowing developers to fix issues immediately in their editor or before code reaches production.

Key Takeaways

• Incomplete assertions are a pervasive problem where expect() statements lack assertion methods, causing tests to never fail
• Argument order matters in assertions—actual values should precede expected values to generate clear, helpful error messages
• All tests require assertions—tests without assertions provide no meaningful validation and should be fixed or removed
• Static analysis tools like SonarLint and SonarQube/SonarCloud automatically detect these test code issues in real-time and during CI/CD pipelines
• Proper exception testing requires explicit assertions that exceptions are thrown or not thrown, rather than relying on implicit test failure