The era of convincing management to invest in automated testing is over; the new battleground is trust. With AI generating tests rapidly, flakiness has emerged as the primary threat to CI/CD reliability. A recent project by developer m4rri4nne addresses this by combining a custom scoring engine with the Strands Agents SDK and Google's Gemini model to not just detect flaky tests, but diagnose their root causes.

The Scoring Engine Mechanics

The core of the flaky-test-detector project is a deterministic Python engine that parses JUnit XML reports. It groups test outcomes by commit SHA, flagging any test that yields both passing and failing results for the same code base. The FlakinessScorer class calculates a score by dividing the number of flaky commits by the total commits analyzed, ignoring skipped tests and those without commit metadata. Tests are ranked based on a configurable threshold, defaulting to a score of 0.1 or higher across a minimum of five commits.

Agent Integration with Strands SDK

Moving beyond simple detection, the project integrates an AI agent built on the Strands Agents SDK. The agent, configured to use gemini-2.5-flash, is equipped with two custom tools: fetch_json_report to retrieve historical data and export_analysis to save findings. The system prompt instructs the agent to analyze error messages for patterns like timeouts, race conditions, or network instability, providing a confidence level for each inferred cause. This transforms raw data into actionable insights, such as identifying that 'ConnectionError: connection reset by peer' likely points to gateway instability rather than code logic errors.

Practical Deployment Strategies

The tool is designed for GitHub Actions but includes a workflow that runs weekly, generating reports as artifacts. The author suggests running the analysis weekly for continuous deployment teams or per-deployment for monthly release cycles. While the AI analysis is optional, it serves as a first-pass triage, allowing human engineers to focus on genuine infrastructure or code issues. The JSON output format ensures the data is machine-readable, facilitating future integrations with visualization tools like Grafana to track flakiness trends over time.

Key Takeaways

  • Flaky tests are defined as inconsistent outcomes on the same commit, eroding trust and hiding real bugs.
  • The tool uses a simple ratio (flaky commits / total commits) with a default threshold of 0.1 to flag issues.
  • Strands Agents with Gemini 2.5 Flash analyzes error messages to suggest root causes like race conditions or timeouts.
  • Integration is currently optimized for GitHub Actions, requiring specific commit_sha and run_id properties in JUnit XML.

The Bottom Line

Automated testing without automated diagnosis is just noise. This project proves that even a lightweight agent can turn flakiness from a blocker into a debuggable metric. This is the kind of pragmatic AI application that actually ships value: it doesn't replace the QA engineer, it just stops them from drowning in false positives.