Ronit Gupta’s team at the Flip the Switch Hackathon just proved that AI agents can do more than just write code snippets in an isolated IDE. By building Incident-Sentinel on the Switch platform, they created a collaborative SRE war room that slashed production incident resolution time from 45 minutes to a blistering 3 minutes. The architecture relies on three distinct AI agents working in tandem within a shared chat channel, ensuring that no critical action happens without explicit human sign-off.

The Nightmare Scenario Solved

The problem is universal for anyone on-call: PagerDuty screams at 9:42 AM because a checkout microservice is hemorrhaging HTTP 500 errors. In a traditional setup, ten engineers flood a Slack channel, manually parsing thousands of Datadog logs while debating which commit in version v2.14.0 broke the build. By the time a safe database rollback is approved, 40 minutes have vanished, costing SaaS companies significant revenue. Incident-Sentinel automates the tedious forensic analysis, allowing humans to focus solely on the final decision.

Architecture: Specialized Agents in Shared Context

The core philosophy here is treating AI as teammates, not chatbots. Instead of one monolithic prompt trying to do everything, the system uses Switch Rooms and Agent Templates to deploy specialized roles. The @sentinel-triager agent queries telemetry via FastMCP to identify error rates and correlate them with recent deployment SHAs. It isolates stack traces, such as NullPointerExceptions, providing the evidence needed for diagnosis. This shared context ensures all team members see the same data in real-time, eliminating the 'telephone game' of incident response.

Human-in-the-Loop Execution

Once the root cause is identified, the @sentinel-remediator agent assesses the blast radius, specifically checking for database schema compatibility. It then formulates a rollback strategy and posts an interactive approval card in the chat. This is the critical gate: the on-call engineer must explicitly type 'APPROVE' before the @sentinel-operator agent executes the rollback. After execution, the operator runs synthetic health probes to verify zero errors and automatically compiles a standardized post-mortem report, closing the loop with full auditability.

Key Takeaways

  • Collaborative AI outperforms isolated tools: Shared chat environments provide necessary context for complex troubleshooting.
  • Specialization beats generalization: Dividing tasks among triager, remediator, and operator agents creates more reliable workflows.
  • Human oversight is non-negotiable: Autonomous diagnosis and execution are powerful, but production rollbacks require explicit human authorization.
  • Reproducibility matters: The team built an enterprise infrastructure simulator using FastAPI, allowing users to test the entire incident response loop without live cloud access.

The Bottom Line

This isn't just a hackathon demo; it's a blueprint for the future of SRE. If you're still manually correlating commits with error spikes, you're leaving money on the table. The code is open-source at github.com/ronitgupta138/Incident-Sentinelβ€”go clone it and stop firefighting like it's 2015.