IncidentWatch spots ticket spikes before internal alerts fire. IncidentSweep clears the related backlog in an approved batch.
When a product incident occurs, the support queue can be one of the first places it shows up. Customers start reporting the same issue, sometimes before engineering alerts fire or anyone has declared an incident.
Those reports keep coming while the incident unfolds. At first, support may not know what is wrong or have an answer we can confidently provide. By the time the issue is resolved, hundreds or thousands of customers may be waiting for a response. Support has to track those customers through the incident, then make sure each gets the right answer when it is resolved.
We use Grok Bot to handle both parts of this work. It provides us with AI teammates that take ownership of operational jobs, particularly repetitive workflows that otherwise fall between people and systems. As we shared in a recent blog, the SpaceXSI customer support team uses Support Bot in Grok Bot to handle customer-facing communication.
Now we’ve added two more AI teammates that we call on specifically for incidents. The first, IncidentWatch, monitors the support queue for spikes in similar tickets and alerts our team when it sees one. The second, IncidentSweep, handles the other end, identifying affected tickets and closing them in an approved batch.
During one incident on September 18, our Bot, IncidentSweep, cleared 204 tickets in about 10 minutes with a support engineer reviewing and approving the batch. Previously, we would have had to reply to those tickets one by one over the course of hours or use a bulk macro that gave us much less control over who received the message.
Detecting incidents through customer reports
Customers often notice problems before our internal alerts fire. When several customers report the same issue at once, those tickets can be an early signal of an incident.
IncidentWatch is a Grok Bot teammate we built to detect that signal. Every five minutes, it reviews the previous 15 minutes of tickets for spikes around the same problem. If the pattern does not match a known incident, IncidentWatch alerts the support engineer on call with the apparent issue, number of matching tickets, and examples to investigate.
A support engineer decides whether the reports point to an incident and works with engineering to investigate. In the meantime, IncidentWatch holds tickets that clearly match the issue.
Once the team confirms the incident, we post it to our public status page. Support Bot, our primary customer-facing Bot, returns to the held tickets, checks them against the incident, and responds to customers whose reports match. As the incident progresses, it can return with updates.
Clearing the incident backlog
Once an incident is resolved, there may still be hundreds of related tickets in the queue, and each needs a final response. Traditionally, support teams have had two ways to clear them, and both have drawbacks. They can respond to each ticket individually, which can take hours, or use a bulk action, which is fast but leaves little room to catch a ticket that should not receive the message.
We built IncidentSweep to combine the control of the first approach with the speed of the second.
A support engineer starts by telling IncidentSweep which tickets to consider, which to exclude, and exactly what message to send. IncidentSweep finds the matching tickets, reviews each one, and sets aside any it cannot confidently connect to the incident.
Before sending anything, IncidentSweep shows the engineer the proposed batch, the tickets it included, and the exact message they will receive. The engineer can spot-check the tickets and make changes before approving the send.
IncidentSweep then sends the approved message in small batches and records each action. It never adds tickets after approval, and if anything fails, it stops and reports what happened.
Give Bots jobs, not tasks
Grok Bot introduces a different way of working with AI. Instead of turning to an AI tool for help with individual tasks, teams can create AI teammates that own jobs and carry entire workflows through to an outcome.
Customer support has been a natural place for us to put this model to work. IncidentWatch and IncidentSweep take on jobs that used to require someone on the team to watch, sort, and act on hundreds of individual tickets. Across SpaceXSI, we’re looking for more workflows where AI teammates can take on the same kind of ownership.