Timeline

Key events in chronological order:

Screenshot 2024-11-27 at 10.21.29 AM.png

Screenshot 2024-11-27 at 10.25.01 AM.png

Root Cause Analysis

It appears the Tightknit app has been responding to a significant portion of events it has received with a HTTP 500 response.

The following is the response from Slack support regarding the why events were automatically disabled for the Tightknit app:

These Events would have been shut down because we were sending you events, but your App was not acknowledging receipt of them. If we do not get acknowledgement from your App that it is receiving what we're sending, then we will eventually stop sending them.

This applies to any App wich receives more than 1000 Events per hour, and the shutdown occurs if less than 5% of those payloads are acknowledged correctly.

This requirement is described here: https://api.slack.com/apis/events-api#error-handling

It looks like we were receiving a high volume of HTTP 500 responses from the Events Request URL for your App when we attempted to send Event Payloads to it in the lead up to the Events disablement.

I can't see when exactly these HTTP 500 Responses started, but it seems that we've been getting them at a fairly high rate for at least the past week until it eventually surpassed the threshold of less than 5% of the calls being successful. In the last hour before events were disabled there were 5969 Events dispatched to your App endpoint with just 276 of those were successfully acknowledged with a HTTP 200 response.

We realized this was being driven by a combination of the following:

Impact Assessment

Affected customers: All Slack workspaces with Tightknit app installed

Service degradation: The Slack API Events which the Tightknit app specifically subscribes to were lost during this time. This does not include commands, shortcuts, or interactive actions (e.g. clicking a button, submitting a modal). The full list of subscribed events includes: