Twitch Chat (from a Twitch Dev)

[night]

@brildum

And just to provide some more context, we saw a 5x increase in IRC authentication / JOIN attempts, over a 2 minute period which lasted for more than an hour. We were able to drop those numbers over the 2hr downtime with some changes to our servers and clients which allowed the system to recover. We’re still trying to determine exactly what caused the increase in traffic.

I can’t really prove anything, but I would guess the extended length of outage might be caused by clients trying to reconnect (bots run a ton of connections + on-site users), while the initial spike could have been an incoming attack. If that’s the case, the earlier part of this thread where rate limits are first discussed comes to mind (since they are really the only way to prevent the total crash of the chat system). Perhaps a blanket rate limit on incoming connections and joins per server (regardless of IP) would prevent this from happening. Rate limiting is definitely an area of chat that needs work, probably sooner rather than later. If you get any progress on this front and need third party testing I’m sure myself and/or moocat would be willing to lend a hand.