Webhooks: How long should we retain events ids to prevent duplicates?

[zero]

@BarryCarlyon

The heat death of the universe!

My storage provider would be very pleased to hear that! Just kidding lol

The problem here is not just Twitch resending an event, but an attacker obtaining a valid payload/set of headers and spamming them at your server. Which is the true nature of a replay attack

Yeah, you’re right but that’s only true if you assume that after deleting them of your storage layer you would treat them as valid events. My idea was to consider invalid every event which is already processed or older than this maximum time. So you wouldn’t accept expired events.

It is not much different from how JWT works - signing with HMAC the JSON claims and timestamp (the same as the headers here) and considering older messages to be invalid/expired in a short time span just in case the claims have changed.

@BarryCarlyon

Like if you get a duplicate stream up/online for a stream you thought was already up then you do a check against the stream cache in your DB rather than a message ID compare for example. Similar for the subscription data. Can’t become a sub twice on the new sub feed within 28 days ish (weird math not persisting)

So in summary: do what works best for your application, the data it consumes from Twitch and what it does with it.

You might in fact not need to dedupe by ID at all in some cases. Theres a number of “what ifs” involved depending on what you do really.

Yeah I have thought of other approaches. I’m just weighing my options and thought that if the maximum time was short that would be the most stateless option and with less resources required.

@BarryCarlyon

Generally most of us run a rolling cache and sit on them for 24 hours or so. But I don’t apply this rule to certain topics due to the nature of the data meaning a replay occuring is irrelevant to my application and it’s state.

Yep, I spawn expensive tasks in response to these events (stream.online/stream.offline) and instead of dealing with event deduplication I could ensure my server to process those tasks one at a time per streamer and ignoring events with the same broadcaster_user_id. But because it is a multi-threaded paradigm the complexity would be higher, it would become stateful because I would need to store the state of those workers so I only accept one at a time instead of just spawning workers as events arrive, and then to be able to access this state across different threads I would need mutexes and/or other synchronization tools so I thought that maybe just deduplicating events would be easier. Also I’m using my own library for helix and if I move the event-deduplication part to the helix client I’m developing that could be reused in the future in other projects or even open-sourced, so that’s a plus.

Hmm, so my idea was maybe something like a redis cache for storing the IDs of those events (stream.online and stream.offline) and just set a TTL of 24h or something like that and also ignoring events with a timestamp longer than those 24h.

The thing is, I was hoping for twitch to specify a maximum time after which it never resends the same event instead of choosing blindly a number like e.g.: 24h. On second thought that could depend on the event but maybe the maximum of all the event types would be useful as a reference.