Cache subs list, then you can compare the user to that cache. If they subbed between the time you cached the data, and the time you performed the check then guess what, that subscription is an event in chat, and pubsub, (and in the future perhaps a webhook too), all of which you can add to your cache when it happens. You can also get a general idea of when to expect a sub to expire too, but a cache can just be refreshed. Also, if you’re connected to chat you can use userstate data to also help keep the cache up to date between refreshes but obviously this wont have full coverage.
Again, cache the list of followers. When a new user follows you can get that event from a webhook. If they try to unfollow and refollow to try trigger another alert, then you can refer back to the cache and see that they had already previously been a follower and no need to re-trigger any alerts. Sure you wont have precise timings on when someone unfollows, but do you have significant evidence that this is an issue for your app? Out of the 500 viewers you mentioned, how many have you seen unfollow during stream AND where it would be of significant impact to your apps functionality that it can’t wait until the next cache refresh?
Even for more important events where up to date information is needed, such as a follower only giveaway, you can use a cache (updated to new followers through the webhook) to limit entrants, and then when drawing a winner perform a one-off lookup on that specific user for their following status at the time of drawing a result.