In my bot, for limited use, I do sleep a second between calls, to be safe, when trying to rebuild follower lists. However, with the number of calls you are doing, you may want to do it more efficiently. I assume you are doing something with the data after you retrieve it from Twitch, possibly some JSON extraction or something and then placing into a database.
Did you consider taking the system time, in milliseconds, after the call to Twitch, then again after you have performed any data operations, and then sleeping for the difference (or not at all if the data processing takes longer than a second)? That could at least reduce your “downtime” that is incurred with sleeping if you are processing data, or doing some other unit of work, in between calls to Twitch.
I am sure too, if you took all the correlated data, that you would find some users that overlap from the channels and then reduce your queries some, I don’t know by how much, but I would think some.