If you are doing follower notifications you should be using webhooks instead of polling. You can subscribe to 1000 topics (likely channels in your case) per client ID which is enough for most users.
Edit: Looks like you can’t specify multiple to/from IDs like you can on other endpoints.