For this to be viable for a large number of viewers/channels, you need to cache the follower list.
For example, for each channel your bot is on you could poll through the followers endpoint to get a full list of followers which you would store in a cache. Then when you award points, you only need to compare the user to the cache you have rather than making any queries to Twitch to check that.
To keep the cache up to date you could use Webhooks to receive notifications of new followers which you can then add to your cache. Periodically you’ll also want to query Twitch for the total number of followers and compare that to your own cache so you can check if there are any users who have unfollowed as the Webhook intentionally doesn’t notify of unfollows.
This isn’t 100% perfect, there may be times when your cache is inconsistent at the time of awarding points, but there will always be an inherent margin of error in systems like this, but with appropriate monitoring of the cache and updating as needed (and within the rate-limits) will help to keep the error rate to a minimum, and in the event of an error I don’t think awarding greater points to someone who isn’t actually a follower at that point in time isn’t a big deal.