Yes.
Should you do it for a server based on Follows? Probably not, as griefers would be able to connect to easily.
Alas I’m strapped for time outside of paid work.
But the end points for follows is documented here:
And in the “subscriber” method you’d just swap the call to the subscribers endpoint to the follows.
And you’ll also need something to collect relevant names to whitelist (Yggdrasil username in the case of Minecraft)
You may struggle with un-whitelisting on unfollow due to the often large number of pages to read to get all followers.