I’m seeing the exact same behavior and I think Hephaestus is correct as well. At least that was my guess too.
What I’m doing to handle this is creating a local cache with an age. Removals are only removed after five minutes. This keeps oscillations from making my bot go crazy.
– edit –
If you want to get an idea of how I did it take a look at the Followers class in my Twacker text-to-speech chat client.