I haven’t done anything with whispers yet myself, as my bot hasn’t needed it, but I did some research on this a few months ago, and at the time, I believe there was a different cluster of servers for whispers than from normal chat (I think the “group” chat servers?).
But I think this all changed since the switch to AWS. Can anyone else on this thread confirm/deny?
If that’s not it, I also have a link to this thread for whitelisting a bot account for whispers, to avoid being spam filtered. The thread ends with a link to a form for submitting accounts.
Good luck!