Check for errors in the raw response, you are likely being blocked due to spam.
Whispers are problematic if a real person doesn’t initiate it via he website.
Bots should generally only reply to a user, and when they are reply-ing they should reply in the route as the request.
So a PRIVMSG is replied with a PRIVMSG and a whisper is replied to via whisper (and a notice on regular irc with a notice)
You shouldn’t (generally speaking) reply to a PRIVMSG with a whisper or send unsolicited whispers, which is likely the issue you are running into.
Additionally as per the docs
To send a whisper you need an oAuth key with the whispers:edit scope applied to it