Had the same issue using ZNC, and it was definitely IP caching. Switching to irc.chat.twitch.tv solved the issue.