I don’t really know what “follower games” refers to, so please explain what that means so I can better help.
Working with IRC, in my opinion, has proven to be incredibly easy. Twitch has some quirkiness due to a host of features it does not support, as well as things liked cached joins, but it still isn’t bad. Working with Python sockets is painless. It is my language of choice as well, and I have not come across anything out of the ordinary.
If this is intended to be a multi-channel bot, you’re going to need to work around some Twitch limitations. 100 commands per 30 seconds per connection (for channels in which your bot is modded) is the main one. I would suggest making your bot a class, and spawn multiple instances of it as needed to work around limitations. Some people also use separate connections for sending and receiving.
My true “should you build it from scratch or not” answer can only come after you answer the first part of this reply, however. Good luck.