The JOIN event often is delayed, by, from what I have seen in the bot I develop on from 1-5 minutes (usually closer to 1-2 minutes though, sometimes longer). The LEAVE event can be similar as well, so that sounds normal to me. There is an unsupported API that you can use to get a list of chatters, assuming it is still available, I haven’t tested it in ages:
https://tmi.twitch.tv/group/user/usernamehere/chatters
Do keep in mind that is cached and even that API will never be a real-time representation of who is in chat, more than likely.
I do not think the API sucks as it mostly follows the IRC RFC standard and is pretty spiffy in the way that it maintains compatibility with a great number of IRC clients and is fairly easy to interface with.