Exactly what @losfarmosctl is saying.
I assumed it, because you didn’t complain about twitch not deleting them but instead trying to solve this “issue” by yourself.
If you stream and chat on a site that is available to everyone, then there is no privacy. If anyone can sign up to see “private” content, the privacy-factor would be zero.
I think the best solution would be to just hide everyone that is not actively chatting from the chatterslist #DontCallTheLurkers