As far as I’m aware, this happens because Twitch moves your stream to a transcoding server. Every viewer has to reconnect again to the stream once the switchover occurs. There’s no way to stop this unless you always have transcoding (i.e. become a Twitch partner).