Depending on how your viewers have the embed open, it could be the browser unloading the player. For example if a player is muted and not a primary tab Chrome will often unload the player (and going from ReRun to live could cause a period where the stream is offline, and so unless the user is actively on your page it could be the browser simply unloads the player until they visit the page).
Twitch don’t disclose how viewers are counted, all we know is that there is some protection against view bots, or viewers watching multiple embeds, not being counted as a viewer to prevent abuse. So depending on your site there’s the possibility of it running into tha, we simply don’t know the specifics of the system but it may be what is impacting viewer count when switching states.
But until then you could be negatively impacting your users. If you still wish to do this then I suggest watching some introductory JS videos, particularly on DOM interaction, so that you can learn about how to do this and the potentially issues associated with this approach.