How do you determine which mod is responsible for it? Some channels have dozens of mods, not all of them are going to be online at the same time, so if there’s an issue with an app the broadcaster may not even be able to find out which mod authorized an the app at fault (or even if it is an app, and not just a compromised mod account). It would require the broadcaster unmodding everyone to kill all permissions those mods may have passed on to 3rd parties, which is not an ideal workflow during a live event.
You’re still giving the app a token to access the API. While the design of the app may be to only do something when a moderator presses a button, there’s nothing technically stopping apps from doing whatever they like with a token after you’ve given them it, even while the mod isn’t there. And there are also apps that don’t require user interaction at all after the initial OAuth process.