I have not had this problem myself. And my test model somewhat matches what you describe.
If assets are locally on a private server then it’s not Twitch Caching them as a Twitch server doesn’t even exist in the chain. It really does sound like a problem in your browser here/your side, which is weird.
Make sure that you are clearing the cache for www.myExtensionAssetsServer.com not for Twitch.