When the host starts, it starts for everyone at the same time, as while you may be 30s behind on the actual audio/video portion of the stream, the message that is sent from Twitch to indicate that the stream is over and is hosting another channel is sent outside of that audio/visual stream, so regardless of whatever delay there may be for the stream the host message isn’t delayed.
When the stream is loaded for the channel being hosted you’ll start with an empty buffer again, and which will increase as and when your connection needs to buffer which will start to add tot he delay.