If the buffer keeps hitting zero for you (a buffer empty), then it implies your internet speed is not high enough to download the stream at it’s current bitrate. Unfortunately, the stream bitrate is set by the broadcaster and Twitch can’t change it for every viewer, leading to your situation.
Quality options (transcodes) are our current method of trying to deal with this issue, but giving a stream quality options is extremely expensive for us, so we’ve been unable to offer it to all channels. However, we are always trying to give more streams quality options, and hope that even small streams will benefit from them in the near future.