If I understand correctly, the buffer increases if your connection is unstable. In other words, if you are buffering the player will increase your delay to accommodate the instability of your connection. You might also seek to lower the quality if you continue to have issues.