No you don’t
You get a new access token, with the refresh token and your Client Secret. You can lose the access token and still get a new token, if you have the refresh token
POST https://id.twitch.tv/oauth2/token
--data-urlencode
?grant_type=refresh_token
&refresh_token=<your refresh token>
&client_id=<your client ID>
&client_secret=<your client secret>