The limit is per user per app. So you can have 1 million users, each with an Access Token and Refresh token, and they will all work fine, you could even get up to 50 tokens for each of those 1 million users and they will still all work. It’s just that no individual user of your app can have more than 50 tokens.
In most situations you’ll only need 1 access token and 1 refresh token for each user, and your app will handle refreshing it as needed.