Not sure what you are using to sign/verify and not sure if this is even related, but some libraries, such as the popular jsonwebtoken do not use a base64 encoded secret by default.
“…there are other libraries that expect base64 encoded secrets (random bytes encoded using base64), if that is your case you can pass Buffer.from(secret, ‘base64’), by doing this the secret will be decoded using base64 and the token verification will use the original random bytes.”