Refresh token doesn't always work with API v3

Why does token refresh doesn't always work? We always have to reauthorize.

When a token expires and then we try to refresh it, we always get the below error and mind you, it is the same refresh token given to us when we requested for an access token.

{\"error_description\":\"unknown, invalid, or expired refresh token\",\"error\":\"invalid_grant\"}\r\n."}