Server Sends a Response
After the request is verified, Salesforce sends a response to the client. Token responses for the OAuth 2.0 SAML bearer token flow follow
the same format as authorization_code flows, although no refresh_token is ever issued.
Note: A SAML OAuth 2.0 bearer assertion request looks at all the previous approvals for the user that include a refresh_token.
If matching approvals are found, the values of the approved scopes are combined and an access_token is issued (with "token_type" value "Bearer"). If no previous approvals included a refresh_token, no approved scopes are available,
and the request fails as unauthorized.
If there is an error in processing the SAML bearer assertion, the server replies with a standard OAuth error response, including an error
and an error description containing additional information regarding the reasons the token was considered invalid. Here is a sample
HTTP/1.1 400 Bad Request
"error_description":"Audience validation failed"
Refer this for Detail Answers and Search for "Using SAML Bearer Assertions" this heading in pdf