We can only test classes with mock callouts but cannot do a callout directly from a test class. (If we do test class fails). Is there a reason?
The reason is, as any good unit testing framework would require, to ensure that the unit tests are not reliant on any external resources. Unit tests should be entirely self-sufficient.
You don't want your tests failing simply because the external system that your callout invokes is temporarily unavailable from the Salesforce org, for example.
The unit test is testing your code, not the external resource.
UPDATE: As commented by sfdcfox, you also don't want your unit tests potentially modifying live data (if your callout effects change in the external service).