As Oleksiy says, you can't create formula fields in tabular reports.
You should be able to use cross-object formula fields on your target object
For example, if object
Foo__c has a lookup to
Bar__c, then you can create a formula field on
Foo__c, let's call it
quantity__c * Bar__r.price__c
TotalPrice__c in the report.
If this doesn't work for you, another option is to use an AppExchange Product like Conga Composer where the results from one or more SFDC reports/queries can be combined together and manipulated with excel formulas/pivots before presenting to the user. I've used this in the past where one SFDC report simply isn't sufficiently expressive to bring all the different data sources into one nice single sheet presentation