Do I always need to have total on the object or can I create a
non-grouped-by report and have a calculated field?
Report formulas only work on aggregates, including SUM, MIN, MAX, and AVG values. If you tried a report formula for your example, you'd end up with something like
SUM:Quantity * AVG:UnitPrice, which would obviously lead to incorrect results unless you grouped by UnitPrice.
If I need to put it on the object, is there any reporting tool used
where I can base a report off of a SOQL procedure?
You do need to put a field on the object for per-record calculations. You may need to build a "Custom Report Type" in order to get additional fields/objects than what standard reporting allows. Finally, you can also go to the Developer Console, click on the Query tab, and type in a direct SOSL or SOQL command and get output, which I often use to get meaningful statistics without the hassle of a UI. However, larger queries may time out, as the console only waits a few seconds before declaring "no response from server." It depends on the size and complexity of the object and query in question.
Or how can I get more robust reporting in SalesForce?
If you still can't get the desired results using custom report types, joined reports, or the Developer Console, then you'd have to turn to alternative reporting engines, such as Crystal Reports (standard disclaimer: I do not endorse any specific products, and have never used nor been employed by the linked company). You may also find that simply exporting to Excel may be beneficial in some cases, using a combination of the Excel Connector and pivot tables/charts/etc.