A true rollback doesn't necessarily exist in Salesforce. Unfortunately, there are going to be certain aspects of the process that need to be a manual step as they do not currently exist in the API (Approval Process for instance).
However, there are a few steps you can take to mitigate problems you may run into. Obviously, you want to thoroughly test your entire application before you put anything into production. This is also true for your deployment. It is a good practice to do a dry run deploy before you do your final push to production. Make sure everything is fully working and tested in a Sandbox (preferably a recently refreshed Full Copy). It is important to note that once you do your refresh to the Full Copy sandbox that all Prod changes should be stopped. If they are absolute, must-have changes, these changes must also be done in the Sandbox. The goal is to make your sandbox replicate production to find any issues.
Once everything is fully tested, you need a backup of the previous version of the metadata. This can be done a few ways. If this is a long standing application, you may have a tagged version available. If you have an extra sandbox slot available, copying to a developer/config-only sandbox will store the meta data (this would be my preferred approach as not everything is available via the metadata API). You could also do a manual backup just pulling down all of the data using Ant and the Metadata API. Next, you need to make sure you back up your data. You can use the Data Management > Data Export functionality available through the Admin for a weekly export, or you can use the Data Loader. Either way, you are going to want to do a backup of your data.
So, you have everything backed up, but currently you don't have any way to really get the system back into a state that it previously was in. You can now make a destructive changes package file that you can run using Ant to remove all of the recently entered Metadata. Unfortunately, this is just going to remove things and not revert changes you made back to their original version. If you need to compare Metadata versions, you can use a piece of software like SnapShot. Unfortunately, this is all still a very manual process. You need to keep a strict log of all of the changes you are making to revert anything you do if needed.
Salesforce also provides a rollback of data only (NOTE: This is data only. Not Metadata!) for a fee for a 24 hr period as well.
So, the moral of the story is you really don't have a great way of doing rollbacks in Salesforce. At least not in the general sense of normal application development (like replacing a WAR file for a Java app). It is crucial to test heavily well before the production deploy. Regression test everything and keep a good set of unit/integration tests. Good luck!