Has anyone used AES256 to encrypt and store Highly confidential data in force.com Database. What are the steps to follow? Help me with some insights into private/public keys and certificates, which of these are necessary? Is browser support essential?
There are no public keys in AES encryption. You can store the key inside your code or within a custom setting, but you should make sure this key is not generally available to be viewed. In a managed package, your code will be obscured, and the key will be hidden, but if you want to increase security, make it a custom setting and have the key automatically generated upon installation. The custom setting should be set to "protected" so the key will not be exposed to the installed organization.
For additional information, consider reading the article Secure Coding Storing Secrets, available on the DeveloperForce Network. As stated in the comments, you may also choose to use Encrypted Text Fields, which automatically apply a mask of a chosen type and are automatically encrypted and decrypted by the system. Apex Code will always see the decrypted version, so one should take care not to expose the stored secret. Users with the View Encrypted Data permission will be able to see the decrypted values in reports and elsewhere, so Encrypted Text Fields must be properly secured through profile permissions.
All encryption and decryption are handled by the server, so browser support is irrelevant. There are no certificates to deal with, since it is simply a private key. By default, all connections to salesforce.com are TLS secured, preventing casual viewing during transit, and is theoretically safe from "man in the middle" attacks, thanks to the security measures of TLS. No public keys are used while encrypting, storing, or decrypting the data.
In general, the following steps will occur:
Honestly, encryption within the platform is dead simple, but the trick is to make sure the keys are protected.