There is no way to achieve the desired result declaratively.
Field level security cannot be changed on a per record basis. A user has the given level of access to said field on all records they can access.
The closest I could imagine would be to put the sensitive data in a child object, and use apex managed sharing to grant access to that child record to the owner + hierarchy (the hierarchy bit w managed sharing might be tricky - that is not my expertise).
Instead of managed sharing, you could just use process builder to sync the parent record owner w the child record. Then you can set the org wide defaults on child to private, and check the share w hierarchy option, which would share these records up the role (not manager) heirarchy.
Another option could be using a visualforce page or component running in system mode to determine how to display that field. This however would not allow for reporting on that field.
Also note that formula fields don't evaluate field level security for the fields they reference. So a formula field could determine if running user equals owner, then display the value, else blank. This would w the VF option, or w the child record option if you add logic to write the value from child to the hidden field on parent which is referenced by the formula field.
One final note - grant access using hierarchies grants access via the role hierarchy, not the manager hierarchical relationship field on user. Manager sharing is now supported declaratively, but last I looked it had to be set per user, and couldn't be set to share all records of said object w the record owners manager.