I know this was an older question, but you kind of can with a formula field...
We had a need to limit visibility to a "Reserve" field on our Opportunities to only the record owner, and the Roles above. Another wrinkle was to only allow the record owner, or manager to update this "Reserve" field, but not peers
the solution I came up with was to create a formula field, "User Visible Reserve" that displays the value if UserID = OwnerID, else display $0, as well as a new field called New Reserve.
I created a workflow that would take the "New Reserve" and pass it to the "Reserve" field IF UserID = Owner ID, and a Validation Rule that would display an error if there was not a match
I also used 2 Page Layouts, one for the Reps and one for Managers. The Rep layout did not have the "Reserve" field only "User Visible Reserve" and "New Reserve" The Manager page layout had all 3
Although a determined Rep could create a list view or report to see the "Reserve" field of Opps owned by a peer, those permissions could be revoked, and this provides a decent solution for our business