According to the documentation RecentlyViewed
LastReferencedDate - The timestamp for when the current user last viewed a record related to this record.
LastViewedDate - The timestamp for when the current user last viewed this record. If this value is null, this record might only have been referenced (LastReferencedDate) and not viewed.
System is behaving correctly. Opening a contact record doesn't update LastReferencedDate of Account.
I have written following query to verify this LastViewedDate and LastReferencedDate
SELECT LastViewedDate, LastReferencedDate, Account.Name, Title, Email,
WHERE Id = '80090000000WeV1'
When I open the contact detail then above query will return
Now, from the Contact detail, if I click on Account record link and go to the account detail and fire the same query. you will see Account LastReferencedDate is updated where as Contact's LastViewedDate and LastReferencedDate are remain same.
Now, from the Account Details, I reach the Contact related list and open the contact record and fire the same query. You will see there are no change in Account's LastReferencedDate but contact's LastViewedDate and LastReferencedDate are updated.
Finally, as per documentation, A record is considered viewed when the user sees the details associated with it, but not when the user sees it in a list with other records.
Updated based on comments: When LastReferencedDate and LastViewedDate can be different?
From the Contact Detail, change the account from the lookup and save the contact. At this time, Account's LastReferencedDate and LastViewedDate will be different.
Also, without changing Account/Contact record, you can create a Case and select the account from the lookup and save the case. Then also, LastReferencedDate will be different from LastViewedDate.
Conclusion: Any change in relationship record will update LastReferencedDate of the Master (e.g. Account). It can be either creation of other related records from Master.