It's quite possible that the easiest way to achieve this would be to do it asynchronously using a scheduled batch job, it may be possible in a trigger but a batch could be less smart about what it's doing and as such would be easier to write, read and maintain.
Your batch could run on all overall rating records created in the last hour say, filter out multiples for each contact (keeping the latest for each of course!), and then update the related contacts accordingly if the rating for each is more recent than the current one.
The logic would be nice an simple compared to a trigger and you'd be far less likely to have to worry about governor limits. Note that you'd need a lookup field on the contact to the current rating object to make the date checking way easier.
I'd also suggest adding an optional switch member variable that you could use when running code manually to find all ratings created after date x, so that you can initialise the system to begin with.
Should you need to ensure that records are up to date immediately after being modified by a user (and you don't care about bulk data loads) you could write a trigger that uses the same logic (rip it out to another class to use in the batch & trigger!) if
trigger.new.size() == 1. This sounds hacky but it's a nice way to distinguish between a bulk load/update vs. a user using the system as expected.