It's more beneficial to think of API version as an all-or-none upgrade. Even though all code should be interchangeable, in reality, things tend to break if you have version skew. For example, some versions of JSON encode null values, others don't. Some fields available in 28.0 and above will crash code that's version 27.0 and lower, in some cases (such as this relatively famous bug). Some objects aren't available in some versions of the code.
It's better to be at one solid version (e.g. all code at 24.0), rather than having code that varies wildly from 20.0 to 34.0. Avoid blindly updating everything without first researching all breaking changes. Make a backup of your code and/or plan to revert everything if you decide to upgrade and things break. Make sure you have unit tests for all your classes, and that those unit tests run at the same version as your classes.
On the other hand, if something works the way you expect in code that's versions older than what you have, then leave it at that version until you need to make a fix, introduce an update, etc. If it's working now, odds are it will keep working for the foreseeable future. However, all bets are off if you start using a newer version in newer classes, which may cause the older classes to behave incorrectly. This goes doubly so for utility classes that may be used in many different classes with many different versions. Keep them at the older version, or plan on updating every class that uses the utility class at the same time. Test thoroughly.
Try to bump up all your classes at once, and just one version at a time. It's far easier to research breaking changes between one version than 10 versions. Make sure you perform a Run All Tests afterwards, and be sure you have good unit tests (those that include asserts), rather than code coverage unit tests. If you've been in the habit, now would be a good time to catch up.