As far as I can tell, the system uses two simple rules: the first compiled unit's version, or the API request version if no compiled code is involved.
The first rule can cause problems when a lower-versioned class or trigger runs first. As I've said before, you should make a conscious effort to set all of your classes to the same version. The first encountered version determines the behavior for the transaction.
If no compiled units are in play, then the API request version is used. This applies to all API calls where no code is involved, or where no compiled code is involved (e.g. executeAnonymous that does not invoke any compiled code), or when using the UI (in which case, it is the latest version).
When multiple triggers on the same object are involved, one will be selected to go first, and consequently determine the API version of the entire transaction. In that case, the version will be unpredictable if the triggers are on different API versions. This doesn't apply when those triggers are invoked from a Visualforce page or Apex class (e.g. via @RemoteAction).
I'm not going to test every conceivable combination, but based on simple experimentation, I'd say at least most of this answer is accurate. For that reason, I strongly recommend that you keep all of your code at the same API version to avoid surprises. This includes unit tests, because the unit tests will determine the API version for the called classes.
Edit: I just realized that I should mention that the "header" version that's displayed is not necessarily what each Apex Class or trigger will function at. It determines some behaviors, but certainly not all behaviors. This makes things a lot more complex. For example, the exact version of the API you use on a class should determine if JSON.serialize includes null values or not, or if
String.split('') will include a blank string at index 0. The first class that performs a query seems to set the API version for all subsequent queries in the transaction. It gets really complicated to try and predict what will work and what will not.