When: It is basically the act of installing any managed packaging in your development packaging org. The platform seems to rattle through your metadata files and apply this (it's also evident from the Versions tab when viewing Apex via the web ui). I agree it does seem a bit over the top given you're likely not referencing any components from such packages. The same is not true if you upgrade them however, the package version stays the same as when it was first installed (another recent post here as well).
Why? I suspect the platform is doing it to maintain some kind of consistent versioning perspective at runtime that may be needed directly or indirectly as your code executes. For example if you use Apex Describe over all org objects and fields, the information it returns when it encounters packaged objects will remain consistent, after upgrades to those packages. Or if your package is installed in a subscriber org with a later version of such packages available.
Packaging Of course that above scenario in subscriber orgs does assume your package one way or another becomes dependent on such packages for the platform to even consider recording such information once packaged. Fortunately the packager is a little more considered before making package dependencies that just simply them being installed.
Summary In short it's a bit shotgun in your packaging org, but more refined to retaining versions to packages only where explicit references are made when producing your final package.
Hope that makes senses.... (it's a bit late in the UK) ;-)