The system currently ignores the milliseconds for the purposes of considering two values equal. Here's some code you can run in your org to prove it to yourself:
User u = [SELECT SystemModStamp FROM User ORDER BY SystemModStamp DESC LIMIT 1];
DateTime dt = u.SystemModStamp;
// add 900 milliseconds
dt = DateTime.newInstanceGMT(
Time.newInstance(dt.hourGMT(), dt.minuteGMT(), dt.secondGMT(), 900));
User u2 = [SELECT SystemModStamp FROM User WHERE SystemModStamp >= :dt];
// They are equal despite being 900 milliseconds apart
So, what happened to the records that your code was periodically missing? The answer is that you most likely did not include any row-locking statements, so the records in question were still "in flight" and couldn't be found by the query.
Depending on what you're doing, there's several possible solutions to the problem. You may need to use FOR UPDATE if you're using a SOQL in Apex to make sure that you capture any in-flight records. If you have an integration, you may want to use the Replication API, which accounts for in-flight modifications to make sure you don't miss records in a given window. For Batchable, where you can't use row-locking statements, you may simply need to add an acceptable amount of offset to make sure that you don't miss records, but you may need to account for duplicates being processed.
In the end, it's probably best that you avoid relying on SystemModStamp directly, or avoid running queries that may miss in-flight records (e.g. by running to the nearest millisecond). The problem is not with the precision of the audit fields, since milliseconds are always ignored.