Option 1: Instead of having the list view on contact, point your users to a list view on the junction object. Then on your junction object, add formula fields to whatever you needed in the list view columns. This can also include the lookup to contact/user objects so that they open that object instead of the junction object.
Option 2: I wonder if something may be optimized with your data model. For instance, Contacts can be owned/associated to multiple Accounts, Accounts can be owned my multiple users with Enterprise Territory Management, and once you set that up, you'll get a few new list view options. That's just one example and it really depends on your circumstance, but maybe take a minute to reflect on the overall model before reading my next option. Often things get harder to do when we aren't taken advantage of the built-in model.
Option 3: If there's only about a dozen or so users, then you can get exactly what you asked by doing the following:
a.) On Contacts object, create a custom text field of 255 characters named something like "RollupUsersName__c"
b.) Setup a piece of code/config to roll-up a concatenated string of User's Name from the junction object to the Contact. There's a few ways to do that. You can do this with apex trigger code as the junction object is added/removed. Or another way, you can use the Rollup Helper appexchange app to do this through configuration. What you want is for the RollupUsersName__c to have a value like "John Doe;Jane Smith;John Stiles;Richard Miles;"
c.) Now your list view on Contact can include a condition to say something like "RollupUsersName__c" contains "Jane Smith"
That'll work if you have up to a dozen or so users per Contact and be easy for a regular user to tweak the list view. (You do have a minor risk of similar names that are the same but have an added prefix/suffix, like Junior.)
You could extend to more users per contact if instead of rolling up name, you rolled up some short fixed length user id that was assigned to each user. (Like p001, p002, p003, etc.) That would also remove the risk of similar names. But, also be a bit harder for an end user to tweak the list view to a particular person. Who is p003?