I really wouldn't call it a hitch. It's good security practices.
In order to authenticate into the org, you need a subject who exists in that org and the URL for the org (My Domain in most cases). This subject or their admin then indicates that your app is trusted to access the org on their behalf.
In your scenario, you're essentially proposing that you'd like to access anyone's data in any org without being granted explicit access to authenticate as a specific user. That is a recipe for disaster.
In scenarios such as yours, the typical pattern is the external system maintains information about the subject and the domain of the org that they will need to authenticate with in order to make a requests to Salesforce as that named user.
This general pattern isn't limited to JWT, if you were storing a refresh token (instead of using certificates to establish trust) you would store similar information for the org and the named user in the external system so that you can identify who the refresh token belongs to.
As part of your application's post install steps, you could direct the admin on how to install & authorize the connected app and also decide on whether the external system was going to use a named principal (all requests into Salesforce are via the same user) or per-user (requests to Salesforce are made as individual users) authentication and then your external system would need to capture this configuration so that it can be used later when calling into the org.