I raised this on the Partner Forums too. I got the following from two of the guys at Salesforce. Firstly Dileep Burki:
The postInstallURL supports the specification of a URL to which the installer is taken to, upon the completion of the installation process. Unlike 1GP, in 2GP, we only support the URL and NOT the VF page. This is something that fell below the line when we scoped the effort.
We did not invest effort in "Configure Custom Link" as we felt postInstallURL suffices and we had other competing features that we wanted to invest in.
Then from Vivek Chawla:
As for your question about relative URLs, here's what I've learned from the engineers and from doing my own tests.
Both the "postInstallUrl" and its cousin "releaseNotesUrl" are validated by the CLI during package:version:create. Only fully-qualified URLs, like "http://www.yourdomain.com/your-release-notes.htm", are allowed.
Functionally, since relative URLs are not allowed, it is not possible to use the "postInstallUrl" or the "releaseNotesUrl" to point back to a page inside of the subscriber's org.
This definitely removes, or at least modifies, the post-install configuration pattern as compared to 1GP. Partners still can, and in many cases, should create post-install configuration pages as part of their solution. The problem is that getting to those pages won't be as straightforward for the customer with 2GP.
I need to chew on this for a bit, but as someone who has actively clicked on the "configure" link more than once during my previous life as a Salesforce/AppExchange customer, I'm leaning towards this feature not being part of 2GP as being a real gap.
So in summary this is a gap between 1GP and 2GP and it isn't possible to have a post install URL that points to a page in the just-installed package at this point (currently Winter '21).