Return to Answer

1. Create a cacheable, Aura Enabled Apex method that returns an object holding the Page.pagename.getUrl() value and an indication whether the page is in a community (using Site.getSiteId() != null)
2. Have the LWC invoke this method using a wire
3. Have the LWC use the returned URL value:
4. When in a community: just use window.open with the URL with added query parameters (since communities don't support navigation to a page reference as a standard web page)
5. When not in a community: with NavigationMixin but treating it as a "standard_webPage". The URL itself has the required query parameters added before being used as the "url" attribute for the NavigationMixin.generateUrl call. The mixin promise uses window.open.

1. Create a cacheable, Aura Enabled Apex method that returns an object holding the Page.pagename.getUrl()** value and an indication whether the page is in a community (using Site.getSiteId() != null)
2. Have the LWC invoke this method using a wire
3. Have the LWC use the returned URL value:
4. When in a community: just use window.open with the URL with added query parameters (since communities don't support navigation to a page reference as a standard web page)
5. When not in a community: with NavigationMixin but treating it as a "standard_webPage". The URL itself has the required query parameters added before being used as the "url" attribute for the NavigationMixin.generateUrl call. The mixin promise uses window.open.

** Update:

We actually found that when using with communities that include a URL suffix that the Page.pagename.getUrl() did not add the community URL suffix and therefore didn't work. This was remedied by having a known static resource and using the following:

String theURL = PageReference.forResource('theknownresource').
        substringBefore('/resource/') +
        Page.pagename.getUrl();

This then works outside communities, in a community with no URL suffix, in a community with a URL suffix and in a community with multiple URLs.

OK, so we didn't find a direct way to do it. However, we believe that the following approach should work for normal and community usage. In essence we implemented a component with navigation mixin then used something like:

Importantly, this doesn't actually navigate directly to a Visualforce page, but rather to a custom tab into which the page is placed (clearly the tab needs to be accessible by the user). Note the use of window.open allows the URL to be opened in a separate tab or window within the browser.

EDIT: The "query string parameters" included do NOT seem to be correctly available within the Visualforce page - this because of the use of iframes across domains. Still working on a solution for this at the moment.

UPDATE: Eventually we found a way to do this, i.e. invoke a Visualforce page with query parameters from an LWC, though it was a lot more complex:

OK, so we didn't find a direct way to do it.

Failed approach

Initially we thought that the following approach should work for normal and community usage. In essence we implemented a component with navigation mixin then used something like:

Importantly, this wasn't actually trying navigate directly to a Visualforce page, but rather to a custom tab into which the page is placed (clearly the tab needs to be accessible by the user). Note the use of window.open allows the URL to be opened in a separate tab or window within the browser.

Unfortunately, the "query string parameters" included do NOT get through to the Visualforce page - this because of the use of iframes across domains.

The Solution

We eventually found a way to do this, i.e. invoke a Visualforce page with query parameters from an LWC, though it was a lot more complex:

UPDATE: Eventually we found a way to do this, i.e. invoke a Visualforce page with query parameters from an LWC, though it was a lot more complex:

1. Create a cacheable, Aura Enabled Apex method that returns an object holding the Page.pagename.getUrl() value and an indication whether the page is in a community (using Site.getSiteId() != null)
2. Have the LWC invoke this method using a wire
3. Have the LWC use the returned URL value:
4. When in a community: just use window.open with the URL with added query parameters (since communities don't support navigation to a page reference as a standard web page)
5. When not in a community: with NavigationMixin but treating it as a "standard_webPage". The URL itself has the required query parameters added before being used as the "url" attribute for the NavigationMixin.generateUrl call. The mixin promise uses window.open.

The LWC code looks something like:

handleClick(event) {
    event.stopPropagation();

    if (this._url) {
        // Add query parameters to the URL
        //
        // Note that these aren't added in the standard way (using the 'state' property of the mixin object)
        // because then they wouldn't get passed to the iframe containing the visualforce page. It's possible
        // that this is unintended behaviour of the mixin and in the future Salesforce may change it and
        // properties added manually like this may no longer be supported.
        const urlWithParameters = this._url +
            '?' + this._queryParameters;

        if (this._isCommunity) {
            // Navigation to PageReference type standard__webPage not supported in communities.
            window.open(urlWithParameters);
        } else {
            this[NavigationMixin.GenerateUrl]({
                type: 'standard__webPage',
                attributes: {
                    url: urlWithParameters
                }
            }).then(generatedUrl => {
                window.open(generatedUrl);
            });
        }
    }
}