OK, so we didn't find a direct way to do it. However, we believe that the following approach should work for normal and community usage. In essence we implemented a component with navigation mixin then used something like:

    this[NavigationMixin.GenerateUrl]({
			type: 'standard__navItemPage',
			attributes: {
				apiName: "customTabName",
			},
			// query string parameters
			state: {
				c__showPanel: 'true' // Value must be a string
			}
		}).then(url => {
			window.open(url)
		});

Importantly, this doesn't actually navigate directly to a Visualforce page, but rather to a custom tab into which the page is placed (clearly the tab needs to be accessible by the user). Note the use of window.open allows the URL to be opened in a separate tab or window within the browser.

**EDIT**: The "query string parameters" included do NOT seem to be correctly available within the Visualforce page - this because of the use of iframes across domains. Still working on a solution for this at the moment.

**UPDATE**: Eventually we found a way to do this, i.e. invoke a Visualforce page with query parameters from an LWC, though it was a lot more complex:

 1. Create a cacheable, Aura Enabled Apex method that returns an object holding the Page.pagename.getUrl() value and an indication whether the page is in a community (using Site.getSiteId() != null)
 2. Have the LWC invoke this method using a wire
 3. Have the LWC use the returned URL value:
 4. When in a community: just use window.open with the URL with added query parameters (since communities don't support navigation to a page reference as a standard web page)
 5. When not in a community: with NavigationMixin but treating it as a "standard_webPage". The URL itself has the required query parameters added before being used as the "url" attribute for the NavigationMixin.generateUrl call. The mixin promise uses window.open.

The LWC code looks something like:

	handleClick(event) {
		event.stopPropagation();

		if (this._url) {
			// Add query parameters to the URL
            //
			// Note that these aren't added in the standard way (using the 'state' property of the mixin object)
			// because then they wouldn't get passed to the iframe containing the visualforce page. It's possible
			// that this is unintended behaviour of the mixin and in the future Salesforce may change it and
			// properties added manually like this may no longer be supported.
			const urlWithParameters = this._url +
				'?' + this._queryParameters;

			if (this._isCommunity) {
				// Navigation to PageReference type standard__webPage not supported in communities.
				window.open(urlWithParameters);
			} else {
				this[NavigationMixin.GenerateUrl]({
					type: 'standard__webPage',
					attributes: {
						url: urlWithParameters
					}
				}).then(generatedUrl => {
					window.open(generatedUrl);
				});
			}
		}
	}