The new Licensing Components (Developer Preview) allows you have a single app with Licensed Permission Sets that can be individually sold and billed as addons. This would have a single AppExchange listing, and you could define a pricing model for each license type. However, this is a pilot program, so you'll have to contact your AE for details on availability. If you can get in on this, it would be your best option.
You can also use 2GP (Second Generation Packaging) to share a namespace with two or more packages. In the most trivial model, you could create two packages, and list each on the AppExchange. Since they're in the same namespace, they can even communicate with each other if both are installed. They can be individually licensed and billed as such. This would be the preferred model if you can't have access to the pilot, for now. Licenses are assigned to users to allow access to a specific feature.
You could also use a variant where you have three packages, one that acts as a shared code/metadata repository, and one each for the two apps. This might still be two listings, but subscribers would have to manually install the base package before installing either of the other two. I'm not sure how this would affect Security Reviews, though, there may be extra fees associated with a three-package model as opposed to just two standalone apps. Again, for non-Site licenses, each package would have to be assigned to users. The base package itself would just be a Site-wide license, since its the other two packages that provide licensed access to the features.
Finally, you could use a 1GP (First Generation Packaging) or 2GP package, and have extensions that you install. This can be detected at runtime to wall off parts of an app if they don't have an appropriate package installed. However, you end up having to build a custom licensing system, and it won't be supported by Checkout directly. You're allowed to bill in pretty much any manner you wish, but it's a very manual process to have just a single listing and not use Licensed Permission Sets. Consider this a last resort.
Summary:
Is this doable with a single app?
If you can use Licensed Permission Sets (LPS), it's trivial. If not, it's doable with the FeatureManagement class, but requires a lot of supporting code. I'd avoid this because of the complexity it introduces.
Or do I need to split the app into a core app plus two other extensions that implement the specific licenses?
You can do three packages, but two is better; this lets you just have the core metadata you need in each package, and licensing is simplified. The three package model would only help if you need to share metadata across these two packages.
Would those two be different listings on AppExchange or what?
Not every package ends up on the AppExchange; that's up to your design choices:
Listings |
Design |
1 |
Single Package with LPS |
1 |
Core package with unlisted extensions |
2 |
2GP standalone packages |
2 |
2GP packages with dependency on third unlisted core |
It is not required to have a listing to manage licenses in the LMA (Licensing Management App), only that these packages are linked to the LMA/LMO (Licensing Management Org).