How does licensing enforcement works in AppExchange?
It's enforced via the License Management App (LMA). You install it in your production org, and it allows you to set up trials, extend or terminate trials, configure "site-wide" or "per-user" models, and log in through the support portal to diagnose problems. Changes you make in the app will affect the licenses and expiration date in the subscriber's org.
What are the best approach to decide how to price it?
There's really no way "best way" to decide. Software, like most other services, are supply and demand. As you increase the price, you'll (likely) have fewer subscribers, but if you drop the prices too much, you will lose money in operational costs despite having a lot of subscribers.
To figure out a price, you need to do things like figure out who is going to support your application. Do you have a call center? Will you outsource this support? Will you offer email-only support? Who handles sales? How many people do you have in sales, and what do they do?
Most apps seem to float between $5 to $25 a month per user, or a flat rate of $50 or $100 per month for a site-wide license. However, this isn't a hard rule; you might choose any other number that makes sense for your purposes. The goal is to get your price low enough to get a lot of subscribers, but high enough to make a profit. Nobody can tell you what that means other than people in your organization that have determined how much it will cost to support/develop/sell your application.
How does payment integration works from licensing point of view?
You can choose to use the built-in checkout screen, which may reduce operational costs, or you can use your own method of accepting payments, whatever that might be. The cost for using AppExchange checkout is 15%, with no setup fees, monthly costs, or storage fees. You can choose one-time payments or subscriptions, monthly or annually. You can read more here and in the related sections.
What are some of the less obvious points we need to keep in mind before launching paid app exchange app?
You need to remember the fees. I know this may seem obvious, it's all in the documentation, but you need to remember it when choosing a price. For example, the initial cost is $2,700 for the first year, and $300 for each year afterwards for security reviews. You also need to pay 15% of whatever you make as part of your ISV agreement (for the licenses you collect fees on, not all of your business).
You may also want to use contracts (some organizations do), but that's again dependent on your business model and on how you already operate. Other than the complexity of actually figuring out pricing, the actual management aspect is pretty easy. LMA comes set up with objects, tabs, fields, and reports, and you can customize it with Process Builder, Workflow Rules, Triggers, and more to help automate your sales and renewals.