What is the benefit of transacting through the AppExchange checkout vs having an AppExchange listing and doing the licensing and order fulfilment on our company website,
Reduced complexity. Instead of paying for a merchant account/payment processing separately, you can streamline the entire process. If you're not already using a vendor for card processing, this is far easier in many cases, especially for small businesses.
I've notice a number of other apps do it via their own website so am wondering the advantage of this is since you would still need pay the 15% to SF I assume plus pay for payment processing etc:
If you have other services you also bill for, it can make sense to consolidate all the billing to a single platform. Checkout is optional, but a great choice for Partners that don't have any other need for a merchant account/payment processing.
Is there are flat fee of 15% on all apps?
Yes, all AppExchange apps are charged 15% of license fees.
If the app is free is there any minimum fee to list and transact on the app exchange?
You must pass the security review (renewed annually for a nominal fee), but there are no additional fees as long as you are not charging for the app. You can choose to charge for some and not others if you have multiple listings. You're only contractually obligated to pay on any licencing fees you collect.