RE: 1. How does the licensing works force.com custom application?
Salesforce provides a licensing engine to it's Partners called License Management Application (LMA). It is available to eligible partners (it's free to become one) and after your managed package has been security reviewed (free for free packages) they will provide you with the installation link for the LMA. You can read more about the process in the wonderfully useful ISV Force Guide.
Then configure via an AppExchange login an org wide or per user seats for your package, as well as a trial period. As your package is installed it will automatically generate Leads and License entries in the org the LMA is installed within (known as the License Management Organisation or LMO).
2. Does Salesforce sell licenses on small qty (3 nos) for SME?
Yes, Salesforce sells different editions for numbers of users depending on the size of company and their requirements. Make sure you know what edition of Salesforce your targeting, as not all features available in your Developer Edition org where you build your package are available in all editions. There is a good section on building apps for different editions in the ISV Force Guide here.
3. Limitation of Force.com - especially number of objects per app?
Once you pass the Security review your package will automatically (usually) be enabled to bypass the object, tab and application limits.
For partners who are enrolled in the ISV Program, any managed package publicly posted on the AppExchange no longer counts against the apps/objects/tabs limits for your Salesforce Edition. This effectively means that ISV partners no longer have to worry about package installation failures because of apps/objects/tabs limits being exceeded. This feature is automatically enabled after your app passes the security review.
4. To start with can I proceed with developer edition and let the firm/company use the application and know the experience about the custom app and subsequently sign-up for licenses and subsequently deploy the application in the new org. Is this approach fine & workable?
Yes, but use a separate developer edition to the one your developing and packaging within. Utilise the ability to upload Beta editions of the package so that you can still make changes to aspects that get locked down once you perform a Release package. Typically however clients with sandboxes prefer to install your Beta package in their sandbox alongside their data and other customisations to get a better feel for your app. Note that Beta packages cannot be upgraded, only uninstalled and a new beta installed. Or if a sandbox is used, a refresh of the sandbox and a new Beta installed.