Yes, its possible to install a custom application from an alternate source than app exchange.
Following are different options:
- Developers can distribute links of their managed or unmanaged packages (custom apps), without publishing them to appexchange. Using this links admins can install the application in any Developer/Sandbox/Production orgs.
- Using source code with any deployment tool:
- Apart from packages, developer can share their source code/SRC file which can be deployed to any org with help of Force.com IDE, Workbench or ANT deployment tool.
- Deploy to Salesforce button in GitHub is also quite popular.