Tabs API Status
While there is a means to list the tabs in Application from Apex via describeTabs, there is no means to update these from Apex. You may be able to do this via the Metadata API (which is a SOAP API) by calling it from Apex using the update operation with the CustomApplication metadata type. However doing so from an InstallHandler requires the Remote Site be setup to permit (ironically) the callout to the Metadata API and since this differs based on the subscriber org instance, its hard to predict and thus package the required Remote Site setting to make this work.
Your use case is a common one however, in that you want the user to go through some Setup and configuration before they attempt to use your application. These are a few points to consider that will help address this use case.
- When users install your package, have them install and make visible only to the Administrator, this is an option available during the installation wizard.
- Make the Setup tab in on application the default tab or utilise a Configure link
- Utilise an InstallHandler to send them a link to the Setup tab / page after install
- Leverage Permission Sets to allow the Administrator to easily enable visibility of the tabs to their end users once the Setup has been completed.
- If you want to go to further lengths to prevent access to your app, you may want to consider a protected Custom Setting flag that contains the Setup state of your application, which can be referenced by key pages and an appropriate message be displayed. Though generally you should aim for your app to fail gracefully when setup steps have not been performed.
I hope some of these thoughts help you!