There are two different things here.
- Beta Versions of Managed Packages(These can be private manage packages or App exchange managed packages. Private one's doesn't need to go through security review. EX: You can work with a vendor to build some functionality specific to your needs and they can do it as a private manage package for you.)
- AppExchange Security Review
Beta Versions of Managed Packages: A beta package is an early version of a managed package that is uploaded in a Managed - Beta state. The purpose of a Managed - Beta package is to allow the developer to test their application in different Salesforce organizations and to share the app with a pilot set of users for evaluation and feedback.
AppExchange Security Review: We want Lightning Platform to be a safe and reliable platform for our customer’s core business applications. To promote trust, all apps and components that are distributed to customers undergo a comprehensive security review. Your offering must meet or exceed the requirements set by our security review team before it can be distributed(Unless its a private managed packed specific to the vendor only that doesn't want it on app exchange). After approval, apps and components are also periodically retested.
To your question:
No you can't install it in production org and your're not supposed to.
More about packages here : you can choose to make your package managed release and decide not to make it a public listing in app exchange. In your case your'e looking for private released manage packages
Note: Unfortunately i didn't find a source that says betas can't be or
can be installed in production org!