As the error already mention that there is a depended package which exist in your org and that will not allow you to upgrade it to beta.
Because beta package can't be updated again and thats why it is not allowing you to upgrade it to beta.
So first remove or uninstall the dependent package and then update this package.
The only way to uninstall a Beta Package is to completely uninstall the managed package that the beta is a version of. This means that if a customer is developing on top of the managed packages with their own custom objects, triggers, etc. that references objects within the package uninstalling is a pain of needing to go through and delete all of their custom development work, uninstall the beta managed package, reinstall the released managed package, then re-deploy all of their custom development work, then redeploy (or recreate) their data.
Also from the standard SF doc Packages cannot be upgraded to Managed - Beta if they are used within the same organization as an extension.
Reference