There are two aspects to your question: regulatory and business.
Looking into the latter one, it is up to you to decide on how you would treat inactive contacts who are still opted in. Normally, if no opens and clicks have been registered for e.g. 6 months, a reactivation campaign - potentially sent with a different sender profile, could be initiated. It's aim would be to reactivate the contact and resume engagement. If none of the emails sent as part of the reactivation campaign are opened, consent will usually be revoked - leaving the contact out of future sendouts, hence improving your open rates.
Unless this contact opts back in, I would introduce a data retention process, deleting the contact after 12 months of inactivity. As you can't query your data views for more than 6 months back, you should have a process in place, updating a lastEngaged date field on one of the data extensions in your data model, referenced by the contact key. This will allow us to keep an eye on those unengaged for longer than 6 months. Once 12 months have passed since last activity (open or click), the contact is deleted using contact deletion framework. A process which can be automated using SSJS, as I have described it here: Marketing Cloud SSJS: trying to automate contact deletion
Since you have a high number of contacts who are not active, they should immediately have their consent revoked, and you will see your engagement rise substantially.
The regulatory aspect, where you often are not allowed to store personal data unless you have a justifiable reason for it and a valid consent is provided, varies from country to country - and should be investigated, as it could have more restrictive rules as to when you must remove any data from Marketing Cloud.