For Unlocked packages, you can revert a class from global to public or even to private and it should allow you to do it fine because it simply does not matter.
2GP Managed Packages
You can downgrade an Access modifier of a class from
public or even
private provided you create a new package version from the source with no ancestry. That is in short you cannot downgrade without breaking package ancestry.
To learn more about ancestry take a look here
In short, what specifying
ancestry during the package version creation process does is, maintains upgradability rules for you (Like Rules that are needed to ensure release does not suddenly break something in consumer org).
Effect of creating a Managed Package with No Ancestry
If an org already has the package version installed with the global class you cannot upgrade the org with the new package version (where apex class is public/private) with different ancestry without uninstalling the previous versions. Note this will mean you will have to restore the data along. You want to avoid this.
So in short if you have a Released Managed 2GP package that already has a global class and is installed in a number of orgs if you try going back to public/private, it simply won't be allowed during the packaging process without you taking off the
ancestorId during package creation.
Now if you take off the
ancestorId it simply means you have branched off at that point and would mean destructive change for the customer who already has the app installed.
Global classes should be avoided in 2GP if one can since the public class can be made available to other packages within the same namespace.
Note be above applies to all the components. This is called manageability rules and documented in 1GP package upgrade components behaviour in ISV document and they are all applied. These are documented here