We have quite a few "without sharing" classes in our managed package - we have many different "utility classes" (i.e. not directly user-facing code, such as Visualforce controllers or REST APIs, but rather for system processing, such as data maintenance batches, scheduled jobs, internal calculations etc.) that perform operations where sharing makes no sense at all. As sfdcfox said, this is a red flag for the security review and you need to explicitly document the reason for "without sharing" in each case.
We have done this by ensuring we have apexdoc comments on the classes along with explicit comments on the source code lines that are extracted by Checkmarx, like this:
global without sharing class SomeBatch implements Database.Batchable<SObject>, Database.Stateful { // Sharing False Positive: This is a utility class for data processing which must be ignorant of sharing rules and user permissions
That way, when the Salesforce Security Review Team view the Checkmarx report they can see the code is marked as a "false positive" and if they want the full detail they can read the apexdoc to see why.