The salesforce.com sharing mechanism is permissive, not restrictive, meaning you can only ever increase access through code or configuration levels, not decrease it. This means that if your organization wide defaults (OWD) is set to Public Read/Write, then no record may be less than Public Read/Write to any user, assuming that user has the appropriate profile permissions (but that's not a sharing setting). For example, with OWD of Public Read/Write, it is ineffective to try and create a manual share for Read/Only, because it would have no effect.
The bottom layer of sharing is the OWD. It defines the minimum access to a specific record, assuming the user has profile permissions to that object. Higher levels of OWD increase system performance, because fewer checks must be made to determine if a user has access to a record. For example, queries will run much faster if OWD is at least Public Read/Only, and edits will be marginally faster if the OWD is Public Read/Write. After OWD comes sharing rules, owner sharing, team sharing, manual sharing, implicit sharing, and role hierarchy sharing. It doesn't matter which order these rules are evaluated in, as the highest permission wins.
So, to make most records Public Read/Write, but a few Private, you have to start with Private OWD, then build sharing rules that meet your criteria. Thanks to Criteria-Based Sharing, you can choose to create rules where records of a certain record type, etc are shared with all users. The remainder can then be manually shared, or by code, or by more specific rules. This may have a performance impact on list views, reports, and searches, as the Private OWD will automatically kick off a bunch of other necessary checks to see if users have access to a specific record.