The flexibility of the sharing model much depends on which licenses you are using. If you are using full Customer Portal licenses, then these users will have roles and participate in sharing rules much as regular users do (although they appear in a fixed place in the role hierarchy, under the owner of the account that their user is associated with).
If you are using High Volume (aka Service Cloud Licenses), then these users don't have roles and don't participate in the sharing rules (hence they are much cheaper than customer portal licenses). There are sharing sets available for the high volume licenses, but these are based on profile require a lookup on the record to be shared to either the account or contact associated with the high volume user, so they really aren't that flexible.
As you are using Force.com sites, you are in Visualforce land, so if you are using custom/extension controllers you can decorate your class with the 'without sharing' keywords - this ignores the sharing restrictions of the currently logged in user so would allow you to provide access to common data regardless of the user's actual access.
