You don't have access to users, accounts, contacts, etc. You can access them only by creating wrapper classes for those objects. Given that you're already coding pages, it's not that much more work.
If any custom object has a master detail relationship to a standard object that the user can't see, the can't see the child object either.
There's no chatter...you might want something to post internally when the site guest user does something, but you can't.
No reporting or dashboarding.
The guest user can own records, but can't be part of sharing rules.
The guest user doesn't have any error details on their error page. If you have an apex runtime exception, you'll get an auth error instead of the actual error in the UI.
Sites has its own limitations (page views, bandwidth, etc).