To understand the implications to an apex class of with/without sharing first you need to understand how salesforce's security model works. It really consists of three layers of security:
- CRUD (Create,Read,Update,Delete), or table-level access. This is defined on the profile/permission set.
- FLS (Field level security), or column-level access. Also defined on profile/permission set.
- Sharing, or row-level access. This is much more complex and is based in large part on the configuration of your org.
CRUD is automatically enforced by standard controllers, if you make use of them. FLS is automatically enforced by the apex:inputField and apex:outputField tags, if you make use of them.
Sharing is enforced at the class level based on this keyword.
But what exactly is sharing?
It depends in large part upon you organization-level defaults. By default in new orgs most objects are set to public read/write, which disables sharing for that table until changed. This means that out of the box with/without sharing often won't have much an impact.
However once you do turn "on" sharing by changing this setting to something more restrictive you'll find that sharing is a complex calculation that's done by checking the role hierarchy, record ownership, apex-created sharing rules, point-and-click defined sharing rules, and even manual shares created by hand on a per-record basis.
The net effect of this is that sharing is much, much more complex to calculate than CRUD/FLS (which have simple apex methods for "can I do this?"), so this class-level control on sharing was introduced. If a class is defined as "with sharing" then the system will automatically enforce sharing on all queries you run, and DML you perform.
If your user only had read-level access to Account "X" and you try and update a field on it then "with sharing" would fail, but "without sharing" would succeed. The UI would fail in this scenario, and prevent the user from editing the record, so "with sharing" is a consistent experience for the user here.
If you leave off the sharing declaration then the class will inherit it's with/without sharing from whatever calls it. this is quite useful for helper classes with only static methods, if you leave off the sharing declaration then whatever calls these methods will determine if they respect sharing or not. If a class with no sharing declared is the top-level in a request (e.g. a visualforce page controller) then it will default to acting "without sharing".
Edit: Except when using execute anonymous, where it will behave as "with sharing".