Rather than paraphrasing the difference, here is a great explanation, and I have have also added my own view.
A standard controller consists of the same functionality and logic
that is used for a standard Salesforce page. For example, if you use
the standard Accounts controller, clicking a Save button in a
Visualforce page results in the same behavior as clicking Save on a
standard Account edit page. If you use a standard controller on a page
and the user doesn’t have access to the object, the page will display
an insufficient privileges error message. You can avoid this by
checking the user’s accessibility for an object and displaying
A custom controller is a class written in Apex that implements all of
a page’s logic, without leveraging a standard controller. If you use a
custom controller, you can define new navigation elements or
behaviors, but you must also reimplement any functionality that was
already provided in a standard controller. Like other Apex classes,
custom controllers execute entirely in system mode, in which the
object and field-level permissions of the current user are ignored.
You can specify whether a user can execute methods in a custom
controller based on the user’s profile.
See VisualForce in Practice for a complete guide.
Using a standard controller means that you will not have to write any tests, or maintain any code - that is the main benefit, but you are restricted to more basic functionality and logic, but this is fine if you just want a custom visualforce page for Create, Update and Delete. However, as mention in VisualForce in Practice you could use a standard controller for all of these features, then extend it with a custom controller if you wanted to do something more fancy - like a complicated calculation or have two or more objects within the same visualforce page.