There's a maximum amount of data you can serialize before the batch will fail (and also counts against your heap limits during execution). Static variables serve the same purpose as transient variables in Visualforce; they will not be serialized and therefore not count against heap. On the downside, static variables are reset between each transaction, so they will be null or set to their default initialization value, if any.
In general, if you need to track the data, make sure it is not static, and if you do not need to track the data, make sure it is static. The more data you have to serialize, the slower your batch will run (serialization/deserialization requires more CPU time).
The static variables will be available in unit tests after the batch execution, but the instance variables will not. None of these variables will be observable after the finish method returns in normal execution (e.g. execute anonymous scripts or Visualforce page contexts). If you need access to the results, you need to persist the data somewhere, such as the Platform Cache, a database record, etc.
Static variables generally have two purposes: unit tests and sharing data across methods. The unit test bit is obvious; if you want to see what happened during the execution, you need a place to store it. The static variable can be accessed by the unit test after execution. As far as sharing data between methods, some developers prefer this method, but should probably be used sparingly, since it makes it less obvious which bits of data belong to which method.
Edit:
Alternatively, if I have a batch job that doesn't maintain state (doesn't use Database.Stateful ) how do member and static variables behave?
In this mode, static variables work as before (they are not serialized and reset to null/default value), while instance variables are reloaded from the batch's initial state when it was placed in the queue using Database.executeBatch. No changes to instance variables are preserved between start, each execute, and finish calls.