Yes, the primary purpose of Finalizer is to permit recovery from LimitException. However, this is huge. If a Queueable crashes from a LimitException, it can't retry itself, the context data may not have been logged to a Debug Log, and all anyone can see is a simple "Status: Failed" message.
With this new feature, you can do things like: send an email notification of what happened, save detailed logs of what the execution state was when the process crashed, attempt to restart the Queueable in case it was a transient error, perform callouts to other systems after performing DML in the Queueable (remember, callout after DML is illegal).
It also allows you to write cleaner code. If you haven't read my answer about the Pokemon Catch, take a moment. I suggest that developers avoid catching Exception as a general rule. Most developers abuse Exception, and therefore it should be avoided. Catching Exception directly suggests that the developer has no idea what type of errors are possible in a given area.
As a competent developer, one should be able to anticipate which exceptions are possible and avoid them with simple if statements. Catching only exceptions that can be caught displays confidence and competency, and specific exceptions are more explanative/self-documenting than handling the generic Exception. The try-catch should ideally be as close to the problematic code as possible.
Forcing yourself to think about what type of exceptions can occur, and catching only those exceptions, will help you become a better developer. In fact, most of my code doesn't even have a try-catch handler at all; if it does, it is never around an entire method, but usually just a couple of lines of code, and usually catches a specific type.
Given two choices, either handling all exceptions generically, or handling specific exceptions selectively and reporting unexpected exceptions in a separate transaction, it should be clear which I'd choose to do. Also, as another benefit, you can conceptually write one Finalizer that could work for many different Queueables, providing a standard interface for exception handling and reporting, as well as reducing duplicated code.
So, rather than thinking of this as a simple tool that helps with LimitException, think of this as a way to write more robust code that runs more efficiently when things are running smooth, and has more capability when things go unexpectedly wrong.
Finally, keep in mind that finalizers also run when the job succeeds. This is also an excellent opportunity to log data, make any remaining callouts, or whatever additional post-transaction logic you'd like to do that cannot be done in finally block.
