It is possible to have an extension for a custom controller.
Let's say my custom controller is this:
public with sharing class MyController {
...
}
My extension could then have a contructor that is called when this class is the controller for a page:
public with sharing class MyExtension {
...
public MyExtension(MyController ctrlParam) {
}
}
You could have the same extension work for both ApexPage.StandardController and some custom controller, but you'd need to overload the constructor:
public with sharing class MyExtension {
...
public MyExtension(MyController ctrlParam) {
}
public MyExtension(StandardController ctrlParam) {
}
}
If you want your extension to be reused for a lot of controllers, you might need to plan ahead and use inheritance either from a superclass or an interface so that you don't end up with too many constructors.
I suspect that most customer implementations do not get to this degree of sophistication. Where I would expect this model to crop up more often is in AppExchange app architecture.