oAuth client credentials grant type is not supported by Salesforce at all. As you have discovered, in the Salesforce world a recommended alternative for service-to-service calls is JWT Bearer. It is standardized (RFC 7523), Salesforce was one of the RFC editors/authors.
When the app cert expires, nothing happens. From SF product management:
this is by design. Thought process is that that certificate is issued
by another party whose responsibility it is to enforce the expiration.
The behavior we have is tolerant of lapses in cert management.
Fortunate or unfortunate, that's the current behavior. From a standards' perspective this is a grey area - the signature on JWT is still valid and the public key contained within the certificate is still valid. It may be reasonable to assume that if a certificate acting as a container for public key has expired, Salesforce acting as the oAuth authorization service should not be able to use this public key to verify the signature on the incoming JWT. While this might be a common sense conclusion it's not explicitly spelled out in oAuth or JWT specifications.
According to the doc, you do get notified when the cert expires on the connected app. We have not seen this happen, thus suggesting workarounds below.
One workaround to lack of notification on expired cert is to upload the very same cert to Certificates and Key Management. Unfortunately you can't just upload a cert on its own, Certificates and Key Management (CKM) requires a keypair. You could import a keypair from a keystore in JKS format or generate a signing request via CKM, have your CA sign it and then import the result.
Another workaround is to create a process that looks at the cert attached to the connected app, determines if it's expired and sends a notification. This is far from trivial but doable.