There's an object you can interact with SetupAuditTrail. As such, a SOQL query can retrieve what you'd like and a scheduled Apex class could presumably make this easier to alert you of changes or send out information.
SELECT Id,Action,Display,Section from SetupAuditTrail
I see an example of leveraging this in AuditLogScanner.
However, aggregate queries are not supported so you couldn't group the above by a created date to point to any deployment. You could
ORDER BY CreatedDate to assist with that linking.
However, I think the above has a lot of risk if you have strict requirements with auditing. Ignoring the need to create something custom to support even doing the above - you are indebted to relying completely on the Setup Audit Trail's functionality. It may not be tracking things you need to track (ex. report creation).
If you have strict auditing requirements, you should look towards "how" you get your changes into production - not reactively figuring out what change goes where in your external system.
Your statement that deployments "can never be considered a single-source-of-truth for changes" raises questions on your development/change process. You may not be able to jump into using git depending on your current process, but you should look to have a better view of your metadata since you do have a requirement of controlling/knowing what goes in.
It's much easier to tag your external ticket number in a git commit or, if that's an unrealistic pivot at the moment, a deployment history that a (paid) deployment tool may provide.
In terms of the "exceptions", emergency changes may go quicker into production, but should still follow a process that fits into your change management system/deployment pipeline. Inadvertent changes should happen rarely and if they do not, you should re-evaluate why they keep occurring and possibly think of lessening how many people can do significant changes in production if it's a cause for concern.