All Subscribers is NOT All Contacts.
You can be a Contact and not be on All Subscribers.
Triggered sends with "Add subscribers to this list = false" are one example. This used to be the "triggered send hidden list count", now the "Contacts from Triggered Send" in the Contact Count report.
So, as far as I have tried, Triggered Sends DO count against All Contacts.
Another example would be Synchronized Leads, Contacts, Users. Those are contacts from the moment of synchronization onwards, and not necessarily on All Subscribers, until sent to.
So, in triggered sends, you have to ensure that you either:
a) use a consistent SubscriberKey with other processes, if you have it.
b) give the users you want to send to some other key and delete that "temporary" key at some point afterwards - with all due diligence associated with deletion processes. In this scenario, it is actually easier to do add them to All Subscribers, so that at least you have transparency on what you are dealing with.
sidenote 1: There might be differences with the transactional messaging API. I cannot comment on this and would be intrigued to learn more myself.
sidenote 2: For testing, I tried to send a triggered send through the API where I used null or undefined as the subscriberkey. It would work, and wouldn't add more contacts with each send, but always stay at one record, an actual Contact with Key null / undefined. The problems started when several were done in parallel, then all sorts of crazy data mixups happened. Don't do this.