First of all, All Contacts contain mainly:
- All contacts from All Subscribers
- All contacts from Mobile Connect that are available in all your bus
- Salesforce Contacts
- Salesforce Leads
- Salesforce Users
For a full list check the following article: Contact Definition and Count Determination. Contacts are counted regardless of the fact if you tried to contact them or not.
There's also a nice article that shows how to remove unwanted customers:
Permanently remove Contacts or Subscribers and associated data from Contact Builder
Analyzing All Contacts
If you change the process described in the last article, you can get the list that contains every contact in All Contacts:
- Create a filtered list in Mobile Connect with a single condition
Contact Key is not null
- Create a data extension with a single "SubscriberKey"
- Query the filtered list and send the results to your data extensions:
Select SubscriberKey from [Your MobileConnect list name from step 1]
After the query completes you can anylyze the DE in any way you want - either by queries or externally in whatever software you choose.