What you're looking for is the
TaskWhoRelation sObject, which is a read-only, filtered version of
TaskRelation stores all Task relationships, including the
What (Related To), while
TaskWhoRelation includes records only for
Who (Name), that is, for Leads and Contacts.
You can perform a subquery against this relationship, as shown in the linked documentation:
SELECT Id, Subject, (SELECT RelationId, Relation.Name, Type from TaskWhoRelations) FROM Task
or you can query it directly:
SELECT RelationId, Relation.Name, Type, TaskId FROM TaskWhoRelation
Note that because
RelationId is polymorphic, you won't be able to directly access most Contact or Lead fields through the relationship; you'd need to accumulate those Ids and perform a separate query against Contact or Lead.