One of the idiosyncracies of accessing Profiles and PermissionSets via the Metadata API is that SF only provides permissions for items which are also being downloaded in the same request. That is, you will only get
<pageAcceses> for Visualforce Pages you are downloading at the same time, and only get
<fieldPermissions> for fields you are downloading at the same time:
The content of a profile returned by Metadata API depends on the content requested in the RetrieveRequest message. For example, profiles only include field-level security for fields included in custom objects returned in the same RetrieveRequest as the profiles.
(Source: Metadata API Developers Guide)
The best solution I know of is to also retrieve the fields, classes, pages, etc. that you want to download permissions for.