According to this post he used to access the managerId field on user via a formula field,
why we cannot access this field directly? Is there a permission to control the accessibility via permission-sets or profiles?
Formula fields works regardless of FLS in Salesforce. From Salesforce.
Roll-up summary and formula fields are read-only on detail pages and not available on edit pages. They can also be visible to users even though they reference fields that your users can’t see. Universally required fields appear on edit pages regardless of field-level security.
Since manager is a lookup to USER table which is a setup object, I believe you may requires View All Users or Manage Users permission to view this field.