Although this question is already really old here is what I found out today by chance:
A user needs to have the Manage External Users permissions and must be included in a sharing group of the partner user's manager role.
If you navigate to a partner community user and click on the role, you see what role it reports to and in what sharing group the role is. I think per default this is "Role" & "Role, Role, Internal and Portal Subordinates". This means the user you'd like to have access to has to be in the reporting role or in a subordinate role of the reporting role
Example:

As you can see, the partner role, reports to the role "Geschäftsführung" and is shared within "Geschäftsführung". Due to our role setup, which has "Kundenbetreuung" as a direct subordinate role to "Geschäftsführung", all users in "Kundenbetreuung" which have "Manage External Users" can access that functionality.

Sub roles of a direct subordinate role of the role a partner role reports to do not have access. I had to learn that the hard way ("child" role to "Kundenbetreuung" in my example).