As you have separate visualforce page for managing users, you should be able to achive this, by moving the logic for manage users (deactive, create, edit and reset passwords) to a separate 'Without Sharing' apex classes .
there is no way to give the user the permission Manage Users without seeing Setup option