If I want a specific field to be visible only to a specific user or group of users on a layout, I set up field-level security for this field. I want to be able to do the same for my custom button added to a layout. I want a specific button be visible only by a specific group of users.

Is there a way to set a 'button-level' security in order to achieve this? It would be perfect if I could use a permission set.