You can use record types for this purpose. Here's what you'll need:
- Two record types
- Two page layouts
For simplicity, we'll call them the Locked and Unlocked record types and page layouts.
In the Unlocked page layout, remove the button, and in the Locked layout, add the button. Also remove the Record Type field from all layouts, or make it read-only so users can't manipulate it.
Assign the Unlocked page layout to the Unlocked record type, and assign the Locked page layout to the Locked record type. Assign the Unlocked record type to all profiles, and assign the Locked record to none of the profiles.
In the approval process, initial submission actions should include a field update to change to the Locked record type. In the final rejection and final approval steps, set the record type back to the Unlocked record type.
Now, when users initially create records, the unlock button will not be visible, but when they submit for approval, the button will appear, and when the record leaves the approval process, the button will again disappear.
Alternatively, you could write a Visualforce detail page, but that'd be a lot of work just to display/hide a button.
Personally, I'd just as soon give the user an error if there's no point in clicking button. It's a lot easier to write a couple of lines of JavaScript than to go through all of this customization just to show/hide a button.