I want to add some custom logic to the upload attachment feature when attaching items to cases. Is it possible to customize this form, or is it possible to customize the button that is display in the attachments section of the Case form?
You can also override the entire Case View page and add a custom VF component that manages a Notes and Attachments 'related list' with your own custom logic.
one caveat -- if you are using Google Docs in addition to Notes and Attachments, you will not be able to use your replacement VF page to upload new Google Docs and associate to the Case; you will be able to associate existing Google Docs to the Case. See New Google Doc fails when used in VF page (known issue)
Unfortunately salesforce does not provide an option to configure attachments section. The only way I can think is to create a inline VF page on the Case detail section and show attachments related to the case and add your own custom buttons with custom logic.