Users (and super users) of the Self Service Portal can submit and view attachments but can't delete.
How do I Add a delete attachment or remove attachment button to the "attachments list in the "view cases" page.
Yes, it seems at first, but I couldn't quite test your scenario as Self-Service is no longer available :-(