Your query is not specific enough. There are some types of shares you cannot manually delete, such as Owner, Team Member, Sharing Rules, etc. You can try deleting them with the allOrNone=false option to delete as many as possible, or filter the query to include only Manual shares (these are the only types you can directly delete).
The user who is running this class/method must not have the permission to 'Edit' this field.
Make sure the running user has 'Manage Users' system permission. But this is a powerful permission. Check details here