Deleting uploaded files using a custom delete button

Question

I have a custom lwc component to upload files to a record. I used Lightning file upload for this. And after files are uploaded I show them in the form of a datatable with a download button for every file download. I am also having a delete button for every file to delete. I have written the logic but I am getting an error field integrity exception when trying to delete them.

deleteFile(fileId) {
        deleteFile({ fileId })
            .then(() => {
                this.files = this.files.filter(file => file.id !== fileId);
                this.dispatchEvent(new ShowToastEvent({
                    title: 'Success',
                    message: 'File deleted successfully',
                    variant: 'success'
                }));
            })
            .catch(error => {
                this.dispatchEvent(new ShowToastEvent({
                    title: 'Error deleting file',
                    message: error.body.message,
                    variant: 'error'
                }));
            });
    }
}

There is an apex controller to this too.

@AuraEnabled()
    public static void deleteFile(String contentDocumentId) {
        if (!hasDeletePermission()) {
            throw new AuraHandledException('You do not have permission to delete this file.');
        }
        
        System.debug('Deleting ContentDocumentLink with ID: ' + contentDocumentId);  // Debug log
        
        ContentDocumentLink[] filesToDelete = [SELECT Id FROM ContentDocumentLink WHERE ContentDocumentId = :contentDocumentId];
        // Check if there are any links to delete
        if (!filesToDelete.isEmpty()) {
        // Delete the ContentDocumentLink
        delete filesToDelete;
       }

       List<ContentDocument> docToDelete = [SELECT Id FROM ContentDocument WHERE Id = :contentDocumentId];
       if(!docToDelete.isEmpty()){
        delete docToDelete; // This will delete the actual file
       }else{
           throw new AuraHandledException('No ContentDocumentLink found for the given ID.');
       } 
    }
   
   private static Boolean hasDeletePermission() {
    // Check if the user has delete permission on ContentDocument and ContentDocumentLink
    return Schema.sObjectType.ContentDocument.isDeletable() && Schema.sObjectType.ContentDocumentLink.isDeletable();
   }
}

Is there anything mistake in my code? I think the logic is fine. But I am getting this error field integrity exception while trying to delete them. Can anyone help me with this?