I have installed a managed package that comes up with a protected custom setting.I have its name and other information. For some testing purpose, I need to update its record in the org. I have tried to do the same using some Apex code, but it didn't work out. Is it impossible to accomplish the task I am trying to do?
No, protected Custom Settings aren't updateble from subscriber organization. It is updateble only from the same namespace.
If Privacy for a custom setting is Protected and the custom setting is contained in a managed package, the subscribing organization cannot edit the values or access them using Apex.
Refer to documentation
A protected custom setting that is part of a managed package can only be modified by Apex code that is within that managed package.
From the Access Custom Settings documentation:
If Privacy for a custom setting is
Protectedand the custom setting is contained in a managed package, the subscribing organization cannot edit the values or access them using Apex.
The only exception to this might be if the owner of the managed package did an LMA application subscriber support login into the org.