There is no native code that you can just call to determine if a file is password protected. Password protection is implemented in different ways in different file types. For example, ZIP and PDF use two different schemes.
You can read the specifications for a file type to figure out how detect password protection, but it will be a manual process. For example, I invented a way to detect signed PDFs in native Apex. It's very manual, and limited in file size, but it is possible.
Calling an external service is generally ideal, because otherwise you're having to reinvent the wheel for each file type. That said, it's mostly a matter of figuring out the correct signatures to look for, and then using a relatively simple search function. Most of the time you'll spend is researching an appropriate signature to look for.
Some file formats may have specific bits/bytes set at specific locations you can look for, so it might be easier to detect password-protected files of some types with just a few lines of code. Other systems encrypt key bytes, so you might also be able to just determine if those bytes do not match the expected signature, and you'll know it's encrypted (or not the right file type).
