The simple answer, as already stated in the comments, is that you need to use double-backslashes to escape things in regex in Apex.
So insted of
String fileNameRegex = '/^\[~`!@#$%^&*()_+=\[\]\{}|;':",.\/<>?a-zA-Z0-9-\w\s\]+$/';
You'd need
String fileNameRegex = '^[~`!@#$%^&*()_+=\\[\\]\\{\\}\\|;\':",.\\\\/<>?a-zA-Z0-9-\\w\\s]+$';
Note that in addition to the doubling of backslashes, I also:
- Escaped the single quote in (what I'm assuming is) your character class that you're trying to build
- Escaped the pipe and closing curly brace
- Unescaped the square braces near the start and end
- Getting a literal backslash character requires 4 total backslashes in Apex
- The leading and trailing
/are removed (they aren't needed, and in fact make your regex non-functional)
That is sufficient to get this to run (though not necessarily produce the desired result).
It looks to me like you're attempting to detect printable ASCII characters. If that is the case, then you can make your life a whole lot simpler by using POSIX Character Classes.
i.e. String fileNameRegex = '^\\p{Print}+$';