I strongly suspect that something is going sideways because you're not properly escaping the non-safe URL characters (like \
and [
); in a URL, they must be in a certain style known as "percent encoding". Here's how I'd approach your problem:
ApexPages.PageReference ref = new ApexPages.PageReference('https://api.example.com/output');
ref.getParameters().put('filter\\[abc\\]\\[\\]', 'xyz');
String theUrl = ref.getUrl();
req.setEndpoint(theUrl);
This produces output like:
https://api.example.com/output?filter%5C%5Babc%5C%5D%5C%5B%5C%5D=xyz
Things like "curl" are smart enough to do this for you, but in Salesforce, you always need to use either EncodingUtils.urlEncode or PageReference.
Edit: The URL doesn't have backslashes in it; the backslashes that were used are required by the shell for curl
to properly detect the brackets. So, simply removing those should fix the issue:
ApexPages.PageReference ref = new ApexPages.PageReference('https://api.example.com/output');
ref.getParameters().put('filter[abc][]', 'xyz');
String theUrl = ref.getUrl();
req.setEndpoint(theUrl);