If you have My Domain set up (and, if you are configuring SAML, you should!), you can configure Salesforce for Outlook with the domain (e.g. mycompany.my.salesforce.com). As described in the docs,
[...] click Change URL and pick the server to which you want to
connect. If the server you want isn’t listed, select
enter the URL, such as a custom domain used by your organization.
Now the plugin will use the custom URL to do OAuth, rather than the default login.salesforce.com. The Salesforce authorization server will recognize the incoming My Domain hostname, look up the org config, and redirect to the SAML identity provider (in your case ADFS) appropriately.
The process is well described in Single Sign-On for Desktop and Mobile Applications using SAML and OAuth.