For the Favicon to work, I had to upload my image to the orgs assets library: Under Branding>"Select an image to generate a palette" :
Only then was I able to reference my icon, which I had to upload as a 16x16 png image.
Finally, I was able to reference it using the file-asset's relative path as follows:
Notice that the icon name when I reference it has a versioning appended to it,
<link rel="icon" href="/file-asset/coveofavicon?v=1" type="image/png" />
You can use your browsers dev tools to identify the relative path for your community assets, for reference purposes, here is another link provided that helped resolve this:
<link rel="icon" href="/sfsites/c/file-asset/Favicon_FileName?v=1" type="image/png" />
I did not find this documented anywhere =(, hope this helps other people.
This will change some of your community stylings, since it will detect the colors your image/favicon uses, however, this is easily reversible by re-selecting the original colors you had under the branding option.