Can I have a public APEX REST API e.g. APEX REST api that is accessible without authentication?
Yes you can. You will need to configure a public site and make the webservice available through it. There is an excellent tip on the developerforce blogs explaining it in more detail.
I implemented this a few months ago and ran into some issues, but did get it running :). The biggest of which is that your request must have the proper Content-Type Salesforce restful webservices reject
x-www-form-urlencoded and refactoring the legacy system was a bit of a problem.