Are you sure the rich text field is visible to the site profile? It's not included in the regular list, you have to go to the site settings and access the profile from there where you can then check whether the field level security is good. I assume it is since you have the URL , but then I don't know why you can't access the image. Have you tried using straight tags instead of using the Visualforce image tag?
We use rich text fields for our website and that displays images from the same server without problems; the image below is accessed via this URL:
I suspect that you might need to modify the domain part of your image URL to use that of your site, as it is the URL will be for internal access and the site user wouldn't have access to it. You can get the site base URL using: {!$Site.CustomWebAddress}, for example I get the URL of static resources on our site like so:
{!$Site.CustomWebAddress}{!$Resource.ResourceName}"