Without getting terribly technical, Visualforce is essentially implemented as JSP; the Visualforce markup is compiled into JSP and run by the Java runtime that renders all of salesforce.
The code runs at about the same speed as native JSP pages, which isn't saying much. You can expect a maximum speed of about 100-200 ms for simple pages. In practice, any page that's worth doing will take close to a second from the initial request to the final rendered state.
The remaining time includes sending the view state, deserializing, serializing, and rendering, plus the additional bandwidth costs, which are most noticeable on slow or laggy connections.
The new, experimental server-side view state feature should bring the entire lifecycle down in response time, but I wouldn't consider that a panacea for all slow Visualforce pages.