There's no one true path! But you've already touched on a couple interesting points like the historical availability of Visualforce on various devices.
I like to compare and contrast from the perspective of developer and user:
Apex and Visualforce:
- offers the lowest barrier to entry for developers,
- guaranteed forward compatibility on all browsers,
- powerful enough to cover 90% of business requirements,
- exposed to more devices in future without redevelopment,
Service orientation vs CRUD orientation
- CRUD is easy to handle in VF without ANY apex
High responsiveness or perception of responsiveness
- if you desire cutting edge snappiness < 300ms, you have to go stateless,
- non VF clients have more control over comms and UI, otherwise these are coupled,
Governor limit usage in the host org
- if you can't deploy code to the org, you must use the REST API, not VF
Future state of the platform
- Salesforce can already scaffold significant proportion of UI from data model,
- similarly, they could, at will, generate mobile-optimised outputs from your VF,
If you want the slickest, most responsive, latest and greatest interface for your special requirement NOW, you have to build it and maintain it.
If you are can't afford to maintain your interfaces, build them in VF. You will be happy with the consistency and usability.
If your requirements are highly specialised (drag and drop, two way sockets, pixel perfection according to your designer etc) you have no choice but to go full custom.