You have to use: Contact.Phone
<apex:outputText value=" {!Case.Contact.Phone}"></apex:outputText>
as ContactPhone is available from Contact, basically it behaves as a formula field taking value from the Contact record.
Again, if the Visualforce is using StandardController of Case it works fine
<apex:outputText >{!Case.ContactPhone}</apex:outputText>