I've encrypted phone field is there and it referred in formula field. I need to remove special character in the formula field (tried with SUBSTITUE) but it is not allowing to change because it's a encrypted field. Could anyone suggest how to do it.
As mentioned in the comments, SUBSTITUE() is not a supported Function for using Encrypted Data in Formulas.
There does not seem to be a direct/simple way to achieve what you are looking for. The workaround would be to create a physical field instead of a formula field, and populate it via trigger or workflow, then encrypt it.