What is needed here is to check each field, is it's not empty check if the other two fields are empty. As soon as one is not empty the validation should fail.
Following formula should get you on your way:
IF(NOT(IsBlank(Field1__c)) ,
OR(NOT(IsBlank(Field2__c)),NOT(IsBlank(Field3__c))),
IF(NOT(IsBlank(Field2__c)) ,
OR(NOT(IsBlank(Field1__c)),NOT(IsBlank(Field3__c))),
IF(NOT(IsBlank(Field3__c)) ,
OR(NOT(IsBlank(Field1__c)),NOT(IsBlank(Field2__c))),
true
) ) )
To put it in words:
If field1 is not empty -> check if field2 or field3 is also not empty
If field2 is not empty -> check if field1 or field3 is also not empty
If field3 is not empty -> check if field2 or field1 is also not empty
else none of the fields is set -> so fail because exactly one has to be filled in.
You can even simplify the logic because in the second step you already know that field1 is empty. So a simple check that field3 is empty is sufficient.
The third check is in this case redundant as field1 and field2 are already empty (previous checks told you so). Simply put false in case field3 is filled in.
So simplified:
IF(NOT(IsBlank(Field1__c)) ,
OR(NOT(IsBlank(Field2__c)),NOT(IsBlank(Field3__c))),
IF(NOT(IsBlank(Field2__c)) ,
NOT(IsBlank(Field3__c)),
IF(NOT(IsBlank(Field3__c)) ,
false,
true
) ) )