As per documentation you are not allowed to use RichText fields in formula https://help.salesforce.com/articleView?id=000329422&type=1&mode=1
If you want to calculate length of data, I would suggest to use trigger for it.
You can simply use length function for it just make sure to sanitize your rich text field first. For example like :
Remove html tags using : stripHtmlTags()
Dont count multiple spaces : .normalizeSpace()
If you dont even want to count white spaces you can use regex to
replace all white spaces with '' or you can use deleteWhitespace()
function introduced for Strings.
Example :
Assuming field is on account
for (Account a : trigger.new){
String temp = a.fieldName__c.normalizeSpace().stripHtmlTags();
Integer count = temp.length();
}
Now after having the count you can save it in some other field or throw an error depending on yourscenario.