You can't reliably use LTA, RTA, or ETA fields (Long, Rich, Encrypted) in formulas, but you can in Apex Code. My solution would be simply:
trigger CheckWorkCount on SObject (before insert, before update) {
for(SObject record: Trigger.new) {
Integer count;
if(record.RTA__c != null && (count = record.RTA__c
.normalizeSpace() // Don't count duplicate spaces
.stripHtmlTags() // Remove all HTML
.replaceAll('\\S','') // Remove all non-spaces
.length()) > 99) { // More than 99 spaces is more than 100 words
record.RTA__c.addError('Please limit your input to 100 words or less. You currently have '+count+' words.');
}
}
}
I don't know of a non-code way of doing this, unfortunately.
If you want to store the count in another field, you can do that as well.