Did you mean \w*? Currently, your regex is returning a pattern (true) if it finds whitespace then a nonword character...your use case is perfectly reasonable for regex and if it's exceeding runtime caps then it's likely your expression.
I tested this and it works fine, even with multiple spaces:
NOT(REGEX(TRIM(Name),'^(\\S+\\s+){0,79}\\S+'))
We want the entire expression to evaluate to true (to throw the error) - so for NOT(REGEX(A,B)) we want string A to match expression B if A is a valid input.
Patterns:
TRIM(Name) -> eliminate initial and ending whitespace from text area string
\S+ -> finds a 'word,' defined here by any character except whitespace, one or more times
\s+ -> followed by one or more whitespace characters
{0,79} -> repeat this 0 to 79 times
\S+ -> include the last word
For another interval of n to m words inclusive, you would just make the repeater {n-1, m-1}... e.g. if you wanted 3 to 5 words, it would be NOT(REGEX(TRIM(Name),'^(\\S+\\s+){2,4}\\S+'))