I don't believe that Salesforce provides a REST resource to do what you're asking for.
You may be able to accomplish this by (ab)using the List Views resource
/services/data/v52.0/sobjects/Account/listviews/00BD0000005WcCNMA0/results?offset=[your value here] as there is no documented limit on that offset parameter like there is when you use OFFSET in a query, but the results of that call are not the most friendly to work with and you're limited to 15 fields. A comment on this other question suggests that the max offset here may still be 2000.
Realistically, you'll probably need to expose an Apex class as a REST Resource. You could set up a GET method that accepts a few URL parameters, and feed that to an instance of the StandardSetController class to handle returning the paginated results (which can handle things up to 10k rows). The
getRecords() methods are the ones you'd be primarily interested in.