Is it possible to return zip file via REST API?
Possible, yes, wise, no. Neither Apex nor the Lightning Platform is especially well-suited to this kind of binary file manipulation, but some very clever people have built a library to manage ZIP files in Apex. (Disclaimer: I have not used it myself).
Are there any limitations to the above logic?
Yes. All of the ordinary governor limits apply. In this case, I suspect Heap Size may be the first one to bite you, since the string manipulation the library above must make in order to interact with the binary file content and perform compression is not memory-efficient. Or fast.
I would encourage instead returning from the API a list of
ContentVersion Ids and having the remote system send a separate request to the native Salesforce REST API to retrieve the body of each, or implementing this API endpoint on a middleware platform that has fewer governor-style constraints.