Are there any design requirements/restrictions for an AppExchange app? We would like to create a popup, which will work a bit like a chat screen that some web shops use, but smaller. It would be completely hidden unless there is a message to display. Is that allowed in classic/Lightning? I couldn't find any documentation on it.
Yes, as far as passing a security review, the Security Review FAQ places limits on what you can do with your app. Perhaps the most relevant part for you is the following section:
Q: I failed my security review because of “JS in Salesforce DOM". What does that mean and how can I get around it?
As long as you can build something that does not violate the security rules, you're free to do as you will. Without knowing more details, it's hard to tell you exactly which way you'd want to go, but I suspect you'll want to make something that can integrate with the Chatter API so that Salesforce users can get notifications on the device of their choice.