Duplicate criteria could vary from business to business, if you are not building an app-exchange app, this could be confirmed with relevant guys in the client's org. If the app is going to hit app exchange, then this duplication criteria / fields etc could be kept in custom settings, so that customers can tweak it.
Apart from that, I fear governor limits as a big challenge, things that you might need to consider, like:
- Heap size, if string (CSV) is too huge for normal apex runtime heap(6 MB) limit, remember other apex classes and structures will share the heap as well.
- "Total number of executed code statements" if you are looping and doing checks on each row, this limit could be easily touched.
- Plus other limits of describes / soqls fired etc, depending on your logic.
I am not getting into design as not sure about the end product, but splitting the CSV into chunk of rows and processing these chunks via Batch Apex etc could be something to consider if you are hitting too many limits. Again this depends, if your clients are fine with async interface from UX standpoint i.e. end users submit CSV:
- If chunk could be processed immediately, parse and show user next screen.
- Else tell user that you're processing the CSV and will notify when done. Here you can use batch apex etc to process the huge file and notify user's via PushTopics or email.