Triggers are data centric, they are not aware of the processes around them and nor should they be, it would lead to to much strange behaviour in different contexts and confusion for end users. For example, what about use of the Salesforce API's for example, same problem. So couple of options...
Action Override for Clone. What you have here, is basically a client process requirement your trying to deal with at the database level. Which is causing some design side effects. Though in fairness we often consider these routes, due to the fact that there is no 'process' entry point to the client logic in Salesforce. While not 100% bullet proof (for example SObject.clone), you can in fact get close enough via an override of the Clone action on your Custom Object under the 'Standard Buttons and Links' section. This way you could implement a VF page to display just what you need and override the StandardController save method to clear the fields you don't. Leaving all other contexts intact and no need for the trigger, leaving your Data Loading experience as expected.
Control Field. is to continue with the route you have taken, that in my view is also not so bad. The trick to making it feel better is the naming and meaning of the field, ideally something users and devs will relate to in relation to the other fields it effects. So called 'control fields' serve only to adjust the behaviour of triggers. These typically default (via the field definition) to the most commonly desired behaviour. Thus in the majority of use cases (UI, API's etc) where your records are created or updated, they need not be considered by the user or developer. It is also good practice user experience wise (if your use case allows it) to throw validation errors if the trigger logic senses fields with values set or contain certain values that disagree with the 'control' fields indicated behaviour.
If you go for 2, I would invert the meaning of your control field, e.g. 'Clear None Cloneable Fields'. Default it to False in the field definition. From a UI perspective place it in a section with the fields it effects near the top (or hide it as you wish). In your trigger, if it's false, then flip it during the on before insert, so that it becomes set to true by the time the record is eventually saved. Later as users attempt to Clone records created this way, it will be carried forward and fire your logic.
Tricky stuff though to design, general rule of thumb is know and test all your use cases, typically all the UI actions / operations and CRUD combinations (you don't have to test multiple SF API routes here, Data Loader or even DML in tests will do). Enjoy!
P.S. If you want to try 1 and need an example let me know, I'll update answer.
