The Data Import Wizard is governed by a few permissions. See What permissions do I need to import records? for more details, but basically, if you want to restrict all access, you can remove the "API Enabled" permission for users. This permission will also prevent users from doing pretty much anything outside of the UI, so you'll have to test it thoroughly.
However, the Data Import Wizard is governed by a bunch of other permissions, such as "Import Personal Contacts", "Modify All Data", "Import Leads", "Import Custom Objects", as well as object permissions. All imports are governed by the user's sharing access as well. So, for example, without "Modify All Data", users could import data to their own records, but not other users'. This is generally as harmless as allowing Inline List Edit, allowing users to update their own data without risking everyone else's.
In summary, just make sure your users' permissions are correct, and they won't be able to do any more damage than they could do manually (if not a bit slower).