No, you won't be able to combine those two things. One of the core features of Lightning is that you cannot reach across namespaces to interact with the DOM entities rendered by other components, including standard components like <lightning:dataTable>
.
This is protected by Lighting's Locker Service. In particular, see DOM Access Containment:
A component can only traverse the DOM and access elements created by a component in the same namespace. This behavior prevents the anti-pattern of reaching into DOM elements owned by components in another namespace.
It will not allow your components, in the local c
namespace, to access DOM components rendered by the lightning
or aura
namespaces.
You can use either <lightning:dataTable>
or jQuery Data Tables, but not both as part of the same table at the same time.