Aura (Lightning's underlying engine) runs in two modes: system-mode and user-mode (or, "trusted" and "untrusted"). Trusted code is written by salesforce.com and boots up before any components are loaded. It has full access to the entire DOM. Untrusted code, in contrast, is written by someone that isn't salesforce.com, and only has access to its specific shadow DOM. This isn't domain-based, but rather DOM-based. Each component can only see what it has access to within its current DOM, but may communicate through approved channels (namely, Aura events).
The determining factor if a library will work or not is if it conforms to well-defined behavior in regards to the DOM. If it tries to step outside of its own shadow DOM, it will be blocked by the security measures of Locker Service. The allowed operations are defined by if they can be used to subvert the security measures in place. Security doesn't literally check every single library and individually stamp a seal of approval. After all, a developer could tweak the code slightly by using conditional compilation, etc. Instead, they analyze the code statically to see if it works as is, and if not, they add exceptions to allow the library to work, as long as it would not subvert security.
As a developer, we can see the secure API that outlines what a library can or cannot do. As long as it obeys the rules, it is free to operate as it desires. In addition, CSP (Content Security Policy) prevents arbitrary loading of resources, which is why there's an additional caveat that the code must live in a static resource or inside a Lightning bundle. Security has tested several different libraries outlined here. The list is not comprehensive and doesn't even guarantee other versions of the library may or may not work. You can write your own libraries, as long as it conforms to the security outlines, and it'll run just fine.
In many cases, sometimes a recent tweak to security may render a library unable to load. In these cases, sometimes the library needs to be patched, or the security rules reworked to allow the library to run normally. In addition, even if the library is perfectly solid, some extensions/modules may violate the rules, and they won't work. For example, jQuery works just fine, but if you try to use module XYZ, it may not work. Or, the library works but the user's code is at fault.
So, as far as things go, developers interested in what's allowed should check the Locker Service API (above). By using only approved properties and methods, the library should work seamlessly. It also gives you a decent idea of things to look for when things go wrong. You can catch access exceptions if you're not sure Locker Service is enabled or not, but there's no general purpose API that just tells developers if Locker Service is enabled. Developers should always develop with Locker Service enabled to make sure they're in compliance.
There's an image located here that basically describes how things work. For today, components are sandboxed by namespace, but the article also outlines that components will most likely be sandboxed per-component in the future. Components should always strive to respect their own borders and use only approved methods for communicating outside of its own DOM.