Specifically, is it faster to query for a single record from a Custom Setting vs a Custom Object? Or is it about equal performance?
This feels a bit like an X-Y Problem question since Custom Objects and Custom Settings have very different purposes.
Custom Settings are designed to be cached by the system; you don't directly query them yourself, but instead use APIs to get them. These settings will, therefore, be faster to access than SObjects. You should also note that these settings are intended to be largely read-only so it would not be a good idea to use these for transactional data.
There are various limitations with Custom Settings (both in terms of the types of field you can use and in terms of record management) when compared with both custom objects and Custom Metadata Types.
The latter were introduced as a replacement for many usages of Custom Settings, and have the benefit that their instances can be packaged or included in Change Sets.
It very much depends on exactly what your use case is as to which of these three you would use.