Why I can't create relationship between custom object and custom setting. When try to create lookup field in my custom object, custom settings are not listed in available objects.
No, you cannot create a relationship from a
Standard Object or a
Custom Object to a
Custom Setting (nor the other way around).
As for why, the simple answer is that it is a platform limit. They were designed with this limitation in place, and it is a constraint you have to keep in mind when selecting
Custom Settings as part of your solution.
From a technical perspective, I imagine part of the reasoning was to help the cloud architecture Salesforce has put in place scale to billions of transactions per day. As you may know,
Custom Settings are stored in the application cache:
All custom settings data is exposed in the application cache, which enables efficient access without the cost of repeated queries to the database. This data can then be used by formula fields, validation rules, flows, Apex, and the SOAP API.
Injecting data dependencies to or from this layer of the application would increase the technical complexity of implementing that caching mechanism, at the very least, if not make it impossible.