I made everything according to ( https://developer.salesforce.com/docs/atlas.en-us.lightning.meta/lightning/ref_lightning_overlayLibrary.htm ) SF dev page.

there is

<lightning:overlayLibrary aura:id="overlayLib"/>

in my component ( or < lightning:notificationsLibrary aura:id="notifLib" / > ), controller has code:

   var c = component.find('overlayLib');
    c.showCustomModal({
        header: "Modal header section",
        body: "Content of Modal window",
        showCloseButton: true,
    });

however, when i run this code, i can see error in a console:

TypeError: $A.J.Vg(...) is undefined

and nothing is displayed. showCustomModal returns a promise, so something is happening:

Promise { "pending" }
<state>: "pending"
__proto__: PromiseProto
catch: function catch()
constructor: function Promise()
finally: function finally()
l: function then()
then: function then()
Symbol(Symbol.toStringTag): undefined
__proto__: Object { … }

other Lightning related things work (components, controllers, etc.)

How to use Overlay/Notifications Libraries ? How to make them visible ?