Edit: Ok I didnt fully understand question, so I'm adding new answer with old one staying just for reference.

You can of course create footer for your modal with buttons that will handle modal close.

Example mainly from the docs:

Modal footer component: 

    <aura:component>
        <lightning:overlayLibrary aura:id="overlayLib"/>
        <lightning:button name="cancel" label="Cancel" onclick="{!c.handleCancel}"/>
        <lightning:button name="Close" label="Close" variant="brand" onclick="{!c.handleCancel}" disabled="true"/>
    </aura:component>

Footer controller:

    ({
        handleCancel : function(component, event, helper) {
            //closes the modal or popover from the component
            component.find("overlayLib").notifyClose();
        }
    })

Creating modal controller:

    ({
    handleShowModalFooter : function (component, event, helper) {
        var modalBody;
        var modalFooter;
        $A.createComponents([
            ["c:modalContent",{}],
            ["c:modalFooter",{}]
        ],
        function(components, status){
            if (status === "SUCCESS") {
                modalBody = components[0];
                modalFooter = components[1];
                component.find('overlayLib').showCustomModal({
                   header: "Application Confirmation",
                   body: modalBody, 
                   footer: modalFooter,
                   showCloseButton: true,
                   cssClass: "my-modal,my-custom-class,my-other-class",
                   closeCallback: function() {
                       alert('You closed the alert!');
                   }
               })
            }
        }
       );
    }
    })

I on purpose made one button disabled in a footer so you can enable that in your code



I assume that modal is a child component.

You need to send component event from child to parent and in parent event handler close the modal.

and of course modalBody is your already created modal

-------------------------------------------------------------

In Parent

    <aura:handler name="myComponentEvent" event="c:componentEvent" action="{!c.handleMyComponentEvent}"/>

Parent controller

    ({
        handleMyComponentEvent : function(component, event, helper) {
            component.find("overlayLib").notifyClose();
        }
    })

You can optionally stop propagation of the event here using

    event.stopPropagation();

if you do not want your event to propagate further

In child

    <aura:registerEvent name="myComponentEvent" type="c:componentEvent"/>


    handleUploadFinished: function (component, event) {
        var uploadedFiles = event.getParam("files");
        console.log(uploadedFiles[0].documentId); // See? it works.
        var myEvent = component.getEvent("myComponentEvent");
        myEvent.fire();
    }

And event itself

    <aura:event type="COMPONENT">
    
    </aura:event>

Obviously with desired event name, in example 

    componentEvent