I ran into the sam issues and solved them by resorting to a Visualforce wrapping my Aura component:
- Quick Action UI not rendered as modal -> fake modal look with
stencil images
- Inconsistent Navigation behavior -> Redirect using more powerful Apex

Wrapper VF page
<apex:page standardController="Account" applyBodyTag="false" extensions="EditAccountCtrl" tabStyle="Account" recordSetVar="accounts">
<apex:includeLightning />
<body><div id="addHere" /></body>
<apex:form >
<apex:actionFunction action="{!redirectToList}" name="close" />
</apex:form>
<script>
$Lightning.use("c:myApp", () => {
const accountIds = {!serializedAccountIds};
$Lightning.createComponent("c:editAccounts", {accountIds}, "addHere");
});
document.addEventListener("showToast", (event) => {
sforce.one.showToast(event.detail);
});
document.addEventListener("close", () => close());
</script>
<style>
body {
background: url("https://res.cloudinary.com/hy4kyit2a/f_auto,fl_lossy,q_70/learn/modules/lightning_design_system/lightning-design-system3/images/9830e3e95f11941fe2d0f3a14230f714_unit3-listview-wireframe.png") no-repeat;
background-size: cover;
}
</style>
</apex:page>
and a Controller class which is better able to redirect to the caller. In my case its a hardcoded list view but I think you can also find out what was the original page.
public with sharing class EditAccountCtrl {
private Set<Id> accountIds = new Set<Id>();
public String serializedAccountIds { get; private set; }
// CONSTRUCTORS
public EditAccountCtrl(ApexPages.StandardController ctrl) {
accountIds = new Set<Id>{ ctrl.getId() };
serializedAccountIds = JSON.serialize(accountIds);
}
// PUBLIC
public PageReference redirectToList() {
return new ApexPages.Action('{!List}').invoke();
}
...
}