For a simple reduction in latency/load times, you need to deal with "Ahead-of-time (AOT) compilation" (AOT) and "tree-shaking." Basically, Angular bundles tend to be large in size, and those sizes will affect your component's loading time. The thing about LWCs is that the runtime is small, and the component bundles tend to be small. I would argue that, aside from some technical differences, like preferring Material to SLDS, a total port from insert framework here to LWC is almost likely worth the cost in development time.
Canvas apps simply cannot load before the parent page loads. That's the nature of iframe apps. If you want to spend time fixing the canvas app, consider webpack-bundle-analyzer, which helps identify which bundles are causing the most problem. The effects gained by such an effort will be limited by the effects of AOT compilation and tree-shaking. A "total rewrite" sounds daunting, but the efforts more than pay off for themselves.
My 2021 role had me working on a project that was a complete React app. It consisted basically of 2×5MB bundles that had to load for every page. It was slow (5+ seconds per page load) that was barely even helped by cache. I rewrote most of it in LWC, and got the page load times down to 0.2 seconds, including the Lottie animation we had for loading. I honestly almost recommended removing it because it was adding to the load time. It was ultimately rejected as being "too expensive" even though it would have saved our customers minutes per day just in loading times.
You can, of course, load React or Angular onto a component marked with
lwc:dom="manual", and it works pretty well. Removing the canvas means your code can load in parallel, often saving up to a second or more in timings to load the components. However, I'd argue that this is still less efficient than simply porting to LWC. LWC is, simply put, one of the best frameworks to use in Salesforce, since it's already used in Salesforce.
I'm wondering if there are some simple things we could do to reduce the latency vs. a total rewrite.
I've already addressed this, but ultimately, porting to LWC is the preferable option here. Your customers will thank you for the port.