Apps that optimize images, custom reports, shipping integrations all of those and many more will not affect your store performance. The troublesome are the ones that use ScriptTag to load remote JavaScript code without using the theme templates. The idea is good, is simple, reliable and easy to maintain but can lead to performance issues.
Let's discuss about the applications that are injected into your theme from a developer point of view.
The good
Shopify apps provide an easy way to extend your store functionality. A store owner can easily install an application within minutes. The costs are really affordable comparing to custom development. I can really see the appeal here. The default loading is on the "onload" event, this fires when the entire page loads, including its content (images, css, scripts ...). Makes sense in most cases. Some apps have a low impact on the overall performance.
The bad
We do not have control.
Let's say we installed an upsell application that is needed only on the cart page. The application JavaScript will be included on all store pages through ScriptTag. Of course this is not the desired behavior, we are loading much more than we need ending up with a bloated payload.
The onload event execution can also be problematic. Sometime you want your application to kick in early and not wait for the whole page to load. One example would be a geo IP redirect or age verification.
Being on different servers the browser will need to resolve a new DNS for each of these assets. This can take time. You will be loading those JS files from the vendor server, not the top notch Shopify CDN. There is no service guarantee here, that server can be slow or even down.
The ugly
This is the part where you need to be concerned.
I've seen applications that inject a huge amount of code, really communne to see anywhere between 100Kb to 400Kb. With just a handful of apps you can easily reach 1Mb of code. This counts. The JavaScript injected can also bring in additional assets like web fonts or images. Do you want to load Open Sans for your reviews app? Probably not.
A bad approach is for each application to include its own jQuery version. Besides the fact that jQuery is not really needed in 2019 and adds additional payloads, I've seen many times compatibility issues between the theme jQuery and application jQuery versions. If your theme uses jQuery 3.1 and and application is loading jQuery 1.2 you can get into trouble. At some point on a client store we had 5 different versions on jQuery.
If applications are loaded on the onload event does it matters?
It does.
Web performance is a large umbrella covering different metrics like Time to First Byte, Time to Start Render or Time to Interactive. I doesn't matter only how fast your users can see meaningful content, how your site feels is also important. Javascript parsing, compiling and execution takes a big toll on the CPU. This happens on the main thread. While the main thread is busy crunching 5 versions of jQuery, user interaction is also blocked. Users can't properly scroll, click, add to cart, view the product gallery.
I wish Shopify would give both store owners and application developers better tools for dealing with 3rd party applications. display_scope could be used to fine tune where those applications are loaded. Multiple events can also help us to control onload, defer or async execution. But most important I wish application vendors will be more sensitive and responsible on the code they provide.