Hmm, i'm certainly no expert on this topic so take my advice with a heap of salt.
In my observation, I've seen trees totally covered in aphids next to trees of the same species without a single afid on them. Likewise i've seen branches covered in aphids with all other branches untouched.
I do know that plants are able to make a metabolite that can kill off aphids by stopping their digestion, aphids are unable to attack plants that are sufficiently healthy and can afford to create these chemicals. There's a reason aphids don't just wipe out every crop, other than predators. Plants have many biological mechanisms of self defense, some include attracting predators but most include complex biological defenses.
I read about the digestion metabolite long ago but can't find the source anymore.
https://www.nature.com/articles/s41598-018-34821-w
https://www.mdpi.com/2223-7747/13/16/2332
You can read the abstracts above for some other mechanisms, the point is plants have defenses against aphids. Typically the plants that are covered in aphids are the ones that are the least healthy because they either can't spend the energy to create these expensive metabolites or can't get the correct nutrition.
If all of your plants are unhealthy, then you may have to worry about aphids spreading. If only a couple are unhealthy or only these smartweeds are, then the aphids aren't a big deal and attracting some of their predators might actually be beneficial.