I agree with Tyler. If the aphids are not attacking your choice plants, then they serve the purpose of giving the aphids a place to be. If you were to remove the plants that are being infested by aphids, then the aphids will find another place to be-namely on your choice plants. Also, the more aphids you have that are not harming your choice plants, then the more likely your garden will be accumulating some of the creatures who prey upon aphids, like ladybugs.
The more you keep old stalks (like from last years brassicas, for instance, in your garden (rather than removing them at the end of the season), the more habitat for ladybug eggs to be laid and hatched out in the spring. That is assuming that you have ladybugs in your area. I'm not sure what the natural predator of aphids is in your climate/country.
Either way, great to observe your aphids and see what, if anything, is preying on them. If you don't see the population being preyed upon at all, but it is stable, then it is likely that you are just not noticing the predator.
If the population expands and begin to affect your choice plants too much, then you might need to take action. Diluted garlic spray is supposed to be effective. Some people spray with a diluted (organic) soapy
water solution as well, with good effect.
It could be a sign that your garden has too many ants, since some species of ants farm aphids, but ants are almost always beneficial to your garden as they do much the same job as earthworms in many regards. It could also mean that you are nearing the end of that particular weed's cycle in your garden, and this is nature's way of beginning to take it out.