Scale insects: should I do something about them?

Agck, I found some scale insects on our apricot trees. Tiny bark-colored hard-shelled blobs that I can scrape off, or squish with a stick. Ants are interested in them. It's not a heavy infestation now, but should I be worried? Should we do something about it or let it be?

A couple possible solutions come to mind:

More Permaculture: Plant more guild/insectary plants around the tree and wait for natural predators to arrive, and/or for for the guild plants to enhance the tree's health & resistance to the scale.

Strandard Horticulture: Spray dormant oil.

I don't know how apricots are in your area, but if I had a similar problem I would probably just wait for the predators... apricots are incredibly tough here in Kamloops.

We've never sprayed anything like dormant oil so I guess we'll just watch and wait, and I pick off the little barnacles as much as I can though parts of the trees are too high to reach. It's not a really intense infestation (yet) so I hope it just adjusts into the natural balance of things. Apricots are very hardy around here.

Do scale insects damage fruit or only weaken the tree?