Most probably that is caused by the olive lace bug, they
feed on the sap of the leaves and create the characteristic pattern of blotches your trees leaves have.
Here's the description from Olive Business:
Sap-sucking insect that reduces vigour of trees.
Flat with lace-like wings and is no more than 2mm in length.
Resides on underside of leaves.
Causes rusty yellow spotting on top of leaves.
Control includes keeping trees healthy.
You can spray neem oil that has been diluted with
water