The larger rocks, unless buried very deeply under good soil, will forever make it hard to dig for planting, and possibly reduce the amount of
water that can be held just beneath the surface. In the desert climate, I have a feeling that piles or rows of rock around your
gardening patches would help cut drying wind, shade the ground surface a bit, and help keep good soil from blowing away as dust. If you have real winter, a crescent of rocks around the northern/windward sides of a garden patch could make it warmer in spring.
I would not try to remove all the rock (especially the gravel visible in the pictures), just make a plan and toss rocks from the centers of designated garden patches to surrounding rocky strips. How large is the area in question? If it is the side yard by a house, it seems like a feasible manual task which can be done gradually, improving one section at a time.