Plants for gravely soil

We pulled out the bank of forsythia on the southwest side of our pool and planted a line of apples and pears to become an espalier. Due to the gravely fill that was used to bring the hillside up to build the pool, each tree almost sits in its own bucket of good soil which they have been dealing with just fine for two years now. I'm interested in suggestions for what to plant in the gravely soil as so far all I have done is allow weeds to grow and a bit of mulching and compost added to the trees. Weeds that I have observed include jewel weed, wild strawberries, brambles, and a bit of vinca minor.

Beans do well in gravel that warms up sufficiently. If they can get water and light, they'll produce beans and nitrogen. Alder is another nitrogen fixer that thrives in gravel.

Mullein grows on gravel roadsides. Many Mediterranean herbs are good with heat and drought, two things that gravel is known for.