hmmm. well, can you tell how deep the rocks extend? and can you tell how they affect drainage?
haven't spent much time in Pasadena, but it's fairly warm and dry, yes? if you put enough stuff on top of those rocks, I think you could get away with leaving them there. might depend on what you intend to grow. maybe add some woody stuff on the bottom for a nutrient/water sponge (hugel). you could also try some deep tap-rooted plants to loosen the rocks up a bit. won't happen in a season, but given time you'll see an improvement.
on the other hand, there are plenty of other uses for those rocks if you can get them out of the ground. rock piles are great for critter habitat, condensing
water out of the air, moderating temperatures. they can look real nice if you've got a good eye for building the piles (I don't).