Garden where pool was

To till or not to till. Last year our above ground pool was taken out along with a deck and buckets of rock. Top soil was brought in and just dumped on top and few veggies popped up like chard, tomatoes, lettuces. That was before my permie obsession. Digging up this year there's around 3in dirt then 2in pure sand on top of pure clay and some dirt with gravel. The water doesn't drain at all and there's still rock dispersed throughout the 300square ft area.
As a beginner and it being spring now I'm not sure what to do. The whole area is so compacted. I intend to have some trees, perennial and annual veggies. Should I have it all removed or mixed together?

I would maybe dig or even double dig the whole area once and after that hoe only lightly.

What is yur location, Sara Ann?

Yeah I think i'll dig it up instead of depending on master mulching it. I just wasn't sure whether to remove some some of the fill from the area since there's landscape rock leftover. And how deep to dig it since the area is too large to do by hand. I was concerned about the impact of the pool that was there for twenty years - compacted dirt, chemicals... (sorry mother earth)

I'm south of Chicago

I suggest the Back to Eden approach. If you haven't watched the documentery, you can find it online searching on that phrase. You simply need a good amount of woodchips or some other similar mulch.