There's no telling what might have been in that shed, if it was a shed, without a test looking for persistent chemicals and heavy metals. That said, it could have been a
chicken coop!
Are there places in the lawn where there are not any bricks/concrete underneath? If there are, do you see any coloration differences in the grasses or broadleaf herbs that are growing there as opposed to over the bricks? If the plants look as healthy over the bricks, that
can be a good sign; but is definitely NOT definitive of purity.
All of that said, plenty of gardens have been built in worse situations.
The quality of living soil you produce, especially with organic matter (humus), and mycorhizal fungi, will aid in decontaminating/immobilizing hazardous products in the soil.
Good news story you should read to keep your spirits up about what is possible:
How Mushrooms Can Save the World!
Do you have the ability/tools to break up the concrete into useable sizes? What I'm getting at is that you could smash it with a sledge, or a pick, or a rented jackhammer, and break it into pieces you can handle---permies call re-usable concrete 'Urbanite'. Urbanite is a resource. It is easy to stack urbanite and it often can be broken with a straight-ish edge and can thus be used for building garden walls, or even houses. Everything is a resource/opportunity.
I hope this is helpful.