If you suspect anything contains asbestos, I would either get it professionally removed (bagged and disposed of appropriately) or if its not bothering you where it is, then leave it. Asbestos in water pipes has been an issue where I live, until the council replaced them all, so any asbestos fibres that are capable of entering the air or your potable water source should be dealt with. Anything thats in your garden, if your not encountering it every day, and its nots something you're going to dig up regularly, then I wouldn't worry about it. It wont creep up on you one day and surprise you. As for the other stuff in the garden, I'd use it where you can, but just be aware of things that might contain harmful aspects, like perhaps asbestos in the grey roof tiles, (I think your red bricks are AOK) PCB's, Arsenic, etc. Simple things like bricks, old glass, etc are safe, but I'd be aware of anything that can corrode, or leak into the soil. If it was me, I'd be sending that stuff off site.
