would this be toxic?

I can get crushed concrete roadbase really cheap. I am hoping to use it to lay brick and broken concrete paths, and to mix with my clayey soil to make rammed earth garden walls.

I would think that the amount of toxins would be fairly small, and that the alkalinity of the material and the fact that it would be embedded in a rammed earth wall would tend to tie up whatever was present.

Am I right in thinking this?

People with more science than me may have more ideas on toxicity etc, but I'd do it for walls.
Bearing in mind I have a pretty 'relaxed' attitude toward using materials some people avoid as being toxic...
Depending on my ph, path/wall's proximity to the gardens,
I would be a little wary about using lots of something alkaline like freshly crushed concrete -
I think it can raise ph quite dramatically for quite a while.

I'd use tree mulch on paths: I think you might find yourself weeding paths as you can't just rake out weeds if they pop up.
I also like heaving the mulch onto the garden when it breaks down.
It also looks nice and is ok with bare feet

Since concrete itself is a mixture of lime, sand, and aggregate, busted up concrete can have a wide range of properties. Nothing that weathering can't take care though, anything that might be problematic will wash out after a couple of Colorado winters. I would say go ahead and use it, but be cautious and don't use it as a border next to your most prized plantings.

I agree with John, go ahead and use it.

If you are really concerned with contamination you could leach it in some sort of container and draw off the effluent. If you then use the effluent in a mycelia growing environment, you can reduce the contamination as well as get some wonderful additive for your soil that will be of long term benefit.