Right now, they are repaving a five mile section of highway near me. It is six lanes wide and being done with concrete (since it's a grade with snow removal operations). The slab they are pouring is two feet thick. I think that's somewhere in the neighborhood of 150,000 cubic yards of concrete. On this one highway. In this little stretch of barely populated California. Now imagine how much concrete is being used in infrastructure projects around the world... and how many yards it's sucking up. Now think about how many cubic yards a person uses in a foundation for a house.
My point here is, if you are concerned about the environmental impact of cement production — that's a valid concern — but has almost nothing to do with using concrete on an individual level. Military and infrastructure projects are going to use up several orders of magnitude more than individuals ever can.
If you're worried about it being personally toxic, as far as I know concrete is one of the safest materials you can build with. You need to be careful when mixing and drilling into it (wear a mask), but otherwise it's a stable, 100% recyclable material (smash up concrete, mix it with cement, and you get more concrete). When it eventually crumbles, it wears away to chemically stable rocks.
Also, as an aside on this note:
How it is used is crucial. If you "reinforce" it by pouring it around rebar, which, in the presence of moisture, which readily moves through concrete, swells as it rusts, destroying it from the inside in a matter of decades, it doesn't matter how long it would have lasted for. In that application, I think it's pretty vile.
This is not quite right. Rust is triggered by water, but the active ingredient is oxygen. When rebar is encased in concrete, oxygen can no longer reach the steel to corrode it. Any concrete that only lasts decades was built extremely poorly. A properly designed concrete beam will happily last for centuries. That's not to say there isn't a lot of shoddy concrete jobs around the world. But then again, there's a lot of shoddy jobs of all kinds around the world. I will tell you this: concrete of any form or function without reinforcement, will fail with 100% certainty. Even fence post anchors have reinforcement (with the pressure of the dirt acting as reinforcement).