Hey Rob!
I struggle with this durability problem myself.
You might want to consider cement board, or large tiles, if you can get them cheaply.
I have used each of these to good effect.
Concrete in the garden should be fine.
For casting your own walls, there is this series of videos that go into great detail on various mixtures and how to: