I don't think that would be an issue. I've seen plenty of systems that use gravel as a grow base and work fine.
It may be much heavier than the hydrogenated clay pellets, but it still works.
Slate doesn't tend to have pours due to its sedimentary construction, but it does tend to have lots of fissures for the same reason.
But honestly. I personally don't think it would be much fun to work with when planting..