Douglas makes a good point. I'm all for avoiding rigid foam but you don't want something that's wicking subterranean water into itself. What about a poly film barrier underneath the gravel and perlite mix? I'll also put in a plug for biochar as a possible ingredient, as it has similar properties to perlite and you can make it yourself.
Have a look at my mate Blue Forsyth's
article in our society newsletter to see how he used biochar as underfloor insulation.