I feel that rocks by themselves will have limited ability to slow your drainage, perhaps layers of large rocks combined with soil with a small particle size and 'sticky' when wet, like clay, dispersed around/over the rocks. That may reduce the amount of 'sticky' soil you'd need to use to seal the gaps between rocks. If you had access to cardboard, especially waxed cardboard and layered that under the rocks and sticky soil, that would be even better.
It occurred to me too, if you had some material with higher organic matter that could amend your native sandy soil when topping up the hole to your desired level, that would also help hold water. If you could add
compost or manure or shredded leaves that would increase the organic content of the soil and its ability to intercept/retain moisture.