I think granite dust could work, actually, if it is clean enough. Its basically manufactured sand, and if it is fine then it should make a workable mortar when you mix it up. You do not really want to include any silt or clay in your mortar, so you might have to end up washing it. Put a handful of it in a glass jar, fill it up with water and give it a shake. If the "sand" all falls to the bottom and the water is fairly clear - you are golden. If it turns all cloudy and muddy, and stays that way for more than 30 minutes or so. there might be too many fines. This article about sand claims max 4% silt/clay for masonry work.
https://www.diydoctor.org.uk/projects/types-of-sand.htm
If the stuff you have is dirty, I would just save it for the floor you were talking about. Is there really no landscape supply place near you? A lot of them let people come out and get 5 gallon buckets of material for a couple bucks. If you can get ahold of portland cement locally, I can guarantee you that sand is available somewhere too. And at 12-15$ per cubic foot, the portland is going to eat up your budget a whole lot faster than the sand. A 30 foot diameter circle is like 94 feet in circumference - that is going to use a lot of mortar with irregular field stone. Here is a calculator for mortar needed for for placing bricks.
https://www.inchcalculator.com/block-mortar-calculator/
Assuming bricks are about 8" x 4" x 2" then a wall 94' long, 8 inches thick, and 16" high would need about 2250 bricks. That would mean you will need around 1.3 cubic yards of sand. Using rocks that are bigger than bricks will help cut that down, but unless you spend a lot of time finding rocks that fit nicely together, I think you will end up using a lot more mortar than bricks would need.
Anyway, best of luck, and post some pics when your project gets going!