Clay is made up of extremely fine particles, the way to make clay soils closer to friable soil is to add organic material, this gives the clay large particles to cling to and so separates the clay particles a little.
Fine sand would be ok to add to a soil that had plenty of humus in it (friable soil). What happens when you add sand to any clay soil is that the clay particles bind to the sand particles,
If the sand is large mesh and is added along with large quantities of humus this can make the soil begin to break up and start the soil building process.
Fine sand is added to pure clay to make pottery that will not crumble and is easier to shape into vessels.
Clay and fine sand works great if you are making a clay plaster or bricks. I don't think this is your plan.
I would not pass on the sand, unless you have no good place to store it for later use.
I would start my soil build with
compost and humus reserving that fine sand (sounds like it could be meshed for sand blasting) for later in the soil building process.
Once the soil will crumble in your hand it would be ready for the addition of sand, course sand would be my first choice, and perhaps work down to that fine mesh sand.