It would seem that putting gravel in your swale will only increase infiltration as deep as the gravel. But if you build soil structure with a healthy fungal network and lots of living
roots punching down through the soil profile, your infiltration will be excellent and you will not need gravel.
Have you ever watched any of Gabe Brown's stuff on YouTube? He's a farmer in Bismark ND who has practiced no-till farming for a couple of decades now. Google him -- he's got a bunch of talks that he's done for various groups. The reason I bring him up is that he often talks about water infiltration and how healthy his soil has become in this regard. Once your soil is healthy, full of worms, filled with fungi (which glues the soil into crumbly particles that allow air and water to flow through) you'll get great water infiltration.
I can think of one really great reason to NEVER dump gravel around on your soil: you'll forever be digging through it. The guy who owned my house before I bought it had rock mulch on the side of the house. I've been raking that up for 16 years. I think I've pretty much got it all, but still, every now and then, I'll find one of those damn rocks. It took me two years of hard work to get most of it.
Save the gravel for a driveway.
"The rule of no realm is mine. But all worthy things that are in peril as the world now stands, these are my care. And for my part, I shall not wholly fail in my task if anything that passes through this night can still grow fairer or bear fruit and flower again in days to come. For I too am a steward. Did you not know?" Gandolf