This I know something about. I grew up on the Yukon River gold fields, about 100 miles downriver from the Klondike strike near Dawson. Our first cabin was on the banks of the Discovery fork of American Creek, which was the site of one of the mini gold rushes to which Klondike miners flocked after all the good claims got tied down around Dawson. And my family spend three mining seasons working various claims along the 70-mile river in the early 1980s. I have picked and shoveled and panned and suction dredged rather a lot.
If you'll allow me a moment of pedantry, what you've got going on right now is technically a prospecting fail. You haven't failed at
mining the gold, you've (so far) failed at
finding the gold. Mining is the second step, first you have to find some pay dirt to mine
in.
A few basics. Gold is very heavy. It's substantially heavier than rocks and it's oodles and oodles heavier than dirt. You've done some panning, so you know this; you get your dirt and gravel in the pan, you add
water, you swirl, you shake, and then you scrape off all the light stuff and the heavy gold (along with some various kinds of black and brown particles that tend to be various kinds of iron and tin ores and oxides, or even uranium sand in some places) is left in the crack between the bottom and the side of the pan when the pan is held at the proper angle.
What you may not know is that the property of being heavy also affects where the gold is found in the landscape. Let's run through the life cycle of a gold nugget.
A long time ago during volcanic activity, molten gold came squeezing up out of the depths off the earth and hardened into metal, usually in the presence of quartz and other similar crystalline minerals. If you have any exposed rock on your
land that is crystalline (not sedimentary like sandstone) look for the white streaks of quart and quartzy stuff, then crack out those bits of rock and look for gold nuggets. That's the first place to look.
But let's say erosion got there first, the rock got crumbled by natural processes, and the gold nuggets got pulled downhill from wherever they started by the action of gravity and running water. You can find them exposed on talus slopes below cliffs that they weathered out of, or in older more weathered terrain, you can find them in the talus gravel that's under the soil on the sloped parts of your land.
However, gold being heavy, it's usually not sitting on top of the gravel. As soils move, the gold settles toward bedrock. So the closer you can get to bedrock, the more likely you'll find gold in your dirt. If you want to prospect for this gold, the trick is to dig a series of pits or a trench (an excavator is awesome for this) along the contours of your slopes. Take a few spoonfuls of gravel from as deep as you can get (or from layers on bedrock) and pan them to see if you get color. Flakes are a lot more likely than nuggets, but
enough flakes are just as good as the much more rare nuggets. Depending on the size and thickness of the flakes, you might want to see twenty or more in a pan for the "color" to be good enough to be considered "pay dirt", but even one or two large fat flakes might be enough to convince you to dig more pits nearby and do more prospecting.
If you have a stream on your land, that's even better, especially if it runs over exposed bedrock. Gold that makes it into a stream is carried downhill, and the stream is a natural concentrator (like a sluice box). The first thing I would do when prospecting any land is to walk the streams with a pan and a
spoon (no shovel needed). I would spoon sediments into my pan that I found in cracks in the bedrock, I would pick up slabs of loose bedrock and spoon up the mud in the cracks, that sort of thing. If there's gold anywhere on your land "upstream" of you, you'll find at least some color that way. If you don't find color doing this, you're probably not in gold country.
Everything I have just written is a general rule of thumb, subject to a dozen exceptions and another dozen unvoiced complexities. Prospecting is a learned skill that when practiced by an expert, is hard to distinguish from dirty sweaty witchcraft. I was more than an amateur but I certainly never got expert at it.
As for your specific question, by all means yes: do try a pan from the deepest part of every
hugel excavation. If you see any color at all, dig deeper and try again. If you don't find color in a few pans, you aren't in the right spot or you aren't deep enough.
I hope this helps!
P.S. Your gold panning technique matters a lot. Any fool can do it, but if he does it wrong (usually shaking too hard or scraping stuff out of the pan with too heavy a hand) he can miss color. Basically if you've ever watched somebody do it who knows what they are doing, you're probably fine; but if you're entirely self-taught, that could be why you aren't finding. In that case maybe check a few YouTube videos to confirm that you're doing it right.