Should I remove barnyard gravel

We have a smaller-sized dry barnyard that has a small portion covered.  It does get wet and mucky in the winter and Spring.  I hate sifting the gravel out of my compost because of the gravel in the barnyard.  We have tried to scoop and dig it out.  But gravel just keeps surfacing.  If we dug way down and tried to remove it, I don't know how far we would have to dig down.  It was a horse yard before we moved here ten years ago and there was a lot of gravel.  If we did manage to remove gravel, I am not sure if we should just replace with topsoil or what.  I imagine topsoil would be very mucky when weather is wet.  

If not for gravel in animal yard, I would not have to sieve all my compost!

Thanks,
Beth

Is it 3/4" minus? You could try adding a top layer of fines. We have crushed granite gravel in our barn and I hated walking on it, so I tried putting down a few inches of pulverized fines on top ("granite dust"). It helped make it more comfortable on the feet and much less noisy. So, you could try that.. it won't help with the moisture issue, though. Ultimately, I wouldn't worry too much about gravel in your compost unless you are making a fine potting soil or something.

Thanks for your reply and I'm sorry but I just realized I got a reply.  I don't think it is 3/4" minus but I'm not sure what that means except less than 3/4" in size.  Spreading the rock dust on top might be something I could do.  I was thinking though about the cobble I see in pictures of old European barnyards.  They have gravel mixed in with concrete or whatever and maybe I should add more gravel and put concrete down there.  The gravel basically extends from the barn to 4 feet away from it.  Then the goats would have some rocks which is good for their feet.

Thanks,
Beth