Will sad potted plants have failure to thrive in the field?

I was told once that if nut trees are not started off right in their first couple years, then they will have a failure to thrive in the future.  Anyone know if this is true?  If they have a poor start are they even worth taking the time to plant out in the field?  Here's my particular situation.

I planted tree seeds in pots, and some in raised garden beds, all to be planted out this fall.  Sadly, some of the types of trees look way sad in the pots.  The pots are probably too small, and they probably were not watered enough.  On the other hand, all the trees in the beds look great and are much bigger than the potted plants.

The hazelnuts did the worst in the pots.  Many of their leaves are yellowed and browned.  It didn't help that they got temporarily infested by Junebugs.  Some of them are recovering and putting out new nice leaves and have lots of buds.  

Thanks Permies.

I'd try transferring them into larger pots or even into the bed to see how they do, mulch around them real well for the winter and see how they look come spring. If they survive and thrive, then it was probably something about living in pots they didn't like. Data for the future.

I grow a ton of trees in pots and here in Oklahoma boy how they suffer!  But in my container garden where I can protect them from deer and drought, at least they survive.  And so far, the ones I've planted out into the field have mostly taken off well.  

What I do is go through them every spring and fall and dispose of the dead ones.  Ones showing new growth that suffered a lot of dead leaves and stunting during the previous summer heat get repotted into bigger pots, until I get to the point where I have a four foot tree in a five gallon bucket; those I plant out into the field.  I know all about the prevailing belief that a tree grown from seed in its final home will be a happier tree, but I simply don't have a way to protect those.  I do *also* plant a lot of tree seeds, and I've had some success with native persimmons, pecans, and black walnuts that way.  

What I've learned is that if I "kill" a tree in a pot, by which I mean if its leaves turn a funny color or get crisp or fall off, eight times out of ten if I keep watering it, it will eventually sprout new growth, sometimes on the existing stems or sometimes from the base.  For me the death season is the high summer, which is just now showing signs of ending.  I have a lot of unhappy potted trees but only a few that I am convinced are hopelessly dead.  Don't give up too soon on unhappy potted trees, wait at least through a month of cool moist weather to see if they revive -- or better yet, wait until they go through a spring season without any sign of greening up.  That's the only time I truly give up on an abused potted tree.

Thanks so much for the great replies! I think I will plant some in my nursery beds ASAP, and plant others in the field and see how they do.  This tree propagation season (by seed), I will be planting all in the raised nursery beds, they just do so much better and I don't have to mess with watering them.  I'm learning so much.  I plan to plant many more trees by seed this fall.