david a lot of people from what I know like 50% or so of people with meat rabbit colonies take out fryers at weaning (4wks old if buck is in full time, and yes 4wks is old
enough) and put them into growout cages or smaller growout colony or rabbit tractors. the main thing about meat raising and colonies is that the
rabbits are going to be burning off a lot of what they eat from the running around.
you are just plain and simple NOT going to get the great growth on a colony (or forage fed vs pellet fed) rabbit as you would a caged and pellet fed rabbit. I think this is 95% of where people go wrong when thinkin about colonies. they want less handling the rabbits for the same amount of outcome. and it simply does not work like that.
for your rabbits the issue could be you just have not that good genetics in your rabbits and they don't have the genetic potential to grow to 4.5/5lb at 8-12wks. OR could be worms as you said.
how much are you feeding out in the colony per day? colony rabbits will need a lot more
feed then cage rabbits to comensate for the additional exercise they are getting.