I've not raised any castrated pigs. All of mine have come with all their pieces in tact. That being said, I've never experienced a difference in meat flavor between males or females that I've raised. I'm under the impression that taint is as much about environment and feed as it is about genetics and individual variability. So... there's a lot that can be said about it.
If It were me and I had other pigs along with Mr. One Nut, I'd take my chances and raise him out as planned. At the very least you'll get to know if having the testicle made a difference by comparing his flavor to the others in the herd. I suppose the only other concern is that he'd mate with another pig. I guess you're going to have to make a call on that one.
I know that Walter J has a great mound of info on this particular topic. Searching in any of the pig threads will likely take you to some high quality post by him. Alternately you could go to
Sugar mountain farm and spend the next few hours reading all of the intricacies of pork. It's quite a treasure trove and I have always found what I was looking for there.
Good luck.