I suspect some sort of issue with the grafting rootstock growing into the tree, but the branches that are bearing the red fruit could also have been grafted onto the tree.
Look at the tree trunk and find the grafting union. There
should be only one trunk and the grafting union should be a few inches above the soil. Also make sure that the soil is exactly at the level of the tree's
root crown (where the tree starts flaring out at the bottom and the
roots start). Do all the branches start after the grafting union? if any start before the grafting union, they should be removed. If they all start after the grafting union, I'd look at the branches that are bearing the fruit to see if they were grafted on.