Definitely don't remove it. Every tree is different, like humans, some develop faster or slower than others. If it is putting on new growth and doesn't have any signs of blight then it will eventually give you fruit if you are patient.
You could plant more cultivars nearby in a polyculture to help it pollinate.
