It sounds like you're talking about the Caribbean fruit fly (Anastrepha suspensa). They are a common fruit pest here in Florida and are known to attack Barbados cherry.
Suspect you had good predators around to keep aphids in check. Aphids absolutely love this tree. Neem and soap spray only decrease their numbers but don't wipe them out. Tree can still grow but it's annoying seeing the growth tips getting attacked.
