Pawpaw blossom … galls?

We had minimal fruit set this year (lack of hand pollination, bad weather) and most of the unpollinated blossoms just fell off.  A few turned into these weird gall like things - what are they, and should I remove them?

pretty sure those are the dead stems from previous actual fruit. i’ve got a few too but they were there over the winter.