My Goumi flowers, but isn't producing any fruit

My Goumi (Elaeagnus multiflora) has been growing for over 5 years, has bloomed profusely the last 2 years for sure, but I don't even see fruit forming, let alone get any ripe fruit.

Any idea what its problem might be? It lives in between two rhubarb plants which are steadily getting happier. PFAF claims it's hermaphrodite, which is what I would have thought also.

Ideas anyone?