Zone 5 perennial flowers

I'm trying to add more flowers into my life. So help me add some things I should try!

As far as bushes I have lilac, caragana, goosebeery and chokecherry growing very well.

I also have the following growing:

Flax
Iris
Daisy
Echinacea
yarrow
Stonecrop
Lily
catmint
Peonys- though they've never bloomed they do grow up every year. I don't know what I'm doing wrong there.
Rugosa Rose
I have issues getting russian sage to grow, I don't know why, it grows all over the city.

I really want to get some lupin going.