I think I messed up... Antagonizing paper wasps

I am learning that the downside to having a thriving pollinator garden is that wildlife will want to nest in the most inconvenient places. Two weeks ago I discovered a paper wasp nest in my mailbox. I was hoping to remove the wasps in a humane way, but I have not been successful.

First I removed the nest after dusk, but they just rebuilt it again. Then I tried to plug all the holes in the mailbox so they could not get in, but that proved impossible; there are gaps where the door hinges open and shut that could not be filled. So all I succeeded in doing was encouraging the wasps to create a second nest inside the mailbox, this time right on the inside of the door.

The past week I focused on the this second nest. If I left the door open, they would seemingly abandon the nest only to return to it the second the door was closed again. Yesterday, after a few days of leaving the door open and confirming that I had not seen them at the second nest in a while, I knocked the nest off (again). May the permaculture gods forgive because in my desperation I then decided to spray the mailbox with insecticidal soap in hopes that would deter or eliminate them. It did nothing. Today I opened the mailbox, and they were right back at it.

Only now the normally docile wasps are getting more aggressive. Today a couple of them chased me away from my yard when I stopped to look at my flowers more than a few feet away from the mailbox. This was devastating because normally my garden is like a Disney fairy tale with wasps, bees, and other insects buzzing all around me and paying me no mind. So not only did I fail to remove the wasps, but I also made myself a target to their hostility.

I don't want to risk missing mail throughout the summer from mail carriers who understandably don't want to take the risk (I put a warning sign on the mailbox), and I don't want the mail carriers to get hurt if they do attempt to deliver mail to my box. I just unplugged some of the holes at the back of the mailbox in hopes of encouraging them to stay away from the door at least. What else can I do at this point?

I don't know your area.  Here, we have wasps and hornets that make paper nests.  We hang decoy nests which look like paper lanterns and the dollar store sells them. And then spray their next with water from a distance, in the evening to desolve the paper and make them move house.  In the morning they see the decoys and make a nest elsewhere.

It took a lot of experiments like yours to figure this out.  I don't know if it would work on a different kind of wasp.