If you only have a few plants then I would just hand pick them until the natural predator population can take over. There are many insects, birds and mammals that will eat them if they can find them. Pepper spray might work but it will also deter those beneficial critters that will eventually do your work for you if you just leave them be.
One thing that I've done is to plant a lot more variety in my brassicas and way over plant the number of plants I will need to harvest. In my area, they caterpillars prefer kales and collards over the broccoli. By planting these things together I can trap the caterpillars on sacrificial tuscano kale and thereby not worry about the broccoli as much. Instead of picking caterpillars, I cut the whole leaf and
feed it to either the pigs are
chickens thus breaking the pest cycle and returning the nutrients to my table in the form of bacon and eggs.
I've also found that a row cover works well for keeping these pests at bay if you don't have a natural predator population up and established yet. The caterpillars develop into those white(sometimes yellowish) butterflies with the dark spot on each wing. You'll see them fluttering along and landing on your broccoli and laying eggs on the underside of the leaves.
Best of luck