Looks like cabbage moth damage. Check for similar colored green looking caterpillars. I would check your soil. I go with premise that poor soils lead to unhealthy plants which are then eaten by insects.
If you only have a few plants then I would just hand pick them until the natural predator population can take over. There are many insects, birds and mammals that will eat them if they can find them. Pepper spray might work but it will also deter those beneficial critters that will eventually do your work for you if you just leave them be.
One thing that I've done is to plant a lot more variety in my brassicas and way over plant the number of plants I will need to harvest. In my area, they caterpillars prefer kales and collards over the broccoli. By planting these things together I can trap the caterpillars on sacrificial tuscano kale and thereby not worry about the broccoli as much. Instead of picking caterpillars, I cut the whole leaf and feed it to either the pigs are chickens thus breaking the pest cycle and returning the nutrients to my table in the form of bacon and eggs.
I've also found that a row cover works well for keeping these pests at bay if you don't have a natural predator population up and established yet. The caterpillars develop into those white(sometimes yellowish) butterflies with the dark spot on each wing. You'll see them fluttering along and landing on your broccoli and laying eggs on the underside of the leaves.
i was concerned about cayenne so I was out picking and squishing.. ialso noticed that they just keep coming .... I guess they just keep laying eggs. Righ on starting a cover crop... been reading alot on that ... I did plant some asian greens later and they are just starting to pop... they are doing a great job in with my tomato plants.... going to companion plant as much as I can... thanks for the help everyone... this site is a huge resource for me...
"Rather than a problem to be solved, the world is a joyful mystery to be contemplated with gladness and praise." Pope Francis
Those are the largest trousers in the world! Especially when next to this ad: