You don't have to rotate crops. It does help in some situations.
If you are monocroping than rotation helps manage soil nutrients (such as planting a legume crop prior to a high nitrogen feeding crop). If you are inter-planting a variety of species this becomes less of an issue.
Brassicas can be heavy feeders so I would suggest adding plenty of compost, something nitrogen rich like manure or alfalfa. Or consider adding some legumes. You could sow some clover on that side of the bed and then plant the broccoli seedlings into the clover.
The other reason to rotate crops is to break the pest cycle. Cabbage
root maggots can be a real problem for brassicas. If there is any chance your bed had them than you don't want to grow any brassicas there.