I do rotate my beds and do it not only based on family, but on fertilizer, light and soil disturbance needs. I have 12 beds though. With 4, I do agree that you could simply rotate their use clockwise annually. (A five year rotation is generally what is recommended. With 4, you can almost get there.)
I do have some other thoughts though. Depending on where you live, some of your crops could be
perennial so you may not want to rotate them. Chives, thyme and the perennial variety of chard grow yearround where I live. You might want to consider creating an herb bed for your perennial herbs next to your vegetable garden. Most perennial herbs (e.g., rosemary, oregano, thyme) do not need or want rich soil so they don't need to be in a vegetable
garden bed.
As far as depleting the soil is concerned, many annual vegetables are known to be heavy feeders (e.g., tomatoes), but I find that almost all annual vegetables produce much better if they are fed what they need. If they are not, they will deplete the soil. Every time I plant a new crop, I start by adding at least a few inches of good
compost, and I also use organic fertilizers based on each plant's needs. If you don't want to use fertilizers, I'd recommend at least adding ample amounts of compost annually. Composted
chicken manure is also a great, fairly balanced, inexpensive soil additive.