Kinda Keyhole garden/ kinda compost bin and raised garden together

I am starting to plan out my first veggie garden (I finally have a backyard woohoo!) and I plan on having 3 raised beds. I was thinking of adding a sort of built in compost bin inside each raised bed (on the end) where I could throw scraps and if I add worms to my beds (which I'm 90% sure I will do) they can get some extra food and sorta create a vermicompost bin.

I know that keyhole gardens allow for the nutrients of the compost to naturally infiltrate the garden beds but I don't want to use that shape. I just want to have a sort of symbiotic set up where I can vermicompost and let the worms be helping out the entire garden bed instead of having a worm bin in my house trying to keep it away from my toddler.

Is this a dumb idea? Is there any advice on making this feasible and smart?  Am I making this more complicated than it needs to be?

I have attached the drafts of my garden and what I want to do to create this franken-garden (fran-key-garden?) (garden of fran-keyhole?) and appreciate any help. For added context, if it matters, I live in Las Vegas, NV (Zone 9). In the Garden layout pic, along the back where the marigold and yarrow will be planted is a basin for water to drain under the gates along all of the neighbors yards to get to the sewer drain. I am going to dig out the clay dirt that is there and fill it with soil (while maintaining the basin/trench) and wild sow marigold and yarrow which are native to the area and great pollinator plants. This allows me to add pollinator plants to the area without losing space for my veggies while also utilizing an area that will collect any rain run off that comes through when we get our little bouts of rain.

I think it will work OK.
I'm moving towards pockets of soil in beds of leaf compost, but the principle is similar.
I would probably put multiple worm towers throughout the beds,rather than one bin on the end.

William Bronson wrote:I would probably put multiple worm towers throughout the beds,rather than one bin on the end.

Oooh, I've never heard of a worm bin until now, but I just looked it up and that seems like a perfect idea! Thank you! Would Red Wrigglers be the correct worm for a worm tower? I've seen a few things that said they only live in compost bins/heaps and I'm not sure that the worm towers work exactly the same...Is there another worm that would be a better fit?