It's totally fine to go ahead and add extras. Worms compete with each other for food resources, but other than that they are not territorial or aggressive in any way. Also, it's worth noting that as your setup matures, you'll notice your food disappearing quicker. Bacteria are actually still doing most of the work in a worm
compost, just like in a regular compost pile, so they can take a little bit to fully colonize and step up on production. Adding a little sprinkling of finished compost to your pile can help speed up bacteria colonization and adding a little dirt can provide some grit to help the worms digest their food.
"There is nothing, Sir, too little for so little a creature as man. It is by studying little things that we attain the great art of having as little misery and as much happiness as possible." - Samuel Johnson