If you will be keeping both the rabbits and quail in pens, I would imagine just about any dog would try to protect it's territory. If the rabbits and quail won't be in predator-proof pens, you might want to be a bit more choosy about what type of dog you get because then the dog may become the predator. There are several types of dogs that are bred to protect livestock, one being the Great Pyrenees. If you are willing to pay a pretty penny for a
LGD, a lot of times people will raise them around
chickens to make sure they won't go after the smaller livestock.
You could also look into guinea fowl. They are known as 'guard dogs' and will sound an alert if there are any intruders and will even kill smaller animals like snakes. Although some people don't like them because they will alert to anything that moves and can be loud. I've been on the
fence about getting some for a while now...
Right now, my best guardian is my peacock. He will chase anything out of my yard that isn't supposed to be there. I just got a new goat the other day and he jumped on her back because she was an outsider. Once he realized we wanted her there, he let her be. Peafowl probably aren't the best for 1/2 an acre though, they wander quite a bit. Not terribly helpful, but I just wanted to point out that an LGD doesn't necessarily have to be a dog.