I think it is a great idea. Go for it and make it happen. You can build it incrementally.
Have you listened to the recent gourmet restaurant review podcast? If not, check it out, it will really put a bee in your bonnet. Thanks to Paul and Jocelyn for that episode. I found it fascinating, I listened to it three times. It inspired my girlfriend and I to suggest doing a pop up restaurant with permaculture/organic/local food at our friends' tavern where she works. It looks like it may happen. Our intention is to get food from local permaculture or organic farms, plus what a network of us are growing, and over time increase the food that our group is growing to eventually be the majority of what is served. This way we won't be pressured to produce everything initially.
Paul and Jocelyn mentioned that there is an Italian word for a restaurant that serves food that is grown on-site. I tried finding the word with some Google searches, but couldn't. If anyone knows it, can you post it? I think it would be a great meme to promulgate.