The first thing I though of when reading this is this house.
http://permaculturenews.org/2011/02/07/freeform-earthbags-between-boulders/
Which incorporates large rocks into the walls. Not sure how it's held up but certainly looks really cool if you live in a climate that this wouldn't hurt.
As far as directly on rock, the only thing that I'd be concerned about is tying it into place to prevent walls from moving. Depending on material weight by itself might be sufficient. I'd just be worried about the base slipping somehow especially in an area with seismic activity. Heaviely bermed might help there as well.