I find this a bit of an ethical struggle, as potting mixes made with purely organic matter (organic as opposed to mineral, that is) tend to become quite compacted and my plants have struggled.
I generally make a mix of perlite, sieved
compost and aged bark 'fines'.
I use coconut coir in my seed-raising mix, and sometimes a bit of that goes in too.
If I raised worms, I'd definitely be using vermicast in there!