Hard, dry compost, what's wrong?

16,800 posts on Compost, I gave up searching and did a new post, apologies...
We have been making compost for ten years but the resulting compost doesn't work well as a substrate. The harvested compost turns dry and hard and crumbly and our plants don't thrive in it. Help!
My hypotheses for what may be going wrong:
- Too much kitchen waste and not enough carbon - the compost is mostly kitchen waste that we "mulch" with either coffee chaff, sawdust or dry fallen and partly decomposed cypress needles. We don't have a huge amount of garden waste to add, most of our garden is covered in tradescantia which is unkillable and I don't want to give it new cushy homes in the compost.
- Too acidic: my husband decided that the compost needs a pre-digestion stage and we now start the compost in an airtight barrel, where it pretty much ferments. Weirdly, this sometimes seems to preserve the waste rather than breaking it down, like a giant mango peels and carrot tops kimchi. When the barrel is full we turn it out into the aerated compost bin, and it stinks (sour, not putrid) for a couple of days.
- Too acidic from those cypress needles: the compost bins are under a big cypress that drops plenty of needles all around. Many of those are decomposed and I often grab a handful of them for covering up the kitchen waste. But might they still be "active" and allelopathic, like pine needles?
- We have lots of earthworms, both pink "Californian" ones and grey, vigorous native ones (living in southeastern Mexico). Can the earthworms have eaten all the nutrients?
I just emptied out a second-stage compost (the areated bin where the fermented stuff had already broken down, full of earthworms) and it seemed perfect: moist, dark brown, fibrous, easy to pull out the avocado seeds and maize fibres that hadn't decomposed yet. I used this for sowing some seeds and for some potted plants. But where we've used the last batch of compost the plants don't seem happy, the soil forms into little gravel-sized nodules.

Context: we live in a town in Chiapas, Mexico, at 2200 m altitude. It's dry from October to April with cold nights and strong, lethal sun. From May to September it gets warm and rains nearly daily, often with crazy deluges. The soil is thick unbudgeable red clay and we're planting on top of that and building rubble, slowly creating soil. We're adding plenty of mulch and cover crops which in turn invites snails and slugs. Sigh. Special plants need to be nurtured indoors where it's an all-or-nothing laser sun or shade situation, hardly any indirect light. The sun weakens the plants and they often succumb to (I think) powdery mildew. Sheesh.  

Would you by chance be able to provide a photo of your compost?

Does the compost rewet and retain water? I'm wondering if it is repelling water, or if it might lack water retention properties for whatever reason.

Do you know if the compost ever heats up or if it's more of a cold compost system? I'm unsure of which variation of aerated pile you utilize.

I'm sure some more 'advanced' compost creators can chime in an help. I'm still pretty new at it myself.

I believe you are on the right track.

The compost we have made looks like rich brown soil.

Does your compost heat up?

Do you have a good ratio of brown manure to green manure?

2/3 brown to 1/3 green is the simple answer.

Here are two threads that might explain:

https://permies.com/t/47495/composting/browns-greens-manures-geoff-lawton

https://permies.com/t/20074/composting/Rookie-Green-Brown-Ratio

Hey Emilia. Those are challenging conditions!

Be advised I am a long way away from Mexico. But here are my thoughts.

From your description it seems like your compost still has a lot of tough, fibrous material in it. That suggests it isn't fully broken down. Some of those materials can take years to fully decompose into compost, and may be better suited as a top mulch. Otherwise they will continue to suck up nitrogen as they decompose and steal it from your plants.

Also, I get the impression that you are planting directly in compost -- is that right? Personally, I find it doesn't work at all -- compost seems to function best as an amendment that's added to a complex soil mix.

I have a short composting season in a dry climate, so I force my compost piles into an anaerobic state for long periods -- weeks, months. They rot and stink and are putrid, but it adequately breaks down the heavy fibrous material from last year's stalks (corn, sunflower, amaranthus, etc.). When I start to fork it out and mix it to get oxygen in, it may still look coarse but my squash plants absolutely love it.

As Anne notes, the mix of greens as a nitrogen source is important. I wonder about your "wandering willie" /  tradescantia. Could this be harvested and fermented in barrels to provide a high nitrogen compost tea, for the purpose of accelerating the rest of the pile? It seems it could be filtered enough to keep invasive stems/rootlets from spreading.

Hope my notes help a little.

Emilia, my understanding is that soil clumps are created at the micro level from bacterial glues, and at the macro level from fungi and worms. Worms help neutralize pH, but it is fairly easy to obtain a soil pH test if you want to see where it ended up after composting.
If compost gets too dry bacteria will try to protect themselves by creating a hydrophobic bubble to retain the interior water around them, but that makes compost repel water on the outside as well, making it very difficult to rehydrate compost. Do you use mulch to keep the applied outdoor compost moist?
Are plants not germinating while the compost is still moist? Germinating seeds in straight compost can be difficult, as compost can introduce pathogens and fungi that may harm seedlings, especially if the compost is not fully mature or sterilized.
Anaerobic digestion is inherently slow compared to moist aerobic composting.

Hi Emilia -
It does sound like your compost hasn't really decomposed all the way for some reason. I've had the clump thing happen wtih compost I purchased in bulk and it took two more years to break down enough that it wasn't clumped up in my garden.
What do you think about adding some other materials? You should be able to get arborist chips from a local arborist, ChipDrop or your local Parks Dept. Plenty of people around here also give away free manure. They can be found on websites that offer used stuff to other people, like Craigslist. And, if you wanted to add more greens, you may be able to get coffee grounds from a coffeeshop near you.
Did you have good luck with starting seeds in compost? I tried it and it wasn't very successful. Too many pathogens. I now use a sterile mix and it works so much better for me.

Jen has posted a very good suggestion.

Adding wood chips to you garden would be good.  Is this something available in Mexico?

Plants started inside need to be hardened off before planting outside as noted the they succumb to the sun and powdery mildew. There is a solution for powdery mildew:

https://permies.com/t/139504/Powdery-Mildew#1093664

Hi Emilia, it sounds to me like you have two different things going on here- bokashi and composting.
The "pickled garbage" barrel you have is basically bokashi- a pre-ferment. People have fancy bokashi setups with special magic sprinkles but I run mine with pretty much no additives and it works the same. If you don't drain out the leachate, you have an anaerobic mess that's kind of like swamp water, full of rotten goodness but also smelly as hell. In bokashi, you drain out that leachate and then you mix the barrel contents with dirt for a few weeks before you have good planting compost.
My setup is similar to yours, except for instead of crazy dry Mexico I'm in always-raining Brazil (the cold part, nearer to penguins than the Amazon). Our soil absolutely demolishes every bit of organic matter I can add, immediately (tropical clay). I use a barrel like yours, and then when it's full I add it to a large, performated trash barrel with larger chipped scraps (the stuff my rabbits won't eat) and whatever dirt I have laying around from crummy planters. If need be I'll dig up ground dirt. I mix it in rough layers and leave it for a month. If I throw the barrel contents into the compost or on the ground, like you say, they get weird and crusty and don't really break down. The barrel is a great solution, especially in smaller urban settings like mine. But try mixing it with some dirt and letting it mellow before using it as compost.
(I used to have a tap on my bucket to drain out the leachate, and use it as a foliar feed on my plants. It smells and my neighbors are crybabies so instead now I just left the tap hole open and I let the bucket drain out in the way bottom of my garden (it feeds a banana tree or whatever lucky plant that happens to be nearby)).

(and just in case you're wondering, i don't do normal compost because my space is very small and i don't have the space to dedicate 2 or 3 cubic meters to a pile that gets really hot, and also every time i've tried it it hasn't worked here, we're just too wet I think, it just turns into a stinky mess)

I recommend concentrating on the worms.  Most things in the kitchen scraps do not need the the fermentation step but some may benefit.  Sawdust is a good support material.  I save mine from fire wood for that purpose.  Lawn clippings can be dried and added or fermented and added.  Make a hole next to the previous addition where the worms are feeding to make a new addition then cover it with damp  material and cardboard or dry material.  As the worms finish one addition they move to the next.  When you reach the other side of the bin dig down and put it on top of the fresh material and start over again. I just dug out the end I took the picture from and used what had no worms in it.  
Using the worm compost think Lasagna; Dig out a planting bed in the clay.  Put a layer of worm compost. break up the dug out dirt so that it will be a dry layer on top of the compost.  water the compost layer perhaps with an ola pot so it stays damp.  Transplant into the dry layer so that the plant roots are in contact with the the compost.  some plants take the sun better so plant them to provide shade for more sensitive ones.

I always laugh when I ask a question and questions come back.  I think to myself "answer the question"!  Here I am asking you questions.  I used to live (In the US) at a similar elevation and it provided similar challenges to composting.  I use a 3 year composting cycle, meaning from the time I start a pile to the time of use is 3 years.  Each year I start a new pile.  So I always have 3 simultaneous piles going.  I do this for a variety of reasons, one of them being I get a really good break down of the materials after 3 years.  This was especially true at a high elevation (which is your case).  I'm wondering how long your normal compost cycle is?  The dryness has me concerned.  Are you regularly monitoring your pile?  The suggestions that you have already received are good suggestions.  Moisture is so often a problem with compost.  I treat my piles like plants and I water them and monitor them intensively because if I don't they will dry out and die.  My piles are often huge (several feet in diameter) but with supervision I keep them alive.  Also at your elevation is it safe for me to assume you have snow?  Is heat generating from your piles in the winter?

Just a few thoughts and I do emphasize with your if your somewhat disappointed with my questions when you're the person asking a question but I'm trying to drill down to the possible issue.  

Happy composting!