I live along the California coast in Santa Cruz and I have been fighting my lawn every since I bought this house a year ago. The soil is very hard and feels lumpy under my feet even though the quality of soil around the house looks great. My guess is this lawn has been here for years without much love. I aerated my lawn (the one that pulls the plugs out of the ground) and watered as suggested on your site but it is hard to keep the soil happy. It seems it drys out very quickly.. And I can't keep up with the watering without taking out a second mortgage to pay the water bills.
other than using a fertilizer how else can I improve the soil and making the current grass happy and soften the soil?
You don't mention what kind of soil you have? Sandy? Clay? Whatever type you have, you need to get your soil in balance.
The first step is to learn how to make your own Aerated Compost Tea to apply to your soil in conjunction with using protein meals for fertilizer/critter food.
Having healthy soil means you have the right biology that keeps your soil from become compacted, drains well, and then you can reduce the amount of inputs required. When the soil foodweb is in balance, you can reduce the amount of watering dramatically as well as your need for other inputs.
There will be an initial expense to purchase a compost tea brewer and protein meals for the first year. But as time goes on and your soil gets in balance, inputs are reduced.
I use their 5 gallon brewer that works great for more properties up to an acre in size. However, I use Alaska humus instead of compost as it is a 'step up' from compost and worm castings. I purchase ingredients from Alaska Bountea. Here is their web site: http://www.alaskabountea.com/
Their brewer system doesn't work that well, but their ingredients are some of the best available.
You will need to get up to speed on the whole process of creating a healthy soil biology. The SoilFood Web is the best source of everything you need to know about making Aerated Compost Tea. Here is their link: http://www.soilfoodweb.com/03_about_us/approach.html
This will probably come across as rather surreal to the reader since this post is ancient ! I recently conducted an interview with Dr Gavin Lishman (Martin Lishman Ltd) one of the leading guys in the compost teas industry who was in one of my compost tea interviews available on Permies (simple directions here, no card needed) which is a supplement to 'Compost Teas' in my details below. Liquid Biological Amendments, as 'T' from Battery Park City Authority calls them are quite substantial in the area of lawn care and there is strong evidence that the compost teas Dr Lishman advocates (AACT) helps in drought resistance and soil health. Pitch care professionals are using them in high profile stadiums and one notable example that do compost teas is no other than Madrid football club which is a good indication that this thing is taken seriously.