Concerns about lead is not a phobia, nor is it overblown. Lead, it's toxicity and harmfulness to people and animals is not really debatable. All heavy metals are harmful and lead is no exception. There is no safe exposure level for lead or mercury in children and the level allowed in adults is very, very low. Lead is insidious in how it affects you since it can harm every single organ in the body, but since the symptoms can look like many other things it is not always diagnosed correctly. Lead decreases cognitive function in both children and adults. High blood pressure, kidney damage, reproductive disfunction all can be caused by lead. All these things are facts and not meant to scare but to inform. Don't be scared, be smart.
Now, most everyone has had lead exposure one way or the other, so everyone had a base-line lead level in their blood. However, if you have a house with lead paint either inside or outside you are without question going to have higher levels in your blood - particularly if it is peeling. No one can see that as a good thing. Lead removal is not in any way a job for DIYers. You will hurt yourself or others or the environment one way or the other.