Generally not. You need to watch out for toxic molds that may be in your house or "off"
compost (black molds are usually not a good sign, avoid breathing, can lead to aspergillosis). Rust on plants doesn't hurt people (as far as I know). Ergot on rye and some other species of related grasses can be a health hazard to people or animals grazing/eating contaminated seeds or hay.