Trees take ages to grow. No money coming innfor 3 to 7 years is tough. But it does give time to build contacts.
However, if you go with a permaculture style growing, the nursery takes about 3 hours a month for a reasonable size start. If you want irrigation style nursery, up that to almost 4 hours a week. It's not a big time investment if you start small. And the risk is way less if your crop gets wiped out due to peats, disease...
The ones I know that successfully started a nursery kept their fourty hour a week job for the first few years. Or they didgthe 60h a week landscaping and 4 hour a week nursery as this builds clients and free supplies.