Drawing the line -what trees do you save?

I was weeding a patch of ground at my sister's house to plant an American Plum and found 4 elm trees and a hickory of some kind, in a 4'x4' patch!
I pulled 3 of the elms and left the other one and the hickory.
Looking up elms, I wouldn't plant one in my urban yard, so I'll probably cut the last one down.
Will I be potting up the other three for the the community garden free nursery?
Maybe?
I would let the hickory grow, but I don't think my brother inlaw would want a huge tree so near the house.
I will try to save that one for the nursery

Where would you draw the line on these trees?
What about other trees, what do you save, what do discard?

Around here the Squirrels and Birds plant a lot of Filberts and Cherries.  Also volunteering frequently are: Maples, Cottonwoods, Ash, Cedar, Fir, Oak, Butternuts, Plums, Cherry-Laurel, Hawthorne, and Holly.  
A lot of these die off on their own from the Summer dry season, unless they are randomly situated in good and protected spots.  The stands of Brambles/Blackberries make for good protective chaperones for many of these volunteers.  
If I find random trees growing in places that I am otherwise not tending much, I will usually leave them to do their work and improve the site with roots and leaf-drop.  Though, with many of them, I have intentions of cutting them down before they get too large.
If they are in my highly tended zones, where they don't fit well into my particular plans, I will sometimes dig them up and pot them to plant elsewhere or give away - or cut them down.  I don't know what my parameters are for choosing which to cull and which to pot up, or which to leave growing where they are.  
If they are a fruit tree, I am always curious what they might yield, so I often let them grow so I can see if it is some great new variety.
Many of the others I consider useful for branch/leaf mulch sources (chop and drop), or for firewood, or other utility/craft purposes.
However, some trees just show up in the wrong place, and it is much easier to remove them when they are small.  Close to building foundations, many trees' roots can wreak havoc too.  
I will say in my situation I err on the side of leaving them to grow even if it makes more work for me later on.

New property and evaluating existing trees, I ask myself three questions.

1. Will the roots make me sad?  Spreader roots like willow can tear up cement to get to water and pipes are no problem.   So much damage from tree roots with spreading and seeking habbits.  That tree got to go.  Same with foundation planting which attracts rats, damages house, and makes cleaning gutters too often.

2. How many fruit trees can I fit in that space?  We replaced 12 trees with nearly 200 fruit trees in he current farm.  Urban lots may not alwas be so fortunate.  Fruit trees provide mulch, stack functions as well as almost any tree, and make fruit.  As the tree matures, the wood can be quite valuable compared to common wood.

3. Can the tree provide shade while i grow up better trees...but if I do that, will there be an easy way to fell the older tree when the time comes?

Those are my lines,  fuzzy questions to ask as each property is unique and has different needs and possibilities.

The hickory is the one I would fight to keep personally. They take forever to get to fruiting size and once you lose one you really feel it. Elm on the other hand volunteers everywhere around here and I have never felt bad pulling one. The fruit tree rule of thumb works well — if a volunteer would take 20 years to do something useful, it goes. If it might fruit in 5-7, it stays.