One argument that you failed to address (unless I missed it) is the one surrounding things like native insects. For example, oaks host hundreds of caterpillar species (moths and butterflies alike) that feed dozens of birds, including warblers, chickadees, and so on. We've been able to quantify the removal of these native trees decreases the population of the insects which in turn decreases the population of the birds. Even if another non-native tree species shows up, there is nothing to feed on it so it's a sterile ground for wildlife. Maybe some deer will learn to eat it but that's about it.
