Oaks are pretty great, and nurseries don't sell them that size
Usually in cases like this, I would not remove it unless I absolutely needed the space immediately for something else.
There may be a compromise position to trim it some and maybe even add a vine (Grape, kiwi, hops..)