So you pretty much 'live' in Albany, New York with UDSA Winter Hardiness of 5b.
Just make sure you pick cultivars/species that have that rating, aka no full size or dwarf citrus/figs/passionfruit for you.
Chill Hours = you don't have to worry about, you get enough.
Summer Heat = you also don;t have to worry about unlike say the PNW
Rainfall = you get about 4inches of rain per month, every single month so, no need to worry either
Soil = you have the same type of soil like alot of the eastern seaboard of USA, so not this ridiculous level of uniqueness.
Semi-Dwarf are perfectly fine for your soil and location and doesn't make much of a difference, the main thing to focus on is species and cultivar selection.
Cultivar Selection
Just enter in your zip code in the website below and it will give you cultivars that will survive your area winter temp
https://www.starkbros.com/tags/dwarf-fruit-trees
Soil Problems aka Nutrient Problems.
To get more nutrients to your plants
Innoculate Bare Roots minutes before planting
https://fungi.com/collections/mycogrow/products/mycogrow_soluble?variant=12125274472560
Worm Tea
Mushroom Slurries
Add Carbon (Woodchip, strawbale, biochar, chop and drop)
And Nitrogen (Dutch Clover and other nitrogen fixing clover)
Soil Aeration (Daikon Radish, Woodchip/Carbon)
Add Minerals (Sea90, Rockdust, Azomite)
Pest Management (garlic/onion family, carrot/dill family, mint/thyme family, innoculants)
Water (Swales, hugleculture, berms, etc)