Potentially, but i admit i don't worry about it too much. They are supposed to die if the soil freezes, but mine have survived in compost piles and my garden under layers of mulch. You're a zone colder than me, though, with a longer, less lake moderated winter.
Essentially all earthworms are an invasive species anywhere there were ice sheets:
https://canadiangeographic.ca/articles/the-silent-migration-beneath-our-feet/
I've definitely seen birds snacking on my worms.