Maybe a lot.
I have a large farm I guess with a northern exposure and live in a northern latitude so I suppose I am qualified to
answer.
The biggest thing is wind. Here in Maine we get a lot of Nor'easters as they are called, and so my north-facing fields get brutalized by the wind. Because of this snow cover in the winter can be problematic with high legume crops such as alfalfa. Most of my fields can only be planted to a 10% ratio of alfalfa to other grass mixtures. That is because alfalfa needs the snow cover to survive since it is a relatively warm season grass. It can handle winter dormancy, but only with deep snow cover. Winter kill is high when it is 20 below (f) and no snow cover.
You are not going to be able to get on your land as quick either because frost will remain deeper in the ground in the Spring and frost will arrive earlier in the fall. Quite a bit earlier. My father lives 517 feet from me and I get a 2 week head start on frost warnings in the Spring and can grow about 2 weeks later in the fall because he is down the hill from me and in a "cold pocket". For grazing that means more winter kill and more reliance on winter
feed, so it is significant. Honestly it all depends on soil and wind exposure. I have gravely loam and am high on a hill so I can get on my northern facing fields just as soon as everyone else, but without those two things I never would be able too.
I never purchased my land though so I can not say if I would buy or not buy. I am a 10th generation farmer so I got what I got, which is what we always had; no more or no less. The good came with the bad, with a northern exposure being one of them. If I had to give my opinion, I would say soil would be more important than aspect. I would rather have ideal soil on a northern aspect then so-so soil on a southern aspect; if that makes sense. Good soil on southern aspect; now that would be best and worth a higher initial cost. Divided over many years, your return on investment would be justified I think. But as I said, I have never purchased farmland.