They can be very important and lack of a specific micronutrient can make or break a crop. Jared Diamond, in one of his books, talks about how minerals get distributed worldwide by volcanic eruptions bringing up new minerals from the Earth's crust. He goes on to point out that Australia has mineral poor soils. Why? Because they are not downwind from any volcanic eruptions that would deposit more ash on the continent. It is an old continent that has no new minerals coming up from below and all that makes it in the winds across the Pacific is rainfall.
You're right to observe that each truckload of produce that leaves the farm takes some of the soil minerals with them. Now if you are bringing in wood chips and manures and biomass (like dead animal carcasses, especially the bones) to build hugelkulturs, maybe you are compensating for some of this, but you don't really know until you do a complete chemical analysis. Not just of N,P, and K, but B, Ca, Co, Fe, Mg, Mn, Mo, and Zn. Some Agricultural Extension services provide these analyses for reasonable cost. When mineral deficiencies show up, it's pretty much too late for the current growing season and you have to figure out how you are going to apply the micronutrient so that it has time to get incorporated into the soil for the next year. Some soil minerals are not that labile, for instance B, and if you are trying to grow fruit trees in B-poor soil, you need to be making foliar applications at crucial times, like right before flowering.
If you add chelated minerals to the soil, you can build up your mineral stores and then they are available when plants need them, but still bound so that they are less likely to leach out. How to make your own chelated fertilizers? Here's a
shameless plug for another post of mine on this topic.