I think it depends on the sweaters, which would have to be entirely wool, and not the 'superwash' or washing machine safe type. Do you already have them? How many, and how big are they? The bigger they are, the less piecing you'll need to do - unless you want to do more piecing.
I'd start by washing them once in warm
water - not hot, yet, to full them
enough to ensure they don't unravel, when you start cutting. Then, I'd probably cut and piece them to about 25% or so bigger than you want your finished mat. You've a few options in how you join the pieces: butting the edges, you can whip stitch by hand, overlock (if you've that type of machine), or even zigzag(though this would be the most likely to pull apart, in the stress of use). If you want to machine stitch, overlapping would work, but I'd strongly recommend zigzagging or overlocking both edges; or you could needle felt them, though this would take the most time, if you're going to do it with enough strength to actually hold together without any stitching. Another option would be Japanese style sashiko or Boro stitching, or Indian Kantha quilt style stitching.
Since the mat will shrink more, in the wash, you'll want to stitch fairly snug, but not super tight, because the mat will thicken, as it shrinks. If you really want to have some fun with it, you could also add some decorative embroidery, crazy-quilt style. Or add a longer, narrow pocket to roll the whole thing up into, or a different pocket, a bit more square, to fold it into, to create a cushion... the possibilities are only limited by your imagination. My only concern is the non-slip characteristics of a regular mat won't be there, in the wool, so it would be wise to keep that in mind, in your yoga practice.