Oh my, I had this same problem. I put down weed matting (landscape cloth) thinking it would control weeds and ended up with a terrible mess. Because it covered such a large area, my husband ended up taking a tiller to it to cut through the mass of grasses and weeds growing through it.
That was a mistake! It ended up shredding the cloth, resulting in hundreds of tiny bits of the stuff everywhere. I tried to rake up as much as I could, but so much of it had to be picked up by hand.
Years later, I still find pieces of it! I learned my lesson and will never use it again. Unfortunately, I can't tell you the best way to remove it, other than to encourage you to use whatever method shreds it the least. Even if it seems to take more time it will save the backbreaking work of hunting and pecking to find and pick up a bunch of little shreds and pieces.
Please let us know what you end up doing and how it works. I'll be curious.