Unfortunately, topsoil is a giant sponge that expands, moves, contracts, and never really packs well. Not great road material.
As mentioned, much depends on your subsoil. If it's very sandy, you're not going to win anyway. But pure clay packs like
concrete.
If the traffic is light vehicles, you may be okay. Any kind of heavy truck traffic will pound your road all to heck.
A soft road can soak up infinite volumes of gravel. Given the cost of gravel these days, it may be worth looking into geotextile. This is like mondo industrial grade landscape fabric. Put this down before adding gravel, and at least it won't mix in.