Wouldn't this decrease friction along the walls
Therein is why square is a bit better - from my understanding, turbulence in the burn tunnel is a *good* thing, so increased friction (to a point) and even fins, "turbulators" and other obstacles can result in a higher efficiency. What we're after in the burn tunnel is a thorough mixing of hot gasses with oxygen rich air - when these reach the super-insulated heat riser ...well, rocketty reburn
Once we get to the heat riser, however, round and smooth wins