I believe you'll need to sneak those rotten boards out of there.
First step is probably to replace that post. Set the new post next to old post, nail the wall boards to the new post, and then you can maybe remove the old post without harm.
Then the boards forming the wall... good luck! That'll be ticklish. Shorter boards maneuver more easily than long ones, so some judicious cutting may be necessary (to replace, say, an 8-footer with two 4-footers).
Some time, some swearing, some sweat, and I think it can be done.