Rifle is fine if it's shooting 1.5-2" at 100 IMO. Most of that is probably you, not the rifle. If that's off the shelf ammo that you can find easily, buy a few boxes of that and you're good to go for the next decade - you don't need to shoot 100's of rounds out of a deer getting gun.
What distance are you shooting deer at? You mentioned wanting to hit a grapefruit at 300m, how does that translate to deer vitals at the distance you are likely to shoot one at?
Again IMO, unless you need to be shooting further than 100m you're probably good to go for an old rifle, no sense in hacking it about for marginal gains. 200m doubles your group size to 4" (assuming excitement and adrenaline doesn't open that size up!). 200m is a long way to shoot something if you're not 110% certain you're going to kill it cleanly.
If you NEED to shoot further, don't mess up the hand me down rifle, you might regret that one day. But a modern bolt action rifle and be amazed how little improvement there is after spending $1000 or more.
Don't make card stock shims - when it's damp they'll swell. If you must make shims make them out of something non porous.
