One way of solving this is to make the buttonholes smaller, again. If using a sewing machine, the zigzag stitch (B in the top half of the diagram) would do this.
Here are the steps I would do if using a machine:
1) Place the button in the buttonhole and mark off points slightly smaller than the button on the buttonhole with a pencil
2) Make a locking stitch at the top of the button
3) Zigzag stitch down until you reach the upper line that you marked off
4) Zigzag stitch back to the top of the buttonhole
5) Make another locking stitch
6) Cut off the excess
thread
7) Repeat steps 2-6 for the bottom portion of the buttonhole
*edit: I forgot to mention but to make it clear, you would sewing shut parts of buttonhole by stitching the fabric on either side together; that is why I would use a zigzag stitch. It helps with that kind of thing.
If doing this by hand, you pretty much do the same thing.
By locking stitch for a machine, I run back and forth over the same point with the single stitch (A in the top half of the diagram)