It honestly sounds like you need to look at a variety of ways of making buttonholes and explore them.
Examples:
~ Try sewing bound buttonholes (also by hand) by using cloth that you then turn inside the garment. This doubles the fabric and provides a strong base where the threads won't pull and distort.
https://www.thelaststitch.com/how-to-sew-bound-buttonholes/
~ Try sewing over a piece of thread or twine to give a guide and clean base for the button hole to be stabilized by.
~ Use stabilizer or a dissolving material to give you a base that will strengthen the buttonholes and stitches.
Look up on You Tube or Threads Magazine for styles of buttonholes that will help you gain skill and confidence. Its a matter of techniques and practice, just like any hand craft or skill.
Kat