Have I made a terrible mistake?....(Castor Bean Plant)

Hi,
I'm a newbie to this type of gardening but last fall I started digging out an area to create a mound to expand my banana plant gardening. I layered the bottom and worked up with layers of sticks,  limbs, leaves, alternating with good draining soil and composted manure. Only thing is, I used a good bit of Castor Bean plant trimmings...stalks and leaves primarily but probably a few seed pods as well. I've been reading mixed opinions about this because they are poisonous. I'm ready to plant my young banana plants but concerned.
I honestly didn't think it would hurt the plants but now I wonder if it will kill worms ND beneficial life in the soil...?
Any conclusions based on experience?

I have found numerous claims that castor oil will not kill worms.

Thanks. I understand, but castor oil is not poisonous,...the plant, however, is. 😒

Hi Mark,

That was my point ..... hopefully someone else can come up with something more relevant.