If those walls have been sitting around for a long time, the plaster is no longer 'lime', but it has now become 'limestone', i.e., the CaO has picked up CO2 from the air to become CaCO3. Nothing wrong with limestone in your planting beds, I wish I had more around here to throw into mine.
If you're sure that the walls contained only old-fashioned lime plaster it shouldn't be a problem, however in a lot of old houses, lime plaster has been over-plastered with modern gypsum based plasters and I wouldn't put those in a veg or herb bed.