The beavers are the only reason the pond exists, and it WILL fail if the beavers are removed.
In the natural world beaver and duck commonly coexist on the same
water sources; as beaver are primarily nocturnal and ducks primaily daytime users, there
should be no issue coexisting.
Concerns might be giardia, dam failure, flooding, tree removal, property damage to wooden structures.
Giardia (a water born bacterial (?) infection) also known as "beaver fever" infecting other mammals on the property, should they drink from the pond (there is a vaccine for dogs).
Dam failure could occur IF beavers left or there was a massive influx of water due to weather or upstream issues. Plan any building on site to NOT be on low ground (unless stilts are used), or near pond to account for this concern.
Flooding could occur due to dam failure, weather or dam enlargement. If the latter is a problem the installation of a "beaver baffle) would solve any concerns over the potential increase in pond size.
Property damage is unlikely, directly; but indirectly, could see damage due to tree falling or water encroachment, both could be easily averted with monitoring.
On the plus side, assuming the colony remains healthy, you would have access to a ready made reservoir, of potentially, unlimited water for
gardening, and (may require filering) livestock also.
The key is are you upslope or upstream for the bulk of the property you intend to use for building, gardening and building? Personally, I would consider them a benefit and a blessing on a property I was purchasing.