Granite is potentially high in radioactive materials, and granite is much higher in radioactives than other types of rocks are in general. The radioactivity can be significant, and bad enough that there have been lawsuits regarding granite countertops in some cases. So unless you're screening your granite dust with a geiger counter and know what you're doing I would advise against it.
Also granite is generally low in nutrients, and most rock dust used for agricultural purposes is basalt-based. Even then rock dust isn't particularly efficient as a fertilizer, unless you have no soil at all, or pure quartz sand, or pure peat, or something like that. Or if the rockdust happens to be free and nearby.
Disclaimers aside, there's pretty much no limit to how much you can use. You can grow plants in pure rock dust, limited only by how well it holds
water and how much nitrogen is available.