Part of the problem of agriculture is what Daniel Quinn calls "locking up the food." Under agricultural systems the food belongs to someone and people have to work for it, rather than it being available to everyone in the community as in hunter-gatherer society and potentially in permacultural society.
This is the most prevalent reason behind famine, as pointed out above - often or usually there is plenty of food but people are unable to get it because of poverty or other political reasons.