Win a bunch of tools from Truly Garden and Loma Creek! this week in the Gear forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
permaculture forums growies critters building homesteading energy monies kitchen purity ungarbage community wilderness fiber arts art permaculture artisans regional education skip experiences global resources the cider press projects digital market private forums all forums
this forum made possible by our volunteer staff, including ...
master stewards:
  • Nicole Alderman
  • r ranson
  • Anne Miller
  • Pearl Sutton
  • James Freyr
  • Mike Haasl
  • paul wheaton
  • Dave Burton
  • Joylynn Hardesty
  • Jocelyn Campbell
  • Joseph Lofthouse
garden masters:
  • Carla Burke
  • Steve Thorn
  • Eric Hanson

The Netflix Prize Won by Using and Valuing Diversity

Posts: 2103
Location: Oakland, CA
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Wired has a great article on how the Netflix ($1 million for relationships between movie preferences) prize was won.

after years of work, the Netflix Prize competition has proffered hard proof of a basic crowdsourcing concept: Better solutions come from unorganized people who are allowed to organize organically. But something else happened that wasn’t entirely expected: Teams that had it basically wrong — but for a few good ideas — made the difference when combined with teams which had it basically right, but couldn’t close the deal on their own.


Taken on its own, the fact that a viewer rated a given movie on a Monday is a horrible indicator of what other movies they’ll want to rent — a crucial part of Netflix’ business (it says its recommendations are better indicators of what people will rent than their “most popular” lists). But combined with hundreds of other algorithms from other minds, each weighted with precision, and combined and recombined, that otherwise inconsequential fact takes on huge importance.

“One of the big lessons was developing diverse models that captured distinct effects,” said [Joe] Sill, “even if they’re very small effects.”

I am posting this in "meaningless drivel," but it isn't exactly outside the world of permaculture, for two reasons.

As a model for non-tech endeavors

The winning strategy, discovered about two years into the competition and adopted by more top-ranking teams as it proved its merit, was to foster an ecosystem of theories, and use heavy-duty statistics to condense any sense made by any of them into the final result. 

One theory applied better to some of the data, and other theories applied better to other portions of it.  The article points out that some of the teams that were farthest from winning on their own, had the most to offer a winning coalition, because their perspective on the problem was so unique.

It's an object lesson in how important valuing diversity of opinion and mindset is, even domains where the criteria for success are artificially clear-cut and mechanical.  If the information we have to work with is like sunlight and rain, sometimes we would do best to imitate a guild of plants.

As a design method

One other thing that struck me very sharply about this article is how similar this competition is to the task of growing food.

The data they have to work with are in a form known as a "sparse matrix," in that there are a huge number of potential combinations, only a few of which do we know anything about.  Imagine a huge table with every place we've seen plants listed on its own row, and each species or cultivar of plant listed on its own column.  How well does that plant grow in that location?  The answer is usually, "We don't know."  I haven't tried growing a Venus fly trap or sea kelp or a giant sequoia on my balcony, for example, but I do know tomatoes do well in the dirt near my yard.  Because it's many rows relating to many columns, mathematicians call this a "matrix", and it's sparse because it's mostly empty.

Now consider a table like this for each sort of soil ammendment, all bound together in a book.  And a book of that sort for each planting date, collected in a 365-volume set.  This is still a matrix, by the way.  A (digital) library like this was the sort that they searched through, and tried to make predictions of which plants would grow best:  If tomatoes do well there, maybe you should try potatoes.  This patch of ground will behave vaguely like others in the same USDA zone, and also vaguely like others with the same sort of soil, and also vaguely like others with the same rate of precipitation...which plants are best at fitting all those criteria, together?

There are other ways of grouping the results.  Some of which haven't even been discovered.  Some of which, if this contest's results are any indication, will be the sort of pattern that human intuition gets very little traction on.

I think the technology they have developed could produce an immensely powerful design tool for permaculturists.  I'm excited.
These are not the droids you are looking for. Perhaps I can interest you in a tiny ad?
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!