In this episode of Rationally Speaking, Julia and Massimo discuss the phenomena of crowdsourcing, and ask: What makes it work? Is it ever unethical? And what are the limits to the wisdom of crowds?
Julia's pick: "The Signal and the Noise: Why So Many Predictions Fail-but Some Don't"
Massimo's pick: The Phi2Phi App.
References: "Longitude"
Nice podcast, as usual!
ReplyDeleteI am going to read The Signal and the Noise as soon as I finish Pinker's book on the decline of violence (recommended as well). Silver's book looks excellent from my skim-through.
Regarding Longitude, it's definitely worth reading, and the movie is pretty good too. The only thing I would say is that the treatment of Harrison's 'enemies,' especially Maskelyne, is pretty sensationalized.
A study was mentioned in this episode where N~10,000 and different groups rated a set of songs. The ratings were found to be very different and were sensitive to initial conditions, according to Julia.
ReplyDeleteI would like to know the author of this study so I can pull it on my research database.
Also Podcast suggestion: How to talk to statistics deniers (those that say, you can make the statistics say anything you want).
Can't wait to listen. I loved Longitude the book and the movie. It's interesting to see Newton in the role of establishment obstructionist who wouldn't allow a mere engineer to solve the problem of the ages.
ReplyDeleteThere's a study on wisdom of the crowds going on now, called Forecasting ACE. They found that information sharing improves forecasts over independent forecasts. Perhaps because it gets people to consider counterarguments.
ReplyDeleteFrom the study:
One experiment investigated whether information sharing among the contributors would be helpful or harmful to performance. On the one hand, there is evidence that forecasts resulting from discussions tend to be more similar and therefore reflect less varied input. On the other hand, discussions can also cause deeper thought. Discussions aren't practical on the ForecastingAce site, but we instituted simple sharing of rationales. The question was, would that help or hurt performance? For a period of time, half the contributors were invited to read other people's rationales and half were not. The results were clear: the ability to read other people's rationales (sharing information) improved performance. Consequently, information sharing became the default on the website on June 13, 2012.
Specifically, the results of the experiment were:
Contributors in the information-sharing group revised their forecasts more frequently, meaning they reflected more up-to-date information.
The Brier Score for the information-sharing group improved relative to the non-information-sharing group.
Those in the information-sharing group tended to begin forecasting earlier in the life of a problem, suggesting they were more engaged. On the negative side, unless they made later revisions in their confidence estimates (which many did), their forecasts tended to be less accurate, as there is strong evidence that forecasts are more accurate closer to the end date.
Pundit Tracker had this to say about Nate Silver:
ReplyDeleteWell-functioning markets are difficult to beat — just ask active managers in the mutual fund industry. And should the political betting markets become more liquid, they will only get better over time. Betfair and Pinnacle, which are believed by many to have greater liquidity, had significantly higher odds of Obama winning relative to InTrade.
We should add that at least one prominent person agrees with this assessment. That person is Nate Silver. Consider this excerpt from his book, The Signal and The Noise:
"Could FiveThirtyEight and other good political forecasters beat Intrade if it were fully legal in the US and its trading volumes were an order of magnitude or two higher? I’d think it would be difficult."
It is my belief that all consciousness is communal. We are each of us a confederacy of minds, and selves, whether it be various neural structures in one brain competing and collaborating, or, on a higher level, an experiencing self and a remembering self ("Thinking Fast and Slow"), or even higher, our personal interests versus the interests of our genome ("Robot's Rebellion"). I think consciousness, language and art are all tools for predicting and maximizing group cognition. Predicting and monitoring the behavior of other people is so important to us, we need to make up fictional people to track (that's my job!). You could argue that this is the primary concern of the legal system and ethics. What someone does is not as important as what they were thinking and what the rest of us are inclined to think of that. I think the recent notion that Reason itself has mostly a social value that supersedes the function of truth finding is very interesting. It's my hunch that a full philosophical commitment to this notion of communal cognition will resolve many issues-- from a workable definition of consciousness to a defeat of the fundamental problem of philosophical skepticism. Time and entropy will be part of the picture. Because, if there is only some reasonable consistency between the experiencing entity a moment ago and the one who is here now, systematic doubt is undone, other minds are real and naturalism is a viable project. This is why I am so infuriated by Boltzmann Brains (which I see as the Zeno's paradox of statistical mechanics, an example of some sort of mismatch between the model and the world). This, as well as a full commitment to Atheism, is why I am infuriated by the Simulation Argument. A brain is a Knesset. A species is an assertion. We think, therefore we are.
ReplyDelete