The Vindication Of Nate Silver

I was all set to write a closer examination of statistician and blogger Nate Silver’s most recent election predictions, a ramp up to during which he was lambasted by a garden variety of mostly conservative voices for either being politically biased, or establishing his predictions on a loose set of statistical shingles.

Only to be informed that one of my esteemed colleagues, David Pittman, had already written such a compendium post.  So hey, why reinvent the Big Data prediction wheel?

Here’s a link to David’s fine post, which I encourage you to check out if you want to get a sense of how electoral predictions provide an excellent object lesson for the state of Big Data analysis. (David’s post also includes the on-camera interview that Scott Laningham and I conducted with Nate Silver just prior to his excellent keynote before the gathered IBM Information On Demand 2012 crowd.)

I’m also incorporating a handful of other stories I have run across that I think do a good job of helping people better understand the inflection point for data-driven forecasting that Silver’s recent endeavor represents, along with its broader impact in media and punditry.

They are as follows:

 “Nate Silver’s Big Data Lessons for the Enterprise”

 “What Nate Silver’s success says about the 4th and 5th estates”

“Election 2012: Has Nate Silver destroyed punditry?” 

Nate Silver After the Election: The Verdict

As Forbes reporter wrote in his own post about Silver’s predictions, “the modelers are here to stay.”

Moving forward, I expect we’ll inevitably see an increased capability for organizations everywhere to adopt Silver’s methodical, Bayesian analytical strategies…and well beyond the political realm.

Live @ Information On Demand 2012: Watson’s Next Job

As I mentioned in my last post, yesterday was day 3 of Information On Demand 2012 here in Las Vegas.

There was LOTS going on out here in the West.

We started the day by interviewing keynote speaker Nate Silver (see previous post) just prior to his going on stage for the morning general session. Really fascinating interview, and going in to it I learned that his book had reached #8 on The New York Times best seller list.

In the IOD 2012 day 3 general session, IBM Fellow Rob High explains how IBM’s Watson technology may soon help drive down call center costs by 50%, using the intelligence engine of Watson to help customer service reps faster respond to customer queries.

So congrats, Nate, and thanks again for a scintillating interview.

During the morning session, we also heard from IBM’s own Craig Rinehart about the opportunity for achieving better efficiencies in health care using enterprise content management solutions from IBM.

I nearly choked when Craig explained that thirty cents out of every dollar on healthcare in the U.S. is wasted, and despite spending more than any other country, is ranked 37th in terms of care.

Craig explained the IBM Patient Care and Insights tool was intended to bring advanced analytics out of the lab and into the hospital to help start driving down some of those costs, and more importantly, to help save lives.

We also heard from IBM Fellow and CTO of IBM Watson Solutions’ organization, Rob High, about some of the recent advancements made on the Watson front.

High explained the distinction between programmatic and cognitive computing, the latter being the direction computing is now taking, and an approach that provides for much more “discoverability” even as it’s more probabilistic in nature.

High walked through a fascinating call center demonstration, whereby Watson helped a call center agent more quickly respond to a customer query by filtering through thousands of possible answers in a few second, then honed in on the ones most likely that would answer the customer’s question.

Next, we heard from Jeff Jonas, IBM’s entity analytics “Ironman” (Jeff also just competed his 27th Ironman triathlon last weekend), who explained his latest technology, context accumulation.

Jeff observed that context accumulation was the “incremental process of integrating new observations with previous ones.”

Or, in other words, developing a better understanding of something by taking more into account the things around it.

Too often, Jeff suggested, analytics has been done in isolation, but that “the future of Big Data is the diverse integration of data” where “data finds data.”

His new method allows for self-correction, and a high tolerance for disagreement, confusion and uncertainty, and where new observations can “reverse earlier assertions.”

For now, he’s calling the technology “G2,” and explains it as a “general purpose context accumulating engine.”

Of course, there was also the Nate Silver keynote, the capstone of yesterday’s morning session, to which I’ll refer you back to the interview Scott and I conducted to get a summary taste of all the ideas Nate discussed.  Your best bet is to buy his book, if you really want to understand where he thinks we need to take the promise of prediction.

Live @ Information On Demand 2012: A Q&A With Nate Silver On The Promise Of Prediction

Day 3 at Information On Demand 2012.

The suggestion to “Think Big” continued, so Scott Laningham and I sat down very early this morning with Nate Silver, blogger and author of the now New York Times bestseller, “The Signal and the Noise” (You can read the review of the book in the Times here).

Nate, who is a youngish 34, has become our leading statistician through his innovative analyses of political polling, but made his original name by building a widely acclaimed baseball statistical analysis system called “PECOTA.”

Today, Nate runs the award-winning political website FiveThirtyEight.com, which is now published in The New York Times and which has made Nate the public face of statistical analysis and political forecasting.

In his book, the full title of which is “The Signal and The Noise: Why Most Predictions Fail — But Some Don’t,” Silver explores how data-based predictions underpin a growing sector of critical fields, from political polling to weather forecasting to the stock market to chess to the war on terror.

In the book, Nate poses some key questions, including what kind of predictions can we trust, and are the “predicters” using reliable methods? Also, what sorts of things can, and cannot, be predicted?

In our conversation in the greenroom just prior to his keynote at Information On Demand 2012 earlier today, Scott and I probed along a number of these vectors, asking Nate about the importance of prediction in Big Data, statistical influence on sports and player predictions (a la “Moneyball”), how large organizations can improve their predictive capabilities, and much more.

It was a refreshing and eye-opening interview, and I hope you enjoy watching it as much as Scott and I enjoyed conducting it!

Thinking Big @ Information On Demand 2012

Nate Silver, author of the blog “FiveThirtyEight,” will be one of the featured keynote speakers at this year’s IBM Information On Demand 2012 event in Las Vegas, Nevada, October 21-25. Silver correctly predicted the results of the primaries and the U.S. presidential winner in 2008 in 49 states through his statistical analysis of polling data, and at IOD will explain how to distinguish real signals from noisy data as well as how predictive analytics is used in politics.

That annual festouche and gathering of all things data is just around the corner.

Yes, that’s right, it’s almost time for IBM Information on Demand 2012.

So in order to start the drumbeat, I wanted to take a few moments and point you to some useful resources as you prepare to make your way to the Bay of Mandalay, and to optimize your time on the ground in Vegas.

First, the new (and official) IBM Information on Demand blog, which you can find here.

The blog includes easy access to some of the social media channels that will be covering the event (including Facebook, Twitter, LinkedIn and YouTube).

Of course, never forget the official IOD hashtag, #ibmiod, where you’ll be able to follow the endless stream of tidings leading up to, during, and after the event.

The blog also has links off to the IOD 2012 registration engine, as well as to the IOD SmartSite so you can start thinking about your IOD calendar now (I do NOT advise waiting until the last minute…talk about information overload!)

We’ve got some exciting guest speakers this year, including Nate Silver, statistics blogging extraordinaire who first found fame with his “FiveThirtyEight” blog, which is now part of The New York Times family of media properties.

Silver analyzes politics the way most of us should be analyzing our business: Through data…and lots of it.

His analysis of political polling data is unparalleled, and in the 2008 U.S. presidential election, Silver correctly predicted the results of the primaries and the presidential winner in 49 states.

His recent book, “The Signal and The Noise: Why Most Predictions Fail — But Some Don’t,” explores the world of prediction, “investigating how we can distinguish a true signal from a universe of noisy data.” Silver tackles some of the big questions about big data, so we’re very excited to have him join us in Vegas for IBM’s own big data marathon event.

At this year’s event, we’ll continue our trend of including tracks for specialized areas of interest, including forums for Information Management, Business Analytics, Business Leadership, and Enterprise Content Management.

And, of course, you’ll be able to find Scott Laningham and myself down in the EXPO center, where we’ll be talking to and interviewing many of the IBM and industry luminaries on the important data-related topics being discussed at the event.

Speaking of data, this will be my seventh IOD in a row, so I’m looking forward to seeing many of you once again.

Meanwhile, keep an eye here on the Turbo blog for future IOD-relevant posts.