Big Moves In Big Data: IBM New Data Acceleration, Hadoop Capabilities

Click to enlarge. IBM just announced new technologies designed to help companies and governments tackle Big Data by making it simpler, faster and more economical to analyze massive amounts of data. New data acceleration innovation results in as much as 25 times faster reporting and analytics.

IBM made a significant announcement earlier today concerning new technologies designed to help companies and governments tackle Big Data by making it simpler, faster and more economical to analyze massive amounts of data. The new data acceleration innovation results in as much as 25 times faster reporting and analytics.

Today’s announcement, which represents the work of hundreds of IBM developers and researchers in labs around the world, includes an industry-first innovation called “BLU Acceleration,” which combines a number of techniques to dramatically improve analytical performance and simplify administration.

Also announced was the new IBM PureData System for Hadoop, designed to make it easier and faster to deploy Hadoop in the enterprise. Hadoop is the game-changing open-source software used to organize and analyze vast amounts of structured and unstructured data, such as posts to social media sites, digital pictures and videos, online transaction records, and cell phone location data.

The new system can reduce from weeks to minutes the ramp-up time organizations need to adopt enterprise-class Hadoop technology with powerful, easy-to-use analytic tools and visualization for both business analysts and data scientists.

In addition, it provides enhanced Big Data tools for monitoring, development and integration with many more enterprise systems.

IBM Big Data Innovations: More Accessible, Enterprise-ready 

As organizations grapple with a flood of structured and unstructured data generated by computers, mobile devices, sensors and social networks, they’re under unprecedented pressure to analyze much more data at faster speeds and at lower costs to help deepen customer relationships, prevent threat and fraud, and identify new revenue opportunities.

BLU Acceleration enables users to have much faster access to key information, leading to better decision-making. The software extends the capabilities of traditional in-memory systems — which allows data to be loaded into Random Access Memory instead of hard disks for faster performance — by providing in-memory performance even when data sets exceed the size of the memory.

During testing, some queries in a typical analytics workload were more than 1000 times faster when using the combined innovations of BLU Acceleration.

Innovations in BLU Acceleration include “data skipping,” which allows the ability to skip over data that doesn’t need to be analyzed, such as duplicate information; the ability to analyze data in parallel across different processors; and greater ability to analyze data transparently to the application, without the need to develop a separate layer of data modeling.

Another industry-first advance in BLU Acceleration is called “actionable compression,” where data no longer has to be decompressed to be analyzed.

Not IBM’s First Big Data Rodeo

The new offerings expand what is already the industry’s deepest portfolio of Big Data technologies and solutions, spanning software, services, research and hardware. The IBM Big Data platform combines traditional data warehouse technologies with new Big Data techniques, such as Hadoop, stream computing, data exploration, analytics and enterprise integration, to create an integrated solution to address these critical needs.

IBM PureData System for Hadoop is the next step forward in IBM’s overall strategy to deliver a family of systems with built-in expertise that leverages its decades of experience reducing the cost and complexity associated with information technology.

This new system integrates IBM InfoSphere BigInsights, which allows companies of all sizes to cost-effectively manage and analyze data and add administrative, workflow, provisioning and security features, along with best-in-class analytical capabilities from IBM Research.

Today’s announcement also includes the following new versions of IBMs Big Data solutions:

• A new version of InfoSphere BigInsights, IBM’s enterprise-ready Hadoop offering, which makes it simpler to develop applications using existing SQL skills, compliance security and high availability features vital for enterprise applications. BigInsights offers three entry points: free download, enterprise software and now an expert integrated system, IBM PureData System for Hadoop.
• A new version of InfoSphere Streams, unique “stream computing” software that enables massive amounts of data in motion to be analyzed in real-time, with performance improvements, and simplified application development and deployment.
•  A new version of Informix including TimeSeries Acceleration for operational reporting and analytics on smart meter and sensor data.

Pricing and Availability 

All offerings are available in Q2, except the PureData System for Hadoop, which will start shipping to customers in the second half 2013. Credit-qualified clients can take advantage of simple, flexible lease and loan packages with no up-front payments for the software and systems that deliver a new generation of data analytics.

IBM Global Financing offers attractive leasing programs with 90-day payment deferrals for the PureData System for Hadoop, as well as zero percent loans for the broader portfolio of IBM big data solutions.

Just How Big IS Big Data?

So just how big IS big data?

This is your opportunity to find out, and, to contribute.

IBM’s Institute for Business Value is conducting a study on big data, and we’d like to hear from you.

The idea behind the study is simple: To develop a fact-based analysis of big data activities in the global marketplace.

Through this research, IBM hopes to help the marketplace better understand some key tenets behind the big data movement: To gain an organizational view of big data and organizations’ primary objectives for investments in this burgeoning area.  To understand better the drivers and leaders of big data activities. To understand the current and planned state of big data activities, and patterns that suggest best practices of big data implementations.

The survey is slated to run through June 29, 2012, and takes approximately 10-15 minutes per respondent.

All responses will be viewed in the aggregate, and individual responses will not be disclosed beyond the survey analysis team without expression permission of the respondent.

The audience for the survey: Global business executives, management and analysts, as well as IT professionals, across all levels of the organizational hierarchy (from C-suite to data analysts).

Once the fielding is completed, the survey results will be analyzed by a wide team of subject matter experts from within IBM, along with a team of faculty from a globally recognized university.

This data will be combined with interviews and case studies to develop a final reporting of findings and big data benchmarks to be published in October of this year.

So, in short, this is your opportunity to be part of the benchmarks that will define the big data era, one that you can use to compare with your own organization.

All participants will receive a copy of the final study, and will also be eligible to download the IBM e-book entitled “Understanding Big Data.”

Here’s the link if you’d like to be part of this exciting big data discovery!

IBM To Acquire Big Data Software Provider Vivisimo

IBM continues to romp and stomp its way through the Big Data space.

IBM announced its intent to acquire enterprise discovery software provider Vivisimo earlier today. Vivisimo has recently positioned itself as the "independent provider of enterprise search solutions" which helps "organizations unlock and optimize the true business value of all their information, regardless of application or source, in order to drive innovation, real-time decisions and actionable insight."

Today, it announced a definitive agreement to acquire Vivisimo, a leading provider of federated discovery and navigation software that helps organizations access and analyze big data across the enterprise.

Vivisimo is a privately held company based in Pittsburgh, Pennsylvania.

Financial terms were not disclosed.  Of course.

IBM estimates 2.5 quintillion bytes of data are created every day from a variety of sources including sensors, social media, and billions of mobile devices around the world, making it difficult for businesses to navigate and analyze it to improve competitiveness, efficiency, and profitability.

IDC estimates the market for big data technology and services will grow at an annual rate of nearly 40 percent to reach $16.9 billion by 2015

That’s where Vivisimo software comes into play. Vivisimo excels in capturing and delivering quality information across the broadest range of data sources, no matter what format it is, or where it resides.

It automates the discovery of data and helps employees navigate it with a single view across the enterprise, providing valuable insights that drive better decision-making for solving all operational challenges.

Today’s news accelerates IBM’s big data analytics initiatives with advanced federated capabilities allowing organizations to access, navigate, and analyze the full variety, velocity and volume of structured and unstructured data without having to move it.

The combination of IBM’s big data analytics capabilities with Vivisimo software will further IBM’s efforts to automate the flow of data into business analytics applications, helping clients better understand consumer behavior, manage customer churn and network performance, detect fraud in real-time, and perform data-intensive marketing campaigns.

“Businesses need a faster and more accurate way to discover and navigate big data for analysis” said John Kealey, Chief Executive Officer, Vivisimo. “As part of IBM, we can bring clients the quickest and most accurate access to information necessary to drive growth initiatives that increase customer satisfaction, streamline processes, and boost sales.”

Vivisimo brings over a decade of experience and innovation in data navigation and visualization technologies for both structured and unstructured data, making it easier for business users to get value from all of their data and content. Vivisimo’s ability to index and search data across multiple repositories is a distinguishing capability, applicable to all industries and clients.

Vivisimo has more than 140 customers in industries such as government, life sciences, manufacturing, electronics, consumer goods and financial services. Clients include Airbus, U.S. Air Force, Social Security Administration, Defense Intelligence Agency, U.S. Navy, Procter & Gamble, Bupa, and LexisNexis among others.

Upon the closing of the acquisition, approximately 120 Vivisimo employees will join IBM’s Software Group. IBM will incorporate Vivisimo technology into its big data platform.

Visit here for more information on IBM’s big data platform or here for more information on Vivisimo.

There’s some early coverage here from TechCrunch.

Information On Demand 2011 Breaking News: IBM Accelerates Big Data Analytics

Today, here at Information On Demand 2011 in Las Vegas, Nevada, IBM unveiled new software that brings the power of managing and analyzing big data to the workplace.

Whether in the office or on the road, employees can now gain actionable insight anytime, anywhere from the broadest range of data and put it to work in real-time.

IBM Senior VP Steve Mills explains the "why" of business analytics at today's press conference here at Information On Demand 2011 in Las Vegas, Nevada.

The new offerings span a wide variety of big data and business analytics technologies across multiple platforms from mobile devices to the data center to IBM’s SmartCloud.

Now employees from any department inside an organization can explore unstructured data such as Twitter feeds, Facebook posts, weather data, log files, genomic data and video, and make sense of it on the fly as part of their everyday work experience.

With today’s news, IBM is placing the power of mobile analytics into the hands of iPad users with a free software download at Apple’s iTunes Store. The new software is designed to help employees in key industries such as financial services, healthcare, government, communications, retail, and travel and transportation use and benefit from business analytics on the go.

Organizations of all sizes are struggling to keep up with the rate and pace of big data and use it in a meaningful way to improve products, services, or the customer experience. 

Every day, people create the equivalent of 2.5 quintillion bytes of data from sensors, mobile devices, online transactions, and social networks; so much that 90 percent of the world’s data has been generated in the past two years.

Every month people send one billion Tweets and post 30 billion messages on Facebook. Meanwhile, more than 1 trillion mobile devices are in use today and mobile commerce is expected to reach $31 billion by 2016.

A 2010 IBM/MIT Sloan Management Review survey of 3,000 executives across 30 industries from 100 countries reveals that 60 percent of respondents said they have more data than they can effectively use.

A new IBM study of 1,700 chief marketing officers from 19 industries and 64 countries further exposes this issue with 71 percent saying their organizations are unprepared to handle the explosion of big data. 

To address these challenges, IBM is delivering new analytics and information management offerings, and skills resources to make it easier to explore and capitalize on big data:

•  New Hadoop-based analytics software on the cloud that can be up and running in less than 30 minutes.  The new software helps employees tap into massive amounts of unstructured data from a variety of sources including social networks, mobile devices and sensors.
• New mobile analytics software for iPad users that makes it easy to explore any type of data on the go with location-aware analytics. Clients can download the free app here: http://itunes.apple.com/us/app/ibm-cognos-mobile/id455326089?mt=8
• New predictive analytics software with a mapping feature that can be used across industries for marketing campaigns, retail store allocation, crime prevention, and academic assessment.
• New software that sifts through all types of data behind the scenes and ranks its quality, makes it secure, and ensures business decisions are based on trusted data.

Big Data Analytics On The Cloud

IBM InfoSphere BigInsights on the IBM SmartCloud Enterprise makes big data analytics accessible for any user inside an organization.

Like the on-premise version, BigInsights on the cloud analyzes traditional structured data found in databases along with unstructured data — such as text, video, audio, images, social media, click streams, log files, weather data — allowing decision makers to act on it quickly. Bringing big data analytics to the cloud means clients can capture and analyze any data without the need for Hadoop skills, or having to install, run, or maintain hardware and software.

BigInsights on the cloud is available in both basic and enterprise editions with the options of public, private and hybrid cloud deployments. The basic edition is an entry-level offering available at no-charge that helps organizations learn how to do big data analytics including “what-if” scenarios with its BigSheets component.

Clients can seamlessly move to the enterprise edition when ready and set up Hadoop clusters in under 30 minutes to start analyzing data with low usage rates starting at $0.60 (US) per cluster, per hour. Both versions include a developer sandbox where clients can develop a new generation of business analytics applications complete with tools and a test and development environment.

Today, market leaders in banking, insurance, retail, communications and digital entertainment are using BigInsights on the cloud to analyze massive amounts of unstructured data.

These clients are analyzing data flowing from social networks, sensors, mobile devices, log files, and voice and video systems to understand consumer sentiment, make computing networks and smart grids more secure, and create new customer experience programs.

IT professionals and students looking to build Hadoop skills can take advantage of IBM’s BigDataUniversity.com, a new web site where users can learn the basics of Hadoop, stream computing, open source software development, and database management techniques to prepare for careers as Data Scientists.

The site includes hundreds of easy-to-use tutorials, videos, and coding exercises geared to build Hadoop, BigInsights, DB2 and WebSphere skills, and many courses are free. More than 8,000  students worldwide have already registered from countries such as Brazil, Russia, China, India, Korea, and South Africa and the US.

Analytics In The Office And On The Road

IBM continues to advance business analytics for the 21st century workforce by delivering expanded mobile device support with IBM Cognos Mobile software for the iPad.

The software enables mobile workers to take their business analytics on the road whether offline or online, allowing for uninterrupted productivity. iPad users can enjoy a rich, visual business intelligence experience to analyze any data about their business including sales, customer, and financial data with reporting, dashboard and scorecards.

Cognos on iPad is designed to help employees in key industries such as financial services, healthcare, government, communications, retail, and travel and transportation use and benefit from analytics on the go.

For example, doctors and dentists can use it to analyze electronic medical records and show patients customized treatment plans and explain procedures based on that analysis; social workers can check the health and well being of children in foster homes throughout a city and update supervisors, police and courts on their status in real-time; and bankers and insurance agents can use it to analyze loan or policy data to create individual products or services for clients.

Cincinnati Zoo, one of the oldest zoos in the United States with more than 1.2 million visitors annually, uses Cognos on iPad to give management instant access, and a single view of visitor and business information to drive new revenue and improve member visits.

The flexibility of mobile business analytics allows managers to bring together sales and attendance data on their iPads from wherever they are inside the park to track purchase patterns and adjust marketing spend based on that information. Using Cognos software, the Zoo has increased in-park spending by 25 percent this year.

IBM Puts Predictive Analytics On The Map 

With today’s news, IBM is delivering new software that allows organizations to gain predictive intelligence on geographic data. Organizations can use the software to understand data, analyze trends, forecast, plan and validate assumptions to drive accurate conclusions.

SPSS Statistics 20.0 software includes a new mapping feature that gives users the ability to add a geographic dimension to analysis and reporting, and allows users to target, forecast, and plan by geographical areas.

This mapping feature can be used across industries to analyze data and create statistics for marketing campaign effectiveness, store allocation decisions in retail, to detect crime hot spots, and for student test score assessments. The software comes with views of the United States, countries, continents, and prebuilt map templates where users can quickly populate them with data including geospatial information from ESRI files.

Healthcare organizations can use the new software to visually pinpoint areas of high accident or illness rates, or identify differences in care across different regions of a state or country.

Government employees can analyze past and present census data by city block or in dense county populations, and identify high crime areas to allocate more law enforcement, or update tax and zoning changes. Direct marketers can locate their most profitable customer base and store locations to allocate advertising resources, and academia can use it to concentrate recruiting and alumni efforts geographically.

The scene from this morning's press conference at IBM Information On Demand 2011 at the Mandalay Bay in Las Vegas, Nevada.

New Software Speeds Governance of Big Data

Big data analytics can be a competitive advantage, however, the quality of the analysis is only as good as the data it’s fed, and the data itself has to be available to those who can use it.

IBM is the only vendor with a market-leading information integration and governance platform for big data that ensures only trusted information is delivered to business users and applications across the enterprise.

New IBM InfoSphere Information Server 8.7 software enables integration with Big Data as both a source and a target for information integration. The proven performance and parallel engine of Information Server provides the massive scalability required for big data. Also new in this release is a next generation connector to Netezza, built for balanced optimization and high performance, and packaged specifically for Netezza implementations, and an operations console to view system usage across all integration jobs, to improve productivity of integration projects.

New IBM InfoSphere Master Data Management 10 software unifies IBM’s market leading MDM capabilities into a single product that handles any MDM requirement. New features include integration with Business Process Management software for MDM-centric business processes, greater connectivity to consuming applications via adaptable service interfaces, and a shared matching engine that maintains the single version of the truth. MDM technology improves the outcome of Big Data analytics by providing a better understanding of customers, products, suppliers, employees and accounts for further analysis.

Clients Turn To IBM To Analyze Big Data 

With today news, IBM also announced that hundreds of new clients are turning to IBM to gain actionable insight on the broadest range of big data.

Whether it’s collecting data to manage the placement of windfarms, gauge customer sentiment on social media sites, or predict potentially fatal infections in hospitals, IBM is helping clients across every industry to put their data to work.

Clients such as Hertz, Beacon Institute, KTH Royal Institute of Technology, Marine Institute Ireland, Technovated, [x+1], TerraEchos, University of Ontario Institute of Technology and Uppsala University are using IBM analytics technologies to address the growing volume, velocity and variety of big data, and use it to make decisions that are transforming their industries.

Additional examples include:

• Danish energy company Vestas Wind Systems is using IBM’s big data software to analyze petabytes of weather data to improve wind turbine placement for optimal energy output. Analysis that used to take weeks can now be done in under one hour.
• XO Communications has reduced its customer churn rates by nearly 50 percent using IBM SPSS predictive analytics software. The company can predict customer behaviors, spot trends, and identify those likely to switch to another carrier, allowing them to take steps to keep their most valuable customers.
• [x+1], an end-to-end digital marketing platform provider, is helping their clients realize a 20 percent growth in digital sales by analyzing massive volumes of advertising data in real-time using IBM Netezza technology.
• Worldwide advertising agency Ogilvy is using IBM’s analytics software for the iPad to help employees assign resources, track utilization rates, and identify new revenue opportunities on the fly.

To read about more clients that are tackling big data challenges with IBM analytics technologies, download the new IBM Big Data Book at http://www.ibm.com/bigdata.

Follow breaking news from Information On Demand 2011 on Twitter at #iod11.

Big Golf, Big Data

How did it get to be Monday so quickly?

Well, it’s here now, and I guess we just have to deal with it.

A quick congrats to David Toms for winning the Crowne Plaza Colonial PGA tourney in Ft. Worth, Texas, over Rapture weekend.  The end of the world may not have appeared, but boy did Toms’ putter show up.

For diehard golf fans, the Colonial is a special golf tournament, celebrating the life and career of one of the greatest golfers ever, Ben Hogan, who was born in nearby Stephenville and who won the Colonial five times (the last time in 1959).  Hogan died in Ft. Worth back in 1997.

Toms, who lost The Players Championship in a playoff last week, came back this week in Ft. Worth swinging (literally!) to beat Charlie Wi and a diverse field to get his name etched on the Colonial Wall of Champions.

I, for one, am glad Toms came back to win, as I was not really eager to see the world end suddenly, particularly during the weekend of such a classic golf tournament.

But the world can end in different ways, particularly with all that voluminous data floating around out there that’s so difficult to keep track of.

On Friday, IBM unveiled some new software and services to help clients with their “big data” needs.

Why this now?  Consider the fact that 83 percent of 3,000 CIOs surveyed in IBM’s 2011 Global CIO Study said that applying analytics and business intelligence to their IT operations is the most important element of their strategic growth plans over the next three to five years.

Specifically, IBM announced new, patented software capabilities to analyze massive volumes of streaming data with sub-millisecond response times and Hadoop-based analytics software to offer scalable storage to handle tens-of-petabytes level data.

IBM also announced 20 new services offerings, featuring patented analytical tools for business and IT pros to infuse predictive analytics throughout their IT operations.

You can read more about this announcement here.

And go here to get the bigger view on bigger data.

I don’t know how much more effective these new tools will be in predicting the next ending of the world, but I’m pretty confident they can help companies make more sense out of all those endless information streams overwhelming them.

Zero Privacy, Big Data, Oxygen-Deprived Birthers

Woo boy, there’s a whole bunch of news breaking on this pre-Royal Wedding hump day.

First, the birther movement took a shot through its bow after U.S. President Barack Obama released a copy of his long-form birth certificate.

You could almost feel Donald Trump’s personal jet decompress on approach to New Hampshire as wind of this release was announced.  After he was off the plane and in front of the microphones, he had this to say: “I am very proud of myself.”

Ooh boy, it’s going to be a long presidential campaign season, particularly as Trump tries to navigate his way through the mine fields of the global media.

But before he can claim to fire anyone, he first probably needs to get his bearings a bit, what with being on the move so much.

And what better way to do so than to buy an iPhone!

Unlike the Donald, Apple is working to get ahead of the end of the story and released this long Q&A about iPhone location data.

First question:

Q. Why is Apple tracking the location of my iPhone?

A: Apple is not tracking the location of your iPhone. Apple has never done so and has no plans to ever do so.

The Q&A went on to explain just how Apple is not tracking the location of iPhones, and that UK data scientists Alasdair Allan and Pete Warden instead found a database of Wi-Fi hotspots and celltowers that can help the iPhone quickly calculate one’s location.

We say pry-vacy, the Brits say priv-acy, you say tomato, I say tomoto.

Apple insists they will shortly be offering up a fix to minimize how much of this data is stored, and also to encrypt that data moving forward.

See, I seem to have the opposite problem.

Whenever I need to use my iPhone, I can never seem to find the darn thing. So maybe they need to offer up a software fix that allows me to locate it when I’ve once again left it embedded somewhere deep in my couch.

Good thing Apple, then, seems to be doubling-down on the practical benefits of privacy invasion.

That is to say, also found in the Q&A: Apple is now collecting anonymous traffic data to build a crowd-sourced traffic database with the goal of providing iPhone users an improved traffic service in the next couple of years. 

Regardless of your own privacy paranoia (my privacy threshold is pretty moderate, and even this story didn’t get that much under my skin), Apple may have not done itself any favors waiting a week to respond, as the imbroglio has only blown all out of proportion during the interim.

But the Apple PR team seems to consistently operate on at least a one-week delay, so no new news there.

And I still love their products, Orwellian or no.

But yes, I do chuckle when I think back to their commercial targeting IBM as the consummate Big Brother in Apple’s 1984 Superbowl TV spot.  I hope Ogilvy Vice Chairman and Chief Creative Officer Steve Hayden gets a laugh from it as well.

Steve was at Chiat-Day working for Apple in those days and guided the development of that classic commercial of Apple woman freeing humanity from the IBM corporate drones (although we much prefer the work Steve and Ogilvy did for IBM on our e-business campaign.)

Speaking of loving products, according to the Wall Street Journal, Yahoo is considering spinning out the engineering group responsible for Hadoop and has apparently been in talks with Benchmark Capital about forming a company.

IBM and Cloudera are already major Hadoop supporters, and considering that Yahoo was the Hadoop dance that brung us all, would say welcome (back?) to the Big Data Analytics party.

What with all that iPhone data being gathered, and all the various short, long, medium, large and other formats of the President Obama’s birther certificates, and all the variant ways the Donald is congratulating himself for having made the President, err, blink  — well, there’s going to be ample big data to analyze for years to come.

I just wonder if it will help lead us anywhere…anywhere but deep back into the folds of my couch.