Gartner’s most recent hype cycle report (“Gartner’s 2014 Hype Cycle for Emerging Technologies” http://www.gartner.com/newsroom/id/2819918) suggested that Big Data was heading into the “Trough of Disillusionment” before reaching the “Productivity Plateau” in 5-10 years time. This is not unusual for a new technology. However, based on a growing body of evidence I don’t believe that it will take 5-10 years from now for Big Data technologies to make that journey. Data Tactics [link] first deployed the production version of its Hadoop based Big Data Engine (BDE) for the U.S. Department of Defense in 2007. Yahoo deployed Hadoop in 2008, and Facebook in 2010. I believe the 5-10 year disillusionment clock started ticking on these technologies around 2008 and we are poised to enter the Productivity Plateau soon.
Perhaps more significantly, I see more institutes that would never be considered as ‘leading edge’, taking the plunge with Hadoop based technologies. This recent article “NHS tears out its Oracle Spine in favor of open source” (http://www.theregister.co.uk/2013/10/10/nhs_drops_oracle_for_riak/) published in The Register shares the success that the British National Health Service(NHS) has experienced with its NoSQL initiative.
The NHS has achieved dramatic results. Most companies that we see are deploying NoSQL systems alongside existing applications or for green field solutions but the NHS went for a rip-and-replace of an existing mission critical application used by some 20,000 organizations to provide patient care. Not only has the NHS seen performance improvements but they’ve also experienced drastic costs savings: 50% on software and 95% on hardware.
More success like this will accelerate Big Data’s time to the “Plateau of Productivity”.