We’ve introduced the concept of Big Data and explored its applications in the real world in some of our earlier blog posts. We’ve also discussed how Big Data can help your business or organization find new insights. We’ve also outlined the benefits of digital business transformation.
It can be hard to filter out what big data can and cannot do from the articles, free eBooks and social media shares on platforms such as LinkedIn.
In this post, we’ll cover:
- History of Big Data
- Big Data Promises vs. Reality
- The Future of Big Data
After reading this post, you’ll have a better understanding of the history of Big Data and be able to cast a critical eye over Big Data literature in the future and arrive at your own conclusions.
But first, to set the context, a little history of the phrase Big Data
- Back in 1990, Peter J Denning, a director at NASA’s Ames Research Center was concerned about how he and his fellow scientists could find meaning in constantly growing datasets. He published this in a paper called “Saving All the Bits”.
- By 1997, Michael Cox and David Ellsworth publish “Application-controlled demand paging for out-of-core visualization” in the Proceedings of the IEEE 8th conference on Visualization. They start the article with “Visualization provides an interesting challenge for computer systems: data sets are generally quite large, taxing the capacities of main memory, local disk, and even remote disk. We call this the problem of big data. When data sets do not fit in main memory (in core), or when they do not fit even on local disk, the most common solution is to acquire more resources.” It is the first article in the ACM digital library to use the term “big data.”
Source: Forbes
Hype and Reality
Now we’re done with the history lesson, let’s dissect some of the hype that was generated in the IT landscape when the term Big Data entered the lexicon.
Big Data will allow me to predict the future!
Analytics and being able to predict the future is at the heart of Big Data. One of the biggest hyped stories was that by mining Big Data, business and organizations would be able to predict the future as if the domain expert had a crystal ball.
While it’s true advancements in machine learning and artificial intelligence in recent years have certainly aided this, Big Data alone has not completely delivered on this.
Machine learning and big data complement each other and will continue to evolve as researchers discover more innovative ways to mine Big Data and to help better predict future outcomes.
Businesses are going to need to implement NoSQL databases such as Hadoop!
The concept of NoSQL reached fever pitch and when the term Big Data gained popularity. One example of this was Hadoop. Hadoop was a NoSQL database born of the Apache Nutch project, and though it tried to be the answer to everyone’s Big Data problems, it never really worked out. Let’s be clear, Hadoop is still around and has value in specific use cases (some might argue is just a file system), but most businesses realized that after experimenting with a NoSQL database implementation that their existing relational databases were already doing what they needed.
You’ll get left behind if you don’t adopt Big Data!
It can be difficult to stay abreast of developments in the IT world, new technologies and frameworks appear every other week. Everyone caught the Big Data bug in the mid-2000s; you are forgiven for thinking that you were missing out of you didn’t adopt Big Data into your IT strategy back then.
Marketing departments went into overdrive to sell the “Big Data vision” when often, for businesses, there was no real need to implement it, or their CRM already offered it.
Big Data will give you access to everything!
While it’s true that Big Data contains lots of unstructured datasets from a variety of sources, making sense of it all from a human perspective is incredibly difficult.
Organizations such as Facebook, LinkedIn and Twitter have also recognized the value of their datasets and you’ll find that getting access to this information now comes at a cost. 3rd party developers or businesses that wish to leverage social data must go through a partner such as GNIP.
Gartner
Gartner, the world’s leading research and advisory company that provides businesses and leaders with objective insights published their “Hype Cycle for Emerging Technologies” in 2016.
In the Hype cycle, which you can see below, the term Big Data was nowhere to be found.
You can see the emphasis is on machine learning, robotics, self-driving cars and artificial intelligence.
Source: Gartner
What does this mean for Big Data?
Tweet this: The absence of Big Data from Gartner’s Hype Cycle doesn’t mean that it’s no longer important.
Vast quantities of data continue to be generated, especially with the explosion of e-commerce of user-generated content found on social media.
It simply means that Big Data has evolved and is entering a more mature phase, machine learning, digital business transformation and artificial intelligence will exploit big data and help users derive further insights.
Big Data Predictions 2018
We can assume that Big Data is here to stay. It’s just not the latest buzz word or a shiny new toy that everyone wants to play with.
Tweet this: But what might the future hold for #BigData?
Read on for our predictions.
Data Lakes
With the explosion of IoT devices (Gartner predict 8.4 billion connected items in 2017), data-lakes will become the dumping ground for raw data these devices generate.
Raspberry Pi[e] anyone?
Multi-type databases
Microsoft SQL Server, Oracle, Cassandra, Hadoop. NoSQL, SQL, JSON, XML etc. There are many databases on offer, not to mention file structures. Businesses will want one central repository to wrap around these discrete databases for effective digital business transformation.
DataLayer for IoT
The IoT is growing exponentially. There are many manufacturers producing multiple models of devices connected to the internet. We’ll see the emergence or standardization of APIs that facilitate access to Data Lakes.
Employment
The Data Scientist is one of the sexiest professions at the time of writing in the tech space. As we see the democratization of analytical tools, the demand for the data scientist will decrease thereby freeing them up to perform analysis or research.
Non-Technical Users
Products such as Google Analytics, Tableau and Import.io are making it easier than ever to generate reports with a few button clicks; these will evolve to include machine learning agents that “auto-suggest” alternative reports based on existing datasets and past user interactions.
The Cloud
As the amount of data increases with the explosion of digital commerce, companies will have to embrace cloud technology for storage and backup of business-critical data.
Summary
In this post, we discussed the history of Big Data, the hype that surrounds it and what the future may hold for it.
Here at Growth Acceleration Partners, we have extensive expertise in data and analytics. We can offer you Data Science as a Service as well as digital business transformation.
Our Centers of Engineering Excellence (COEs) in Latin America focus on combining business acumen with top-notch expertise to help your business.
We can provide your organization with a team of highly qualified data scientists and engineers who have expertise in:
- data analytics
- digital business transformation
- data science
- information systems
- machine learning
- predictive modeling
- software development
- ..and much more!
If you’d like to find out more, then arrange a call with us or send us an email.
