It´s not Big Data all that glitters
Technology glitters and it’s been growing in recent decades. This may serve as a reflection: the iPhone we carry in our pockets has a greater computing capacity than the computers that brought man to the Moon. What was science-fiction some years ago is real today and society sees it as normal.
There are two magic words which have been repeated ad infinitum: Big Data. The development of technology has led everything measurable in some way to be stored so to try to extract relevant information that can be used for a better functioning of life in general and of companies in particular. Technology has adapted to be able to store this information and process it efficiently. But Big Data is just that, Large Data. Large amounts of data have no value in themselves if we do not know how to extract valuable information from them. A current attitude towards Big Data is keeping this huge amount of information just in case. But the way of extracting and generating knowledge from data is more important than knowing how and where to store the information. The Data Science profile (which it is said to be the sexiest job in the 21st century) has experienced a growing demand in recent years because of technological progress and, specifically, because of the need for professionals who know how to extract that valuable information from the heterogeneity and chaos of Big Data.
It is true that only companies with large technological infrastructures (banks, Twitter, Facebook, LinkedIn…) have access to Big Data in the strictest sense of the term and, although Big Data is becoming cheaper and more democratic, companies with a modest data volume should be aware of the benefits and competitive advantages they may have if they put their data in the hands of a team of Data Scientists. Any SME could benefit from someone who knows how to extract valuable information from their data and return on investment would probably be high. One of the skills a data scientist must have is to know how to convey their analyses in a clear and didactic way which someone with no theoretical knowledge can understand and draw their own conclusions adding value to the company. We data scientists have learned that we must be exhaustive and meticulous when applying the methods that best suit the data analysed, but explaining our analysis in a simple way is all the more important. Having deep theoretical knowledge is useless without the ability to make it understandable to non-experts and we must be aware our costumer may not necessarily know what an algorithm is or to be able to write a line of code. This is precisely the biggest challenge we face, to be able to convey our conclusions efficiently.
Moreover, the appearance of new free software (R, Python…) has made the technology necessary for the work of Data Scientists cheaper and accessible to any company and has challenged software which has traditionally cornered the data analysis market and made their use harder for small and medium companies. A small revolution is underway in the way we make decisions since purely emotional reasons are being putting aside and, although business experience will continue being decisive, decisions based on objective data analysis will improve the functioning of any company willing to follow this revolution.
We have to get rid of the idea that Big Data are almost science fiction and that almost everything can be known through them. Big Data has served us to face even bigger challenges and adapt our techniques to that huge amount of data which has to be turned into information. May this article serve to debunk the idea that Big Data in themselves are useless if they are not accompanied by the necessary technology and knowledge to make them useful. It is to no avail that you know where a diamond is without the necessary tools to mine it from the rock.
Technology will continue advancing with MLOps and DevOps and it is in our hands the decision of jumping on the bandwagon to improve the decisions we made in our business or letting others do it and take the risk of being left behind.