Data Preparation – What Is It And Why Is It So Important?
2 July 2021
When you’re cooking, preparation is an essential step. Ingredients need to be collected, peeled, marinated and put where you will be able to reach them when the oil is hot or the oven reaches the right temperature.
This is also true in any business analytics and data-driven process – today data comes from increasingly disparate sources and in an ever-growing variety of forms. The insights you are looking for could lie in Images, communications, machine-to-machine interactions and real-time sensor data. Most probably, they will lie in a combination of more than one of these sources. This means, in order to get them to work together, they need the same care and attention as garden-fresh vegetables and prime cuts of meat do, before you throw them in the pot.
As every project is likely to be different and involve different data, there are no hard and fast checklists for each step you need to take to ensure your data is sufficiently prepped. However, commonly, any operation that takes place on data between it being ingested into your system, and processed through a particular analytical system, can be considered as the data prep stage for that process.
All of them share a common aim – to ensure your analytics processes receive error-free data in a consistent format which they will be able to read, understand and work with. Frequently, this will include:
Data prep strategy
As all projects are different the first step is always to start with strategy. In terms of data preparation this means formulating a workflow process which will cover all of the steps your project needs, and how this will be applied to every different type, or source, of data. To follow my cooking analogy, this would be the equivalent of making sure you have all of the ingredients listed in the recipe, and know what you need to do to them before they go in the pot.
Data cleansing
This means removing data which is inaccurate, damaged or corrupt, or erroneous in some other way which means it is undesirable for it to be taken into account during analytics. This process should pick up errors ranging from mistakes made during human data input, to corrupt data caused by faulty sensors, data transfer systems or storage.
Metadata creation
Labelling of data to make it easier for your analytics systems to know what to do with it, when they receive it at the end of your data prep process. Metadata tags data with “information about information” – for example, when and where a picture or video was taken, or the age and rough geographical location of the sender of a customer complaint email.
Data transformation
This step involves putting data into the correct format for your analytics systems to work with. This means taking the data in whatever format it has been ingested – by scanners, sensors, cameras or manual human data input, and putting it into whichever database format your analytics engines will understand. Data can be compacted to save space and improve speeds at this point, and any elements which will not be read by your analytics processes can be discarded.
Data standardisation
Analytics algorithms and software could expect dates, names, geographical location and a myriad of other features to be presented in a uniform way – checking that all dates are in an eight-digit format rather than six digits, so as to avoid confusion during analysis, comes under this heading. Data can be checked at this point to make sure it falls into appropriate ranges – for example if you are only looking at customers in a certain area, do all of the zip codes meet your requirements?
Data augmentation
Is there anything else which can be added to your data – perhaps from publicly available datasets – which can make it more likely to reveal insights during analysis? Or, it may be possible to extrapolate additional facts based on what is already known – carrying this out ahead of the target analytics will save processing time and ensure you have the highest quality data before your algorithms go to work.
This sounds a bit long-winded – Do I have to do it all myself?
Fortunately, not! Firstly, if your data initiative only involves one type or source of data – just video, or just the names and addresses of customers, or just transactional records, chances are that there are tools already out there which will handle your data just fine in its raw state.
However, with most Big Data projects, the volume, variety and velocity of the data involved is too great for it to be practical to carry out these tasks manually. In these cases, thankfully, a large and growing “self-service” market for data preparation tools has emerged.
Due to the uniform nature of the operations and the repetitive tasks involved, data preparation is an ideal candidate for process automation, and “one-stop-shop” solutions, often delivered through simple web interfaces, requiring a minimum of data science training, are becoming increasingly common.
Just as with cooking, when it comes to business intelligence and data, good solid preparation can often be the difference between success and failure. Whichever of these processes you decide are necessary in your situation, a consistent data prep strategy should be a priority for anyone involved in digital transformation and data-driven discovery.
Where to go from here
If you would like to know more about data analysis and analytics, cheque out my articles on:
- How is Big Data used in practise? 10 use cases everyone must read
- Big Data: Too many questions, not enough answers
- The 6 Key Data Science Skills Every Business Needs Today
Or browse the Big Data and Analytics section of this site to find more articles and many practical examples.
Related Articles
The Employees Secretly Using AI At Work
Imagine walking into your office and noticing your colleague Sarah effortlessly breezing through her tasks with uncanny efficiency.[...]
Battling AI Fakes: Are Social Platforms Doing Enough?
Since generative AI went mainstream, the amount of fake content and misinformation spread via social media has increased exponentially.[...]
Creating The Universal AI Employee Of The Future
Imagine a world where your most productive employee never sleeps, never takes a vacation, and can seamlessly adapt to any role you need.[...]
20 Generative AI Tools For Creating Synthetic Data
The AI revolution that we’re currently living through is a direct result of the explosion in the amount of data that’s available to be mined and analyzed for insights.[...]
How To Tell Reality From Fiction Amid The AI-Driven Truth Crisis
The artificial intelligence narrative swings between utopian dreams and dystopian nightmares, often overshadowing the nuanced reality of its current capabilities and limitations.[...]
7 Ways To Turn The ‘Bring Your Own AI’ Threat Into An Opportunity
As AI tools become increasingly accessible, companies face a new trend: BYOAI, or bring your own AI.[...]
Sign up to Stay in Touch!
Bernard Marr is a world-renowned futurist, influencer and thought leader in the fields of business and technology, with a passion for using technology for the good of humanity.
He is a best-selling author of over 20 books, writes a regular column for Forbes and advises and coaches many of the world’s best-known organisations.
He has a combined following of 4 million people across his social media channels and newsletters and was ranked by LinkedIn as one of the top 5 business influencers in the world.
Bernard’s latest book is ‘Generative AI in Practice’.
Social Media