Data...What? Why You Should Keep Doing Data Integration

Dad, why do I have to integrate data? It's boring! Many IT professionals have been asking themselves this question for years. Although, the more pertinent questions to ask are: what is more useful -- centralized or decentralized information? And why should any enterprise invest money on data integration? In the era of microservices, federation, virtualization, meshes and auto provisioning, everything tends to move to a more decentralized model. This seems like a great approach to provide more agility and self-service to areas that are related to business, and even to IT itself, without always going through the pains that data integration imposes. But is this true? Let’s rewind a little bit and find out. But first, let’s answer the question: what is data integration?
 
What’s Data Integration?
 
Data integration is a process. It requires combining data from different sources into a single and unified view while going through ingestion, cleansing and transformation processes. This doesn’t mean that all your data must reside in a single technology or a single platform, but you must have a way of “joining” all your data by using shared keys or any similar method. Integration needs strategy as it is a thoughtful process, and you may want to spend time on the data that will provide value instead of wasting money on integrating all your data without knowing if it will be useful. Besides that, not integrating data may result in disparate results and low-quality data, which will ensure you lose lots of money: "Garbage-in still means garbage-out – and the cheapest data are the data that we re-use," says Teradata’s own Martin Willcox.
 
What happens when you don’t integrate?
 
Let me walk you through a story that happened to me last year. I had decided to move from my house to a bigger one while also deciding to continue with my current ISP. Long story short, before moving I logged in to my account and filled out a form to declare I was moving. After receiving no sort of email confirmation, I waited a couple of days for them to call me. After realizing that there may have been some issues, I filled out the form again. Same story, no answer, no phone calls. So, I finally made the call myself, waited for some minutes and talked to a service rep. Very kindly he confirmed there was no request, entered all my information and the case was closed. Lucky for me, at my new home, the tech rep showed on time the day we planned, and everything went smoothly. I got my service running so I was happy.

After a few days, a sales rep called me to talk about my plan to move and asked when we should plan to make the switch. I was quite surprised and kindly told her that I had already moved to my new house with the service up and running so she went ahead and closed the requirement. The next day, another sales rep called me. Guess what? He wanted to know when I was planning to move. Again, I informed him that I had already moved, and everything was already done, so he closed the requirement. Two weeks later (yes, there’s more!) a tech rep showed at my door to cut the service. I was shocked because I was up to date with the payments until he explained me that Mr. John Doe (the former owner of the house) finished his contract with the company so there shouldn’t be any service running to that address. Amazing, right?
 
Mark my words, this is a true story. We can easily spot here the problem around lack of data integration and the way siloed repositories are working disparately. Funny thing though, this is an enterprise that is showcasing on the news their modernization journey to the cloud and to an agile mindset to support better customer interactions.
 
So...it doesn’t sound so crazy to integrate your data, right?
 
What happened to this company? They’ve totally lost their way and created a mess with their data by focusing on technology solutions only. While embracing a lot of modern flashy words like “cloud journey”, “agility” and such they kept missing the point about how good data enable better decisions and better customer interactions. My colleague Diogenes makes a great point about integration in his post.
 
I like the idea of visualizing any enterprise as a pyramid. We can have many different management techniques, many business areas, many separate business units, independent branches … but everything is part of the same business. To have a healthy business you need to know how all the parts work together. Can you imagine driving a car just by buying all the pieces and parts? Nope, right? But many enterprises think that a business can be run having data spread all over, in many separate siloes, without any relation between domains and entities. You can do that by applying “brute force” -- trying to bring everything together in curious ways – but you’ll while waste a lot of money and damage customer satisfaction along the way (as my example illustrates).
 
After all, at the very beginning or in the end, you are going to integrate the data as everything integrates into a one and only reporting structure. I would suggest to you, it’s better to do it sooner than later.