Efficiency in the Time of Abundance

Today, the cloud provides unlimited resources that can be provisioned on a whim. What a wonderful time to be alive! Unfortunately, with abundance comes carelessness and waste. Many of the database vendors today have not had to learn about efficient processing and having to manage a system with competing priorities. They instead rely on brute scale and isolation of work to accomplish their goals. But not Teradata. We are the gold standard of efficiency and operations at scale.

But why? Let’s hop into the way back machine to find the answer. We’ll go to 1979, back to when the founders of Teradata were embarking on a journey that would change the arena of business intelligence forever.

A vision of elegant simplicity

Back then the major database vendors of the day were focused on processing transactions faster so companies could run their business cheaper. These men had a different idea. What if you could bring all that data that was gathered and apply analytics at scale to understand your business and run it better.

The idea was to allow companies to take all that data and store it together in a massive (a terabyte of data!!) system to then run deep and broad analytics across the entire spectrum of the enterprise. The sheer audacity of that was viewed with skepticism, who would ever need a terabyte of data? Of course, to make this all a reality, they understood that such a system would have to be scalable, and therefore self-managed, so that even the largest systems would be operated just like the smallest ones.

Born in a time of scarcity

Here we are in 1979. Looking at the “modern” technology of the day, they have Intel 086 chips, running at the speed of .5 MIPS. Disk drives could hold up to 200 MB and weighed 50 pounds. The network was able to reach speeds of 11 MB/s. This was a daunting task indeed.

But all that led to a great success. For when one must do with limited resources, one must find a way to minimize “chatty” operations, maximize local processing, and eliminate bottlenecks at every step of the way.

Without going into all the gory details, they rewrote the book on how data was managed in the storage, they leverage the idea of parallelism and relational set operations, and they created a system of shared nothing units of processing that can be put together with linear scalability.

The last part of the puzzle was realizing that better processing means less resources. This meant the need for aggressive optimization in the query planning but also the need for a rich set of self-managed optimization techniques like indexing that are constantly in synch with the base data. And it worked!

Raised in a world of capacity

Popping forward to the late 80’s, the initial Teradata systems proved that companies could not only store all their transactions in a single system, but they could run unimagined analytics and gain great insights on how to change and improve their business. In fact, it worked so well that there was another problem, keeping up with the need to effectively run ever increasing and varying workloads in a fixed amount of capacity.

You see, back then, we did not have the luxury of “infinite resources” and new compute and storage capacity had to be bought, shipped, and installed – a process that could take weeks or months. This meant companies either had to buy for planned capacity to manage their peak seasons (thus over-configuring) or they had to buy for their normal workload (and thus be under-configured in the peaks).

This led Teradata to create what has becomes the industry leading workload management paradigm. Knowing that not all queries are equal, Teradata has worked tirelessly to have a system that is easy to define a company’s priorities and then the optimizer ensures that high priority workloads get the critical resources necessary for the job.

We make it possible to get the most out of a system by managing the workloads so that we don’t encounter the same volatility – we smooth out the peaks and valleys to ensure that we are efficient in how we meet the business needs. But is also means that companies can now get more workload accomplished without having to add more hardware resources, and therefore more costs.

For a future of unmatched ability

Returning to the present, and as mentioned before, we live in a world of abundance and all should be good. Unfortunately, in the world of “unlimited” cloud resources, it is easy to blow your budget with unpredictable demand. The reality is that companies are still having to work with planning and budgets that cannot “scale infinitely.” Companies not only need to run workloads and meet user demand at scale across the enterprise, but they need to do it with efficiency.

All this taken together is why Teradata has a proven to have the lowest overall cost per query and that is the metric that matters. Teradata brings out the ability to satisfy the demand predictably without giving the CFO headaches.

Now if only I can get my time machine to show what the future holds…. I guess we’ll have to find that out together.