Managing Data Overload in Business Intelligence

The US has over 2,700 data centers. That’s enough for us to allocate 54 of them per state. Obviously, given the HVAC requirements of these facilities, there’s not 54 per state. They’re rather concentrated in certain areas of the country, but you get the idea. We live in a world of increasing amounts of data.

Large data is really simply just a consequence of human fear. As a species, we deal poorly with uncertainty. The illusion of more data gives us some comfort in knowing we may be able to predict what future events lie ahead. Thus the thirst for more data.

While we can actually do some predictive modeling based off the data we acquire, not all of the data we collect is actually useful. This is why it is difficult for managers and business leaders to create successful models. They have so much data to choose from and few are certain of which data sets are truly significant.

When computational power was limited due to hardware technology, business leaders had to be deliberate about which data they wanted collected and analyzed. As computational hardware became more powerful and affordable, the temptation to collect more and more data could not be escaped.

That is how we landed here. The situation in which we find ourselves today.

C-Suite executives ask for reports predicated on data and metrics they don’t have a full grasp of. Managers and employees generate reports with data which has no real statistical significance for their organizations.

“Tableau software looks so beautiful on the screen. Thus, the entire dashboard must be filled with data.” - Brittney

To compound the data problem the issue of job rotation and role replacement factors into the mix. A previous manager found these “metrics important” hence you should as well! Even though that exact phrasing may not be used, it is always implied.

Due to this over-saturation of data, corporations are attempting to implement too many new initiatives at the same time. The thought being “If the data is there, then surely we can go ahead and execute upon it.” However, such thinking usually proves to be checked by reality.

So where should we go from here?

The first place to begin would be at data engineering and project implementation.

When given the chance to collect data from a new source, we are often given the choice of which data we actually want to collect. For the past 30 years, our answer has been all of it. This is wrong and is the first thing that needs to change.

Every business and corporation is different. Their data needs should be different as well. What is important to one business may not be important to another.

Education should be considered as a next step.

Most of the working employees in corporate America have a poor understanding of mathematics and statistical methods. These same people are the ones making decisions about what data is important and what metrics should be used to judge their departments, yet they are completely lacking the skills to do so. Not everyone is going to be a math major, but at least those who are on the manager level need to have a comprehensive understanding of what statistical significance truly is.

My final recommendation would be data blank exercises.

This would involve a corporation taking an entire department and stripping it of all its data resources for an entire month. Why would you want to do this?

It resets the collective brain of the department. Without the data sources being available to them, they’re going to have to think critically about what data they would actually want to be available to them and how they would utilize the data if they had it. New metrics will be created. New KPIs will be formed. This exercise is extremely helpful and should be utilized in every department at least once a year. The world is changing by the minute, yet that report from 1997 still has to be shown by an email to everyone in the department.

It really comes down to just two things, critical thinking and understanding risk. The corporations and businesses out there who grasp this base concept will survive. AI is killing critical thinking in today’s employee and student. Our modern economy and government has done their level best to strip risk away from large corporations. However, the scales will always balance themselves out.

Jonathan Adams

July 30th, 2025