In the age of big data, it’s crucial that the disparate systems that interpret that data are able to do so accurately. Even the slightest mistake or inconsistency could have catastrophic impacts further down the line.
The more data that a company has to analyze, the more likely it is that there will be anomalies within that data that will negatively impact accurate calculations.
What is Standardization and Normalization
When dealing with large data sets, often times there are simple inconsistencies that a human eye could pick up on, but machines with rigid rules will overlook.
For example, one system might spell “Walmart” as “Wal-Mart” or “Wal Mart” – any rational person looking at these names would instantly recognize that they are the same, but computer programs often cannot “think” in that way.
This is just one of the hundreds, perhaps thousands, of similar instances which have plagued shippers.
Enter data normalization and standardization – the process in which data that may be listed in slightly different ways between two systems is normalized into a single data point and, as a result, able to be analyzed effectively.
Why Data Normalization is Necessary for Shippers
Before a shipper can optimize their shipping, they must first examine and analyze their current practices. Without normalized data, this kind of analysis is impossible.
Normalized data will allow shippers to answer the following questions, which are crucial to the optimization process:
- What is my Total Spend by Mode and by Volume?
- Where am I shipping to?
- Which shipments are coming inbound?
On the parcel side:
- What’s my Average Zone?
- What’s my Service Utilization?
On the bulk side:
- What is my Cost per Mile?
- What is my Cost per Unit?
- What is my Cost by Lane?
- What is my Volume by Lane?
While these might seem like complex and time-consuming data points to identify, a robust automation tool can help – but that tool must first be able to standardize and normalize data.
Examples of Normalized & Standardized Data
In order to generate clean data to answer the questions mentioned above, it’s necessary to get normalized data-points for important information, such as:
Carriers have different names for the same charges. For example, one carrier might use the code “freight” while another refers to the same service as “line haul” or by a simple number code such as “400”; all meaning the same thing.
A cynic might say that carriers do this on purpose so that shippers won’t be able to compare them against each other. However, it is this exact comparison that is needed in order to optimize and find the greatest cost savings for shippers.
Cleaning up the data will ultimately allow shippers to look for inefficiency’s and increase value.
Shipper Name & Company/Reciever Name & Company
Company names can often get misspelled or written incorrectly, impacting the quality of data. The Walmart example mentioned earlier is the quintessential case for this.
If the name and company data is not normalized, shippers’ vendor scorecards will be “noisy,” making it impossible for shippers to get an accurate view of the data.
They won’t know how much it currently costs to ship to Walmart and you don’t know how many shipments your shipping to Walmart or what the quantity or what their skews are because again its all noisy.
Origin & Destination Pairing
As with company names, cities can often be referred to by a variety of names – all of which are valid. If you are looking at three different combinations for one lane, calculating the cost per route will be impossible.
And, oftentimes, this ends up being the case.
In order to correctly calculate cost per route, these names need to be truly normalized.
Intelligent Audit provides its clients with a global, all-mode transportation audit, recovery, freight payment, and business intelligence reporting partner.