In the past we published a series of 5 articles; “U.S. Customs Data Primer”, Parts 1-5, about the particulars of understanding, processing and enhancing the daily transactional inbound shipping records published by DHS/Customs. This article will expand upon the fourth article in that series, “U.S. Customs Data Primer Part 4: Enlightenment Through Graphics & Diagrams” which provides a visual guide for the processes we at CenTradeX employed in transforming raw data into trade intelligence.
The 90 second video (slide show) below portrays the original Trade Intelligence vision and mission that fueled the innovative growth and development of CenTradeX.
Developing innovative and powerful trade intelligence applications involves attending to three major areas: Access, Integration and Delivery.
- Access: Helping the target audience(s) understand, find and use the data and application.
- Integration: Normalizing of base data and connecting it with other relevant data sources.
- Delivery: Enhancing the speed, efficacy and beauty with which the combined data is organized and presented.
For the purposes of this series, we will only focus on the second aspect, Integration. Furthermore, having provided a foundation of understanding through the above referenced (linked) article, we will proceed to explore the more technical (under the hood) facets involved.
I refer to U.S. Customs waterborne import manifest data as the “base” data because it is considered (by myself and many others) the most intrinsically valuable, if challenging, international trade data set available. It’s daily. It’s transactional. The U.S. is considered the easiest market to access. It contains a wealth of detailed information about the global supply chain. It represents $1 trillion dollars of trade a year.
U.S. Customs data is #1. It’s THE KING of the international trade jungle. However, a powerful Kingdom is more than just one regal personage. It must include a capable entourage as well. Thus the need for complimentary data sets.
The first, primary step in building a powerful trade intelligence “kingdom” is attending to the King. PIERS has an easy to understand graphic portraying the processes of normalizing “base” Customs data layer.
In performing the seven steps highlighted above, we at CenTradeX developed and refined many sophisticated procedures. Over time, and through much scrutiny and evolution, we constructed a reliable, interconnected system of transforming data into intelligence.
- It involved an array of automated queries and stored procedures for importing new data on a regular basis.
- It involved created programs and “scripts” that would parse, tokenize and reference selected data elements, compare and contrast them with its expanding library and referential databases as well as “learn” better ways of matching and connecting.
- It involved scouring the planet for the best, most reliable, accurate and timely ancillary databases to enhance and expand the KING and the Kingdom.
I thought your post was well thought out and nicely developed, great work! =)
These stories get me excited about what is possible. thanks =)
Great post, I concur completely and appreciate the time you took to write it, nice!
You produced some decent points there. I looked on the internet for any problem and found most people may go along with with your site.