Archive | May, 2012

Import Genius Part 1: Early Beginnings With Roots in the Importing Process

I recently interviewed Ryan Petersen, co-founder of Import Genius and got a good walk-through of their Trade Intelligence interface.  Import Genius is one of the five (what I have labeled) “top-tier” TI providers who offer U.S. Customs data.  The other four are PIERS, Datamyne, Zepol and Panjiva. There are well over a dozen “second tier” providers strewn across the planet.  Each top-tier provider has their own particular slant and angle toward the data.  You could say that it represents the unique personality and objective that the company takes.  The data is all the same.

In the case of Import Genius, true to the personality, background and experience of the founders, they take a direct, pragmatic approach.  In effect they are the street-smart version.  Prior to establishing Import Genius, the founders were importers, getting hands-on-experience locating and vetting acceptable sources in China as well as the multifarious details involved in importing and distributing products in the U.S.

It was during their ten years as importers, in their efforts to secure reliable Chinese sources, that they ran across the U.S. Customs data.  They observed that the data could be manipulated in a much better and efficient manner and set off to make it so.  They wanted to incorporate to their interface the lessons learned and obstacles encountered in the real life learning lab.

Initially, they dreamed of creating a YELP for international trade, including customer feedback on suppliers, etc.  Unfortunately, the scope of their vision and initial launch date in the wake of the global meltdown of  ’08/’09, caused them to scale back a bit on the original plan.  Notwithstanding, they remained true to their core intent, which was to provide an easy way to do business across borders.

Ryan mentioned to me, while explaining the origins, design and unique competitive position of the Import Genius TI interface, that they really attempt to “plug into the existing processes” of their customers.  Thus, when a client initiates task “X”, the interface responds with “Y” throughout the client’s business processes.   He repeated the phrase, “plug into people’s process” a couple of times during the interview.

Essentially, Ryan and crew have created tools that they wish had been available during their importing days. Tools that could have made their job easier and more profitable.

Taking to Ryan was a déjà-vu experience.  As he shared his vision and passion for their product and the innovative, rogue, street-smart company culture they maintain, it took me back to the early days of CenTradeX almost 10 years ago.  I haven’t heard anyone speaking with such conviction and zeal for a long time.  His enthusiasm was, as they say, contagious.

Panjiva Part 4: Under the Hood Explanation of the Normalization Process

In addition to requesting the specifics on third-party data sources to which Panjiva “connects the dots”, i.e. Customs data, I wanted an under-the-hood explanation of their normalization processes.  Frankly, I was dubious of Josh’s claim to have a 70% success rate linking or resolving Manifest records to specific companies.  In response, again courtesy of their “go to” PR lady, Katelyn, they recite the following:

“Panjiva takes great pride in its patent-pending normalization procedure, which was developed by a team of MIT-trained computer scientists. Although the exact details of the algorithm cannot be shared, the fully automated process involves natural language processing, machine learning and clustering technologies. Panjiva first takes shipping-level data for all companies, then combines it with company-specific data and those from Panjiva’s full data set. Together, these paint a more thorough picture of the companies than one particular data source could do alone.

As part of this process, Panjiva keeps entities separate if they are operating at different locations.  This allows the user the option to look at a specific location of a factory or supplier. However, there are also automated and manual mechanisms that can roll these into super profiles to view related entities in aggregate.  Essentially, users can group them depending on how granular – or not – they want to get.

Because Panjiva relies on an algorithm, it is not always perfect and there are some errors due to the roughness of the data. However, the technology  analyzes enough attributes to make decisions that are robust. A few other things to note:

70% of Panjiva records are bound to entities using our proprietary algorithms for identifying when multiple shipping records are actually describing the same company/location entity. The remaining 30% of the records are opted-out, so those are not bound to companies, but these records are available in Panjiva’s Trends interface and via a raw customs search on the platform.

Panjiva does flag companies that are purely shipping companies, as many buyers are not interested in evaluating these companies as trading partners.  This is one of the few manual processes conducted by Panjiva.”

Having engineered similar technologies at CenTradeX, I can tell you that it is not easy.  I will also state, based upon the results set that I evidenced during the Panjiva demo, that they do a rather remarkable job in the normalization process.  From what I have witnessed thus far, no other TI provider, with the possible exception of PIERS, compares.

Changes in the make-up of sourcing “market share” can be viewed as well.

Panjiva Part 3: Connecting the Dots in the World of Global Sourcing

The following list of third-party data sources was graciously provided by Katelyn Henry of Version 2.0 Communications, Panjiva’s PR firm. The reason I requested and now publish this litany of information sources that Panjiva has connected to is because the normalization of the Customs data and “connecting the dots” are two of the most important aspects of transforming data into intelligence. In those regards, Panjiva has done an excellent job.


  • U.S. Department of Homeland Security provides bill of lading information for all shipments that arrive in U.S. ports.
  • U.S. Census Bureau provides trade flow statistics.


  • is the world’s leading online business directory dedicated to responsible and sustainable suppliers globally.
  • MADE-BY aims to expand the market for clothing which is manufactured in a sustainable manner by helping fashion brands clean up their production processes.
  • Social Accountability International (SAI) is a non-profit organization dedicated to improving workplaces and communities by developing and implementing social responsibility standards and assisting brands, retailers and suppliers in meeting labor and human rights objectives.
  • Worldwide Responsible Accredited Production is an independent non-profit organization dedicated to the certification of lawful, humane and ethical manufacturing throughout the world, based on 12 Production Principles focusing on compliance with local laws, workplace regulations, universal workers’ rights, the environment, customs rules and


  • CUSTOMS Info provides the most comprehensive information on companies and individuals on Denied Party Lists (DPLs) maintained by US Government
  • EDDI, Inc. is the most extensive database of known and suspected diverters, counterfeiters and their accomplices in the world, incorporating derogatory information on over 30,000 companies in the US and overseas.


  • D&B has delivered trusted business credit information for over 150 years.
  • DP Information Group (DP Info) is Singapore’s leading credit and business information bureau.
  • Experian Business Information Services partners with organizations to establish and strengthen customer relationships, enabling them to mitigate risk and improve
  • China Export & Credit Insurance Corporation (SINOSURE) With the most active and accurate information nationwide, SINOSURE’s database of Chinese companies contains business information on more than 6.5 million Chinese enterprises, and continues to expand at a rapid pace.


  • Bureau Veritas is a world leader in conformity assessment and certification services.
  • TriVista is an independent provider of Supply Chain, Quality and Risk Management services for manufacturers, importers, and exporters of commercial, industrial and consumer products.


  • Jigsaw is a leading provider of business information and data services that uniquely leverages user-generated content contributed by its global business-to-business community of 1.2 million members.


  • China Council for the Promotion of International Trade (CCPIT) established in 1952 is the largest institution for the promotion of foreign trade in China.
  • is the largest exclusively home furnishing manufacturers directory.
  • Hong Kong Trade Development Council (HKTDC) was established in 1966 as a statutory organization with a mission to create business opportunities for Hong Kong
  • Messe Frankfurt is Germany’s leading trade fair organizer, with 450 million euros in sales and more than 1,700 active employees worldwide.
  • Sourcing at MAGIC is the largest and most comprehensive fashion sourcing event in North America, representing over 700 exhibitors from more than 40
The next article will look at Panjiva’s processes of normalization the U.S. Customs data.

Panjiva has created a slick utility called “trend-spotting” which uses Trade Statistics to look for significant changes in the market

Panjiva Part 2: A TI Product with a Clever Mix of Art & Engineering

There are three very important aspects of any Trade Intelligence application: Access, Integration and Delivery.  These elements aid users in accessing, understanding and applying the trade information they need, when they need it, to make better, more informed business decisions.

Trade data is obtuse.  Hard to decipher.  Hard to make sense of.  Hard to see value in.

International Trade Data – particularly U.S. Customs Manifest data and U.S. Census statistical data – is not collected to help you, the end-user, source products, gather information on your competitors, or prospect for new clients.  By and large, it is collected by our government as a by-product of their attempts to monitor and control international trade transactions  all for purposes of national security and taxes (tariffs).

Integration: It is the job of your friendly neighborhood TI provider, like Panjiva, to take this rather obtuse data, clean it up, make it presentable and put it together with other data in a way in which you can do something useful.  In that regard, Panjiva has done some rather interesting things both in the areas of normalization (cleaning it up) and in integration or “connecting the dots”.  They interconnect handfuls of third-party databases in ways that provide essential value added additions to the fundamental or primary U.S. Customs Waterborne Import Manifest (BOL) data.

Access and Delivery: In addition to overall smarts, money, a clear business objective and latching onto an important market niche that perfectly matches the inherent strengths that are able to be unearthed from the U.S. Customs data, the technologists at Panjiva, led by co-founder James Psota and lead engineer Timothy Garnett (both mega-tech geeks with MIT Computer Science Masters) have crafted an extraordinary “tight” (excellent, cool, awesome) product.

As an artist /engineer myself, I appreciate beauty and excellence when I encounter it, including exceptionally designed and engineered technology.  An important aspect of well-crafted technology is that the components fit sleekly and efficiently together and contribute to the overall purpose and function for which they were created.  Panjiva’s interface is designed to help U.S. Manufacturers source products overseas.  The multifarious data and programming elements of their application seem to play well together and contribute that purposeful design.

As slick as the interface may be, it still comes back to data though.  In the next two articles we’ll focus in turn on Panjiva’s normalization processes and third-party data sources.

After vetting search results, you can focus on a specific potential supplier

Panjiva Part 1: A TI Platform with a Singular Business Purpose

I was surprised and impressed on many levels walking through the Panjiva Trade Intelligence Platform with founder Josh Green.

I had written an earlier piece about Panjiva a couple months ago entitled “Bridging the Continental Divides Between Buyers and Sellers?” so I knew that the key folks had come from the crème de la crème of academia – Harvard, MIT, Dartmouth- and had been successful apprehending both angel and equity investments.  So I knew they were all smart and well endowed.  Don’t you just hate that!

Word on the streets was generally positive.  A cursory review of their website, promotional material and social media updates leave a good impression.  You can tell they are on the ball and “have their stuff together” as a company.  Notably, Panjiva was/is the only top tier TI provider that focuses on one singular market segment and business application, i.e. sourcing.

PIERS has a handful of products: Prospects & StatsPlus (which they acquired from CenTradeX last year) as well as MyPiers, iPiers, Piers TI and Piers Trade Finance which tend to be geared toward particular market strata.  Prospects, as the name implies, is primarily a prospecting tool.  Datamyne, Zepol and Import Genius sell all-purpose applications, trying the “one size fits all” approach.

Panjiva focuses on sourcing, period.  Their website is designed around telling that story.  Two prominent navigational tabs point prospects to either “For Buyers” (those sourcing) or “For Sellers” (those who want to be sourced).  Their pitch to buyers (importers) is elaborate and convincing.

In accordance with Panjiva’s business objective and market niche in the sourcing world, they have apprehended over a dozen third party data sources that they connect (with various degrees of success) to the daily U.S. Customs data feeds.

I admit, after going through their interface and digging into the details behind the scenes, I have become convinced that Panjiva has- by far and away- the very best trade intelligence product on the market today for certain types of U.S. importers (those who are sourcing products and components being shipped via waterborne transport from factories overseas, i.e. non NAFTA countries).  Obviously, that was their clear and defined business objective and that is what they have accomplished, for which they deserve kudos, and because of which they will most certainly garner increasing market share.

In several subsequent articles, I’ll “go under the hood” to examine the particulars that developed the above perspective.

Sample of Panjiva Search Screen. It all starts with the right question.

Zepol, Part 4: Setting the Standard for Standardization & Success

In the previous three Zepol articles, I have extolled the incredible speed of their search engine as well as praised their elegant, well-designed user interface.  In addition, they seem to have consistently invested in product development and infrastructure.  Whereas they were initially the newcomer and rouge to the trade intelligence field, they have established themselves as elder statesmen.

Notwithstanding, in my opinion, there are two major areas of improvement needed, not only to withstand the onslaught of new competitors arriving monthly but also to gain market share among the other four top-tier TI providers.  Namely, better standardization of the manifest data itself as well as integration of other pertinent data sources.  These enhancements are not optional they are mandatory.

Presently, within the blindingly fast search results there are many iterations for the same shipper (foreign supplier) and consignee (U.S. importer).  In addition, NVOCCs and other transportation providers frequently show up as either the buyers (importers) or the sellers (shippers).  Hence, some potential business applications are askew:  validation of prospective foreign sources based upon shipment count, competitive analysis of U.S. importers, and trend analysis of shipments based upon any designated criteria.  These are distorted by the lack of rigorous standardization of the underlying data.

This issue is not limited to Zepol by any means.  Every Trade Intelligence Provider struggles with it.  Some have developed better methods than others. Only two companies out there do it with any measure of success. Both are utilizing third-party company databases plus advanced algorithms to refine the data.  Most don’t employ any advanced standardization procedures at all.  At CenTradeX, we developed arguably the best parsing, standardization and integration processes out there, but it took many years and hundreds of thousands of dollars to accomplish this.  It was an obsession for a decade.  It’s not easy.

In the other area of improvement, integration of other pertinent data sources, Zepol is clearly heading in the right direction with TradeView.  Contrary to their initially held position to only serve up U.S. Customs data, Zepol has added U.S. Census (Statistical) data.  What is especially notable is that they utilize virtually the same user interface for TradeView as TradeIQ.  It’s a fairly seamless and painless transition from searching and reporting on manifest data to doing the same with statistical data. This is no small accomplishment since the two databases are vastly different.

The real key and subsequent challenge will be to connect these two disparate silos of data to one another in significant ways that provide additional dimensionality and richness.  There could also be some interesting results from marrying their new offering, Compliance Monitor with TradeView, but at this point, a potential relationship seems fuzzy at present.

I’ll end this series by recounting my comment about TradeIQ.  Did I mention how freakin’ fast the darn thing is?  I really can’t get over it.  Again, if you want an idea of just how fast, check out the videos on Part 1 of this series, Zepol, Part 1: Fast, Faster, Fastest… Freakin’ Crazy Fast Search Engine. If you are interested in discovering the features and respective pricing of each of their subscription options, feel free to download this Spec Sheet. maintains an extensive video library wherein a dozen of Zepol’s instructional videos can be found. Zepol also maintains one of the better blogs out there.

Trade View results screen shows how U.S. Census statistical data can be used to identify potential source countries for a particular product. 

Census Statistical Data can also be used to spot sourcing trends for products and components.

Zepol, Part 3: Trade IQ – An Elegant, Well Designed User Interface

TradeIQ is Zepol’s original, primary and best-known product.  It’s the searching and reporting UI atop the U.S. Customs data.  TradeView uses the very same utility, and has the same look and feel, except that it sits atop U.S. Census (statistical) data.  Compliance Monitor, their newest offering, is simply email updates on specified Harmonized Codes of all pertinent changes.  There is no UI attached.

It is obvious that TradeIQ has seen a lot of improvement over their initial product launched in 2004.  Besides being incredibly fast, upon which attribute I expounded upon greatly in this initial series, Zepol, Part 1: Fast, Faster, Fastest… Freakin’ Crazy Fast Search Engine, TradeIQ has an elegant, well-designed user interface.  Like, Paul himself, it’s very straightforward, clear and concise.  The logic is easy to understand and utilize.  As far as utility to search and fetch manifest records is concerned, it is my personal favorite.

The search array spanning the top of the UI is neatly organized. Each search field offers users the opportunity to progressively drill down to the specificity they require.  For instance, if I wanted to look up shipments of China manufactured mopeds, shipped via Shanghai to Charleston to consignees in Tennessee, it might look like this:

  • Product: Scooters, 50cc, Scooter, Motorcycle
  • Shipper: World (all) > Asia > China >
  • Consignee > North America > Southeast > Tennessee
  • Intl. Port> World (all)> Asia > China > Shanghai
  • U.S. Port> Southeast > South Carolina > Charleston

If I wanted to broaden the search, let’s say to include all shipments disembarking China headed for Tennessee via any Southeastern port, all I need to do is make 2 additional clicks. The first click back up the “Intl. Port” search tree (China) and the second click two steps back on the U.S. Port (Southeast).  In this method, users easily refine and change the scope of their search.  The logic has remained the same since Zepol first launched their product.  The only difference is that now there are cool buttons instead of links.

The overall layout, user interactivity, organization of search results and graphic displays have much improved.  Conveniently arranged tabs and buttons provide users easy access to various views of the retrieved data. As would be expected, searches (and corresponding results) can be saved and exported (Excel or PDF).  In addition, users can schedule updated reports to be generated and emailed as desired.  Four selected reports serve as the default view for a respective user’s Dashboard.

Sample refined illustrating Shipments of mopeds into the U.S. by various Chinese suppliers between 2003 through 2011.

TradeIQ Profile Report conveniently summarizes search results in “top five” U.S. consignees, foreign shippers, Ports, Carriers, etc.

Zepol, Part 2: Company Background, Evolution & Competitive Position

Zepol was the first company, other than PIERS of course, to offer a commercial searching/reporting utility to atop the U.S. Customs Waterborne Import (BOL) data.  Their voluminous database of 100,000,000 records extends back to January 1, 2003.  With only a smattering of sales in 2004 and 2005, Zepol really began to emerge as a player in 2006.

I remember meeting with Paul in Minneapolis early on.  He struck me as a solid businessman.  His style was clear, concise and savvy.  We explored cost sharing the daily expense of the daily Customs CD’s. On the surface, it seemed plausible.

Our business plans and prospective clients didn’t overlap much.  At CenTradeX our approach was developing custom integrated solutions (utilizing multiple statistical, company, transactional and referential databases) and Zepol’s straightforward business plan was to exclusively offer the U.S. Customs data with a superior search/reporting utility at 20% discount over PIERS base product (which at the time ran around $5,000 to $6,000 annually).  Their singular data product was/is called “TradeIQ”.

I also remember checking in with my business associates at PIERS.  At the time, they really didn’t think Zepol represented any competition whatsoever.  PIERS maintained a comfortable monopoly that they believed was unchallengeable.  My, how things have changed in just 5 years! Five years represents a whole generation, technologically speaking.

We (CenTradeX) never did the deal with them.  We couldn’t agree on the particulars of reselling processed data, and we already had a data partner in China.  I think the new management at PIERS closely monitors the comings and goings of all their competitors. Zepol, recanted on their position to exclusively offer U.S. Customs data.  Their “TradeView” product now offers their same user interface to search and report on monthly U.S. Statistical data (with data extending back into 2007).

Whereas Zepol first competed on price alone (and a slicker search UI). With the emergence of Datamyne, Panjiva, Import Genius along with dozens of ultra cheap competitors coming on the scene, they have been forced to evolve into a different company.  Paul emphasized that they have heavily invested in infrastructure and improving their product as well as in customer service in an attempt to bring additional value to the equation.

Most TI providers see that there is nothing to be gained by a “race to the bottom” wherein products compete on price alone.  It’s about bringing true value to the end-user.  It’s about creating solutions, not selling data.  It’s about looking farther and adapting faster than your competitors.

Zepol’s Home Screen. Entry points to Products TradeIQ and TradeView.

Zepol’s very cool Dashboard providing users four “big picture” graphic overviews of saved searches. Fully customizable by the respective user. Click on the picture to view a larger version.

Zepol, Part 1: Fast, Faster, Fastest… Freakin’ Crazy Fast Search Engine

In this first of four articles on Zepol, I will dispense of the usual company background, forego superfluous narrative that indirectly ties my extensive knowledge and experience in Trade Intelligence to the subject as well as and any other attempt to subtlety impress and go straight to today’s main point.   Zepol’s TI interface is fast.  It is really, really fast.  It is incredibly, spectacularly, spine tingly fast.

As is my M.O., during the product demo provided by Zepol President Paul Rasmussen, we dispensed with the canned presentation and went off the beaten path to look up Scooters with foreign suppliers and corresponding U.S. importers of under 50cc moped style motorcycles.  The default search scope is set to hunting within the last 30 days of Manifest records.  In a second or two we got the results.  O.K., not too bad, pretty fast.  However, I wanted to see how the search engine would perform when really put to the test.

I asked Paul to extend the search from the very first manifest record, back in January 2003 to the very last record in September 2011… that’s around 100,000,000 bills of lading!  Keep in mind we’re searching through multiple textual fields (“Products” and “Marks & Numbers”) for each BOL using text terms: “Moped”, “scooter” and “50cc”.  We’re not talking about numerical fields with a singular numerical criterion.

BAM!  3-4 SECONDS later we’ve privy to over 6,000 shipments – with corresponding detail if needed – on the representative international trade transactions involving imported mopeds.  WOW.

I refrained from inquiring about the specific alchemy that created the magic.  What combination of server arrays, multi-core processors, RAM, query optimization and full text indexing was employed to do this? Paul credits co-founder Jeff Wilson and his tech team.

A couple of illustrations come to mind that may communicate this amazing data feat better.

I went with my eldest son to the Brickyard 400.  I remember pressing our faces next to the racetrack fence as scores of NASCAR speedsters zipped by at 200 mph.  The visceral memory of sheer speed and power is unforgettable.

In the final scene of the movie Secretariat, the remarkable story of the 1973 triple crown winning race horse, said underdog (rather under-horse) soundly beats the favored “alpha dog” (alpha-horse) by an astounding 31 lengths (still holds the record for speed and margin of victory).

Other notable examples could include Bruce Lee’s unmatched speed performing various martial arts maneuvers, Superman’s counter clockwise planetary orbits to save Lois Lane, and the Enterprise when it hits the warp speed button.

Now maybe you get the picture of just how fast Zepol’s search engine goes.  Zoom, Zoom!

Datamyne, Part 4: A Trade Analyst’s Dream Machine & WTD Editorial Note

Last year we ran several articles on the TI Provider Datamyne, Lisa Wallerstein the VP Product Development /Marketing and the announcement of their new interface, Datamyne 2.0.  In this last week’s series of articles, I focused on the specifics of their TI product following an in-depth demo and review.

I must admit that recently, while making my rounds with each top-tier TI provider and getting a deeper look at their products, my viewpoint is changing.

What I am discovering is that there is no one best product and TI provider out there.  Each has its own particular approach, technologies, interface, data sources, value added services, and pricing model that make it the best solution for certain niche or target markets.

After reviewing Datamyne’s interface, I can confidently state that it represents perhaps the best analytical user interface by which to “slice and dice” the U.S. Customs data that I have seen thus far.  It’s a trade analyst’s dream machine.

If you’re looking for a robust TI product that is straightforward and relatively easy to use by which to perform heavy-duty analytics on the U.S. Customs Data, you definitely need to check out Datamyne.

WTD Editorial Note: I’d also like to address an important issue that has come up on several occasions.  I have been accused by some of being overly biased toward PIERS.  In a number of articles and in handfuls of instances it has been noted that I have stated that PIERS products are the best.  By and large, I think the accusation has been well founded.

To be honest, one of the reasons for my bias is that the TI products we developed at CenTradeX (IMHO) were vastly superior (in aspects of innovation, UI access, data integration, graphic delivery and performance) in comparison to any other TI product. PIERS acquired these assets last year.  Furthermore, the team of superb technologists I worked closely with for many years also migrated to PIERS.  Therefore my high esteem has been transferred or credited to PIERS posthumously, so to speak.

Secondly though, having dealt closely with the new management team at UBM Global Trade /PIERS for many months during the due diligence process and afterwards (being somewhat privy to their “thought engineering” and witnessing the changes they are implementing) I have gained tremendous respect for them.  Further, they have several proprietary databases that are “untouchable” by competitors.  They have roots in history and experience that go back over 100 years.  They have worked with Customs data along with the nuances of refinement and standardization for many decades.

So, with regards to my previously perceived biases, I hereby recant my previous position that there is ONE best company and ONE consummate Trade Intelligence platform.  Vive la différence!

Datamyne, Part 3: Micro to Macro and Back in 60 Seconds or Less

One of the coolest things about Datamyne’s data mining interface is the drag and drop feature for adding or subtracting fields of data to the display and export processes.  Similar to creating and viewing Pivot tables in Excel, the UI provides users the ability to dimensionalize their perspective of the data and flip it one way then another.  Again, perfect for analysts who require more complex manipulation of the data.

My personal preference when working with or presenting trade data is first of all to get a big picture view, then logically drill down to finer detail and greater specificity.  This ability is absent in most TI products on the market.  Datamyne’s interface provides that functionality superbly.  It allows you to get both the macro view and the micro view as well.

One of the biggest difficulties I labored with in developing trade intelligence platforms is the tension between making data easy to access and to understand on the one side and providing the detail and specificity on the other.  Although Datamyne’s data mining utility is well crafted, it is still rather complex and fairly intimidating at first glance.  It’s about as easy as something complex can be.  Heck, all the buttons and control options on most video games confuse me.

Another important feature is that once users create and perfect their queries, they can save and retrieve them for future use.  In addition, they can set-up alerts that will automatically email them when certain criteria are triggered, such as when new shipments occur or new suppliers or competitors for their designated product enter the scene.

Not surprisingly, Datamyne’s customers are larger companies with complex supply chains who need to perform custom analyses.  To accommodate this market niche, Datamyne provides more extensive customer support and training.  They are the full service solution in comparison with many of the self-service e-commerce type conveyers of U.S. Customs data that have recently proliferated the marketplace.

Other competitive strengths that were mentioned to me include:

  • The best overall value with prices starting as low as $199 per month for access to the raw unfiltered, non standardized U.S. Customs manifest data.
  • Latin American Coverage:  Datamyne has its roots, data center and most employees south of the border (Uruguay). They offer transactional data for many South American and Central American countries.
  • Statistical data gathered and disseminated for almost 50 countries, updated monthly.
  • Versatile and (relatively) easy to use user interface.

All in all, it represents a very good data mining utility for trade analysts to employ to slice and dice and serve up U.S. Customs data.

Datamyne Data Mining Utility provides capacity to quickly get the big picture as well as drill down on a specific Bill of Lading

Datamyne’s way too cool Drag and Drop feature that allows trade analysts to create pivot table like views of the data

Datamyne, Part 2: Queries: the Tiny SQL Fairies that Fetch Data

The front end or UI (user interface) that sits atop of Datamyne data has the look and feel of an analytical tool.  Even the naming conventions bear witness to their particular approach and designated audience.  All three initial windows into the soul of the machine have the label “queries” attached.

Users begin their search by either engaging with “Rankings & Queries”, “My Saved Queries” or “My Most Recent Queries”.  Obviously, first timers’ starting point is the first item.  It all begins fairly simply by selecting a country (market), database (transactional, statistical) and a year. Then, you choose between “rankings” and “queries”.  It’s all pretty straight forward thus far.

If you select “rankings”, you’ll get back an overview of the data arranged by commodity or geography.  If you select “queries” you’ve just bought the “E Ticket” to Data Disneyland or perhaps more like entering a geek construct of Data Lego Land.

The individual building “blocks” are composed of chunks of U.S. Customs manifest data –foreign shipper, U.S. importer, addresses, TEUs, commodity, product description, ports, weight, etc — several dozen fields of information for each individual bill of lading (shipment) multiplied by millions of BOLs each year.  That’s a lot of blocks.

Queries are formal inquiries constructed in SQL programming language posed by users in order to properly extract, filter and display these data blocks in ways that will solve a problem or answer a question.  Good questions get good answers.  Datamyne’s UI allows users to easily engage in the process of creating, editing and revising their query.

Proper question or query formation is more an art than a science as anyone who has tried to find anything via Google or another search engine can attest.  You have to refine and re-refine your search terms in order to filter out unwanted junk and get to the treasures buried in the dirt.  That’s the mining of data mining.

One of the excellent features about Datamyne’s data mining utility is that all the manipulation and revision happens on one page.  You don’t have to jump from here to there or follow links to a different page or start all over again if you want to alter your search criteria.

I was impressed by the relative speed by which the query results were returned.  Perhaps not lightning (instantaneous –POW) speed but certainly thunderous (one thousand one, one thousand two, one thousand three) speed.  That’s a lot of data to sift through, arrange and display.

DataMyne Home Screen. A Query Smorgasbord. Create a New Query. Fetch Saved Queries. Retrieve Recent Queries.

After Sending out their Query Fairy, Users can dynamically interact with it to refine search results

Datamyne, Part 1: Data Standardization and Product Code Attribution

As I have said many times, Trade Intelligence is composed of many facets: data, technologies, application and people.  It’s people that really boost the IQ.  Notwithstanding, on some level it starts and ends with the data.  If you’ve got bad or ugly data to start with… whatever you build ain’t gonna look pretty.  No decision will be well founded.

It is reported that Datamyne’s standardization process of the U.S. Customs data results in linking 85% of the manifest records (that are not suppressed or invisible) to U.S. consignees (importers).  Believe me, that’s no simple task.

As part of the deal Datamyne can filter out NVOCCs and logistics folks. There is nothing more irritating when you’re doing a search through Customs records, looking for importers of a particular product let’s say, than to have your results display 10 iterations of the same importer along with “junk” or fake consignees, which are obviously not importers.

Future iterations of their product promise to link expanded D&B company information to U.S. importers, further enhancing the value.  They believe they will be able to match about 60% of the non-suppressed records to the D&B database.  It’s all about “connecting the dots”!

U.S. Customs data is fetched daily via FTP.  Datamyne processes and makes a “raw” version (without standardization of names or product coding) available online within 72 hours.  They’re intending to compress that time down to a 24-hour turn-around very soon.  Their standardized, coded version is updated twice per month around three weeks in arrears.

More about data.  Datamyne also attributes a Harmonized Code product classification to over 80% of the manifest documents.  Although usually only to the two or four digit hierarchy, this is a fairly remarkable and difficult undertaking.  With the exception of PIERS, no other TI provider does it.  Although they’ve yet to fully exploit the many benefits of these linkages and the many profound synergies that can be created between statistical, and company data sources, they are clearly laying a firm foundation.

Although cloistered in separate silos, Datamyne maintains statistical data (updated monthly) on almost 50 countries within Europe, Asia and Latin America.

Furthermore, they maintain transactional manifest shipping data (similar to U.S. Customs data) on many Central and South American countries.  Unfortunately for Anglos, this data is only available in Spanish.  However, the availability and expertise with Latin American data remains one of their unique competitive strengths.  Datamyne’s origins, data center and most employees are rooted in Uruguay.

Datamyne has a slick “Drag and Drop” feature that allows users to chose which data elements they want to search on, display and export.