Winter Solstice Taxonomy

A friend of mine has his own business, and he recently had an interesting problem. He sent out to his client list a generic holiday email which did not mention Christmas or any specific holiday. One of the people who got the email replied very negatively about how Christmas is never mentioned specifically by name, and was very upset. My friend happens to have been raised by parents who come from two different religious traditions, and so he not surprisingly wants to wish everyone well. So how can his dilemma be solved by Taxonomy? As I mentioned earlier our team uses multiple dimensions to classify businesses. He could create a taxonomy of holidays. For instance, you could have a category for the midwinter holiday, and you could code each instance to given religious audience: Christian, Jewish, Muslim, Hindu, Pagan, Agnostic, Atheist, etc. which would give us imagery and words for Christmas, Hanukkah, Eid, Diwali, Saturnalia, Winter Solstice, etc. You could then continue this for the major fall, spring and summer holidays. My son's school, interestingly enough, only celebrates the equinoxes and solstices.

In all seriousness, I wish you all a Happy New Year!

Challenges of Classifying a Business

One of the biggest challenges our team has building a taxonomy of businesses is the actual classifying of a business. The process at best is semi-automated. Our classifiers generally have to look at the website of a given business and try to determine what they do. The website, generally speaking, is not written to describe how a business operates, but rather to sell the business's products and services. Each website has its own content, and we used to concentrate on the "About Us" page where the business defines itself. Unfortunately, the "About Us" page is usually some sort of vague mission statement. We even looked at our own website, and found the wording written by our Marketing guy was so general that you would not know what we did! So our classifiers have learned to scan around many pages of a businesses site to learn what they do, how they do it and who they sell to. We have built some scrapers, but our results have been mediocre. We are now currently looking at ways to scrape company websites and intelligently gather info data to further automate the classification. The big problem we saw was that matching words on the website against words in our thesaurus gave us too many false positives. We are now looking at weighting the value of words in our thesaurus and how they matched in the past with verified classifications. Anyone explore these types of auto-classification.

Classes versus Concepts Taxonomies

As I also post these entries on LinkedIn as well as other social media, sometimes a discussion will take off on one of the social media sites, and is not shown here on this blog. One of the most interesting responses to the True relationship between parent and childpost was on LinkedIn and covered what is the nature of nodes in one's taxonomy. Does each node represent a class or a concept? Denise B. of Kent State University raised this thought, and said that often one needs two taxonomies one where the nodes are classes and another where the nodes are concepts. The nature of your taxonomy will then define the nature of the parent-child relationship. In a class based taxonomy, the child is a subclass of the parent where the child has all the kinds of attributes of the parent and then has some special attributes specific to the child class. With a concept based taxonomy, we would then see each node embodies a concept and hence the relationship to a child node is not so strict. Thanks to Denise on the response.

True relationship between parent and child

I recently had to make a change to our taxonomy search. Our previous search would match a selected node and all its children. However, we found a need to search and just match a node and not any of its child nodes. I have seen this feature in many other systems, but we never found the need to implement it in our system till recently. This highlights a deeper question of what do we expect the relationship between parent and child nodes. When we first developed our taxonomy, we had long heated discussions on this issue for our industry tree. Let's take an example of a well known software company, Oracle. Under our software category we have operating systems, business applications, desktop applications, database systems, email systems, graphics applications, etc. Oracle is a big software company that makes all kinds of software, but does not create all the types of software under our software category, but they do most. One of our team members suggested that if a company is categorized as something then it must do all the subcategories. His suggestion was to categorize multiple times for each type of product it does exactly. However, it starts get ugly when a company like Oracle is operating in most of the subcategories but not all. Our general consensus was to categorize a company in the parent category if it covers a good portion of the subcategories, but not all. If the company covers a few of the subcategories, then we will have multiple categorizations. The real key is consistency. We take the approach (and near cliche) that the parent is something different than the sum of its children.

Banyan provides an interesting view on our data.

The folks at Ongig.com, a new video enhanced job site, asked for some of our data to see who are the top 25 most efficient Tech companies based on profit per employee. The winner is SanDisk followed by Google and Apple.

http://ongig.com/blog/wall-street/profit-per-employee-tech#more-1399

We were able to easily provide this because our chief taxonomist had created a top level node in our industry tree called Information Technology which has child nodes for Computer Manufacturers, Software Companies, Electronic Information, and IT Consultants. All these nodes have other parents elsewhere in the tree, but the Banyan helps pull it all together.

I would also like to add a comment by Leo Meerman from LinkedIn. He mentioned that what I call a "Banyan Tree" is better known as a "Polyhierarchy".

The Banyan Tree - a new hierarchy

Most taxonomies are set up in a hierarchical tree format. Our team consists of four distinct trees for each facet of how a business operates. Developing the taxonomy and software over the last 8 years, we found at some point that the tree structure became too strict. There were certain categories that did not want to be under just one parent category. A prime example of this situation is video game companies. These companies make software for entertainment purposes. As this industry has matured, it has become closely link with the big entertainment companies, and they employ teams of artists, writers as well as programmers. Historically, these businesses should be software, but they are also so tied closely to entertainment companies it seems odd that when searching for entertainment companies that these would not turn up in our search results. One solution is to move video game studios to under the entertainment category, but then we lose the software aspect of the business. Our team's solution was to change the nature of our trees. We now allow nodes to have multiple parents, and create what I call the "Banyan" tree, which is a tree from India that has multiple trunks to the ground. 

Looking at the Wikipedia article, we see that the banyan tree name comes from the Gujarti word for merchant, because merchant markets were often located under these great trees. It seems appropriate for a business taxonomy.

Challenges of Searching a Complex Taxonomy

As I have noted, our team has developed a very complex multi-faceted taxonomy which give a very exact categorization of any given business. The complexity we have created comes with a price. It creates challenges for users searching for businesses. A company is categorized by who their clientele is, what business needs they fulfill and by how they fill those needs, and what channels they use. A typical user does not think of businesses in such a manner. For instance, a typical search would be I would like all businesses that sell education software. In early versions of our application, users would have to savvy enough to enter in the search criteria education for the clientele and to enter software for industry (the "how a business operates"). Needless to say, users had to be trained to use the system, and hence the system was only used by our in house taxonomist who would get search requests and he would then give them results in a handy report. This was not ideal, because our taxonomist has many other things to do. The challenge for us was to make our taxonomy search as easy as Google. Nobody has to be trained to use Google, and we found that if training was involved generally that part of our application was not used unless the payback was great. In addition, since we were already fielding queries, our users found it much easier to send an email to our team, rather than run the search on their own. So how did we simplify our search. Well, in an earlier post, I mentioned we found that were phrases that users wanted in our taxonomy that would not fit in a single tree or dimension of our taxonomy. I called these phrases, "Meta-Terms". The example I gave was Trade Magazines which are B2B magazines. In our taxonomy we then map this term to magazines in our Industry tree and B2B in our Clientele tree. So the key to simplifying our search was to use "Meta-Terms" as a model for Google like search, Users can now type into a text box and it will see if it matches an existing "Meta-Term" or what I call an implicit "Meta-Term". Implicit "Meta-Terms" are created  by synthesizing synonyms from our different trees' vocabularies. There are over a million possible combinations from the trees, but some of the combinations are unlikely to exist like (e-discovery software for teenagers). So we have created a list of synthesized terms from the companies we have already categorized (numbers around 23,000). From these categorized companies and all the synonyms, we get a list of about 200,000 implicit meta-terms. Our "Google" search box then matches the user input to our list of terms and runs a query on how that meta-term maps to our multiple dimensions. This is still in testing, but shows tremendous promise.