NoSQL Evangelism - Are we drinking the Koolaid?

Recently I went to a workshop to learn more about Neo4j, a NoSQL graph database. I sat at table with a bunch of other techies, as we heard all about the benefits of graph databases, and how the whole world can be modeled as a set of graph relationships. They say graphs are everywhere. One of my neighbors at the table asked "Have you drunk the Koolaid yet?".

This expression got me thinking about how technology is evangelized. Technology companies need developers and systems engineers to adopt their technology in order to be successful, and they need to convince them to consider their technology, and it especially difficult when developers have already gotten comfortable with other technologies.

I unfortunately remember the origin of the "Koolaid" reference. Back in the 1970's, there were a lot of cults, and at Jonestown, Guyana, 500 people from Jim Jones People's Temple died in a mass homocide-suicide from drink poison laced Koolaid.

As gruesome as the origin is, we still use the expression today to mean blindly adopting beliefs and taking actions without sensibly acknowledging the consequences. And in the 1970's many institutions had broken down, and many people were looking for new institutions to join to feel community and structure. Unfortunately, some of these new institutions were not so benign.

Back to technology evangelism...

I believe we are in a Golden Age of information technology. The speed at which new technologies are being developed is breathtaking. But, how do tech leaders and architects who's job is to adopt the appropriate new technologies and to know which ones to pick. So, we listen to all the pitches from all the up and coming companies about how great their technologies are, and how they will make the software we are creating even better. But, in order to learn if this is all true, we have to invest a fair amount time and money to determine if such and such technology will work for us, and improve our own software. What do we do?

We trust our instincts and drink the Koolaid.

Ventures into NoSQL

Last November, my team was sitting around our conference table trying to figure out how to take our search function to the next level. We wanted to build a mini-Google to effectively search and mine our data in any way possible. We had millions of records of data covering over 100,000 companies in our sectors.

At that point, we had already started to migrate to a Service Oriented Architecture, and so we wanted the supreme search service. Also at that point all our data was stored in a MS SQL server database which had served us well for 7 years. This server was starting to slow down noticeably. So we were ready to migrate to at least a newer version of SQL server. 

Back to our meeting, one of our guys said "how does Google do it?" And I said that they have a massive index stored on a grid of servers distributed all over the world. And at that point, we realized we have to build a massive index of our data on a much smaller scale, and it seemed that SQL was not the appropriate tool. So began our dive into the world of NoSQL. 

There is a bewildering array of technologies with key-value stores, document stores, wide column stores, graph databases, as well as several indexing engines based on Apache Lucene.

 At first we looked at graph databases, since our data tracked companies involved with mergers and acquisitions where companies folded into other companies and then spun off again. In addition, people were often serial entrepreneurs or they would be hired CEOs who would prepare companies for sale. A social graph of this world seemed appropriate. However it seemed that graph databases are highly tuned for find many levels of relationships it was not so obvious how they could tackle text searching and other types of searches.

Next we had a consultant build a small application in NodeJS using MongoDB in the cloud as a data store. MongoDB has the best reputation as a document store, and I knew a few others who were using it with their projects. However, we wanted to test it out, and our aged windows infrastructure at that point could only support Couchbase. So we figured that we could at least get a proof of concept up and running on our current environment. In addition, it had a plugin to connect it to ElasticSearch which is really powerful text and data indexing engine based on Lucene. And so we started...