I woke up this morning in San Francisco to news that SiSense’s financing round is now public – and so now I can offer a hearty public congratulations to the amazing team at SiSense (Elad Israeli, Eldad Farkash, and their outstanding dev team) as well as to Gary Gannot of Genesis and Dan Avida of Opus who together led this investment.
SiSense was the last project I worked on at Genesis Partners, and I was able to see the investment close just before joining Gemini Israel Funds earlier this month. It’s a company I have been tracking for over three years, and one that I believe in greatly. Let me share some of my enthusiasm here.
BI is a four-letter word. When I started tracking SiSense in 2006, Business Intelligence (BI) was a four letter word. No one in the VC community was taking any interest in a small team of engineers working in an office park somewhere in Netanya. Enterprises had spent billions of dollars on complicated BI implementations and had little to show for it. BI systems were extremely expensive and cumbersome – and usage was very low. Aside from a handful of critical reports on KPIs, most business would report that their business users were rarely if ever making use of the sophisticated querying and reporting functions offers by BI packages such as Cognos (acquired by IBM in 2009), Business Objects (acquired by SAP in 2007), or Hyperion (acquired by Oracle in 2007). BI was a critical part of the enterprise stack – after all, reporting on business data needed to come from somewhere – but BI was widely considered a failure and – aside from the critical reporting functions – any incremental BI spend was generally considered wasteful.
The reasons for the failure of traditional BI lie in the database architecture itself – something that I came to understand through hours of conversation with Eldad and Elad and many discussions with BI professionals in Israel and the US. BI systems were originally modeled on their cousins – the operational/transactional databases without which there would be no data to analyze in the first place. Operational/transactional database were designed to facilitate rapid writing and reading of “rows” in the database. When a customer purchases a product, the details of that transaction need to be rapidly written into the database which is (to simplify) basically the addition of a single row of data on the bottom of the database. Similarly, when a customer calls a call center and his/her data needs to be pulled up for the rep, that requires the rep’s software to read from a single row in the database. Operational/transactional databases were optimized to be able to perform these operations rapidly and reliably – row by row by row, one row at a time.
OLAP as duct-tape. But analysis requires an entirely different way of reading (and writing) data. Lets suppose a mobile operator wants to make a table of the top ten sales reps in each month who brought in the customers who generated the most revenue under a particular type of monthly plan. This is a very logical request from the perspective of the business analyst who is asking the question – but from the perspective of a traditional relational database, it’s a nightmare. To figure out the top ten each month, the BI system needs to read every row (a “full table scan”) of the relevant database and store in memory the top ten at every given phase until it can come up with the overall top ten in every month. In addition, the BI system will need to “join” the sales rep table with the customer revenue table – creating a new table – in order to perform this analysis. These types of operations are complex and costly in terms of time and computing resources. To overcome these challenges, OLAP (Online Analytical Processing) was developed. OLAP achieves dramatic time savings in doing these sorts of calculations by pre-processing the data and storing it in a series of “OLAP cubes” that are suited to potential queries of the type suggested above. The challenge with OLAP is that for very large dataset, the OLAP cubes themselves are massive (often larger than the original dataset) and – more importantly – very rigid. OLAP cubes must be determined in advanced and stored somewhere – and changing them as analytical needs changes is painful. Traditional enterprise BI implementations involved just this sort of massive complexity and cost. Business analysts need to determine ahead of time which reports or queries or drill-downs they might want to do, the database team then goes and determines the right OLAP cubes to meet those needs, and then an infrastructure team goes and figures out what storage and computing resources are going to be required to maintain and handle all of those massive unwieldy OLAP cubes that are going to be sitting on EMC boxes waiting for a user to call them up. This staging ground of OLAP cubes is what is called a “Data Warehouse” and its the critical (and most expensive) part of any BI implementation. Without a properly configured data warehouse, none of the analytical tools (such as SPSS) or presentational packages (such as Tableau or Spotfire) will be able to function effectively.
The Qlikview approach, and IPO. The reality is that OLAP (and data warehousing) is really a very sophisticated (and expensive) way of getting around the limitations of the underlying database architecture that is optimized for row-by-row read/write operations and not for data analysis. One alternative approach to this problem has been pioneered by Qlik Technologies, a Swedish company, that has been very successfully selling an in-memory database system called Qlikview. Qlik Technologies, which has filed to go public, has an Israeli connection – although entirely Swedish in original, it received an investment from JVP, which stands to profit handsomely from the IPO. The attraction of Qlikview is that by loading all of the database into memory, their system creates a simple “data warehouse” that doesn’t require pre-configuring and can run on any machine provided there is enough memory. This capability – and the huge demand from the market for low-cost flexible BI systems that actually work – has driven Qlik Technologies to be one of the fastest growing emerging software companies in the past five years – with revenues of over $150M in 2009. The incredible success of Qlikview is a powerful indication that BI is broken – but their approach is not without its weaknesses. First, it requires hardware with very large amounts of memory (this is easier to achieve than a full-blown data warehouse, but its still an obstacle). Second, Qlikview breaks down on very very large datasets – the sort of datasets that Internet companies (even small ones) generate on a regular basis. An Internet company I know of recently set out to try Qlikview and was told by their sales reps – “sure, but only if you buy new, more powerful hardware.”
Enter SiSense – The Data Warehouse Alternative. SiSense – from their small office in Netanya – has set out to completely rebuild the BI stack from the ground up starting with the underlying database itself. Here are the elements of the SiSense approach to BI:
- Going vertical. SiSense started by throwing out the traditional relational (row by row) database and adopting a vertical column-based (“columnar”) approach of the type pioneered by companies such as Vertica. This means that SiSense simply takes an enterprise’s data and writes it into a new database that is optimized for the sorts of analytical queries that are so challenging for relationship databases.
- Efficient swapping from disk to memory and back. Next, SiSense built a really smart memory management system. Unlike Qlikview, which throws the entire database into RAM in order to process it, SiSense selectively pulls individual columns out of the database and puts them into RAM for only as long as it needs to perform a particular part of a query. The vertical structure of its database makes this particularly feasible. The result is that SiSense should be able to handle far far larger datasets than Qlikview. In fact, everything in the SiSense approach is based on responded to queries on the fly. Very little is “staged” in advanced.
- Intuitive mapping and joining. SiSense has created a very intuitive interface that allows even a simple business user to map the relationships between various databases and tables (such as a sales rep table and a customer table) and connect them in a logical way. These tables do not need to be “joined” in advanced (as they would in OLAP). Instead, SiSense stores the relationships between the tables so that it can join them on the fly as required for a particular query.
- Ability to perform ad hoc analysis. As an occasional analyst myself, I know that analysis is a creative thought process. It’s about looking at a problem and saying “i wonder if we’d get an interesting result if we cut the data in this new way.” Traditional BI can not support this sort of ad hoc creativity because OLAP cubes must be defined in advance for specific sets of queries. SiSense, because it does away with OLAP, allows true ad hoc querying for the first time. My prediction is that even large business with full-blown BI implementations will find value in this aspect of what SiSense offers.
There are other advantages to SiSense (such as a beautiful web-based authoring environment and the elimination of the need for complex SQL scripting in favor of a GUI-based query language that any user can quickly learn), but the four advantages above are the most important ones. This is what SiSense means when it refers to its solution as a “Data Warehouse Alternative.” The problem with BI is first and foremost a problem with the way data warehouses are built. By addressing this challenge from the root causes, SiSense has the opportunity to lead a revolution in BI. I’ve seen the system running on very small datasets (my own) as well as on large datasets from internet companies with millions of rows. SiSense has been able to build BI solutions for customers in a few days – solutions that would have taken months of planning, provisioning, and infrastructure deployment using traditional methods.
No longer a four-letter word, but a massive market opportunity. As Qlikview has shown, there is a massive market for BI solutions that actually work – and with the growth of online businesses, this market is exploding. In the course of a few years, BI has gone from an untouchable four-letter word to an opportunity that many people recognize. Web business generate tons of data – even small ones. Every time a user action hits a server, there is a bit of data created in a database somewhere – data that could be analyzed if there was a proper BI system in place. Most web businesses are two small to afford large BI implementations and web-based offerings (such as Google analytics) do a great job of analyzing web traffic data, but don’t interface with other sources of data. In addition, the amount of data being generated on the web is unlike any other business – and it requires low-cost flexible solutions that can handle very large scale data.
SiSense has the potential to become a true Israeli success story. From their small office, they are taking on a massively complicated industry and literally rewriting the rules of how BI is done. It’s about time someone did that, and I wish them continued success on their challenging road ahead.