September 2012

Invisible loading: A new paradigm for loading from unstructured to structured storage

Posted on September 5, 2012 at 11:00 pm.

Written by Daniel Abadi

In my last post on this blog, I outlined the flaws of the ubiquitous Hadoop-DBMS “connector” technology that unnecessarily links together two different systems that have an extremely similar architecture. In this post, I will discuss a new loading technology we … Continued

Recent Hadapt Blog Posts

Classifying Today’s “Big Data Innovators” Friday, December 21, 2012 at 9:57 am | Daniel Abadi

Uncategorized (7)

Keith PrinceOh dear, MPP and Hadoop comparisons based around the premise that "There is absolutely no technical reason why there needs to be two separate systems doing the exact same type of parallel processing". I'd 100% agree that there are some folks who are replicating the same type of workload on two platforms. The early adopter/leaders understand that MPP systems don't do parallel operations in the same way as Hadoop. That's why you'll see Google, Yahoo, Amazon, etc, use Hadoop architectures for certain types of operations and MPP systems (mainly appliances today) for other types. Each architecture is optimized for a different type of workload. And there are some relational databases that do poly-structured really well so it should not be a structured vs unstructured debate too. Yes you can race a rally car at a F1 meeting, it will go around the track pretty rapidly and look great but it won't compete. I nearly said its a horses for courses thing but then I'd be disappearing up my own metaphor :). Apart from the fundamental point that parallel architectures are not the same, I think more articles like yours should fuel a more honest appraisal of how things really work. Then again the in-memory folks will come along and say Hadoop and MPP are both yesterdays news!Roberto V. ZicariTwo comments: - I wonder how in-memory databases for analytics such a SAP HANA fits into your classification - I`d suggest to include MPP database vendors in the classification as well (e.g. Teradata, Vertica) to make it complete.Manisha AroraHow flexible is your interactive query language? I am assuming that you have a certain algorithm you are using to calculate result set like social buzz on certain data set(structured and unstructured), packing it and exposing it as a variable through the BI UI layers. But, when your platform is deployed within an enterprise and is sitting on top of both legacy structured data from ERP systems and unstructured data(social, documents), the business analysts may want to create various variables to show in their UI reports like "Satisfaction index" on suppliers(which will require searching social data for mentions of supplier, sentiment associated with those mentions) and also structured data like service calls generated in customer support support system. How easy is it for the business analysts to create a definition/formula for calculation of satisfaction index and then have hadapt interpret it? It will be great if you guys have an interface where you expose the structured data elements and also packaged calculations on the unstructured data elements and allow the user to build their own queries which combines both of these.