Decision Sciences

Big Data As Defined By Constraints: "Camel Meets Eye Of Needle", Or, "Surviving Your First Week On The Job"

Needle-And-ThreadCamelIf you are an executive charged with a "big data project", here's a prediction. On your first week on the job you'll likely be surprised. And the surprise concerns what you'll actually spend your time doing.

What might be the surprise on your first week on the job? The surprise is that your biggest responsibility won't concern what you thought you signed up for, which is likely something like "generating analytical insights". Rather, your biggest responsibility is almost the opposite, which is focusing on everything but generating analytical insights.

And if you've been dreaming of fishing in sea of data, with a big net to scoop up any of the myriad insights just swimming by your boat, your surprise might even be one of disappointment. Because this blue ocean metaphor, lovely to imagine, is also seriously misleading about the nature of the world of big data. It's more likely that your ocean will consist of nothing but fish, all of which have three eyes and which are inedible! You'll have lots of data but nothing to take home that you can call a great insight.

Here's the problem, which could define the basis of your surprise, your job responsibility and the sort of metaphor that you should use to describe the world of big data. The problem with big data concerns the very definition of big data, which is that it is "big" -- but although big is subjective, there's a very practical and non-trivial definition of "big". Your set of big data is defined as such if it is too big for any of your machines to process in one chunk in a reasonable amount of time. In other words, the world of big data is defined by the system capacity constraints.
. . . read more

Why Big Data Can Be Like A Jelly Fish: No Bones & You Might Get Stung

Jellyfish metaphor for big data without modelingWhat is it helpful to think of big data as "like a jelly fish"?

Both big data and jelly fish can be very beautiful. But if not treated with respect, you might get stung. And how you get stung, at least where data is concerned, relates to the other similarity between jellyfish and big data, which is a lack of bones, or structure.

To be in any way useful, big data needs to be used alongside an interpretive structure or model – the “bones” if you will, without which big data is as amorphous and useless as a jellyfish. The necessity of having this model is a critical challenge for any organization seeking to derive benefit from big data.

Let's prove the point about the necessity of interpretive structure with the simplest possible model.

Consider the results of a query on a joined accounts receivable table. You may have columns representing company names, owed amounts and due dates. And the meaning of the table is completely clear, but only because you know the column headers and table name, which comprise a simple "model" providing the meta-data meaning of the table. Without this "interpretive model" the table could just as easily imply accounts payables as account receivables! . . . read more

Syndicate content