In my previous big data post, we discussed the three necessary criteria for information to provide insights that are valuable. Through this discussion, we learned the key to insights discovery.
...
Updated 8 months ago
Version 7.0
Blog Post
Michael, nice blog post thanks. I hope my dry joke is a good response to the points you raise about the value of the creative process in performing EDA...
Q: How do you find a needle in a haystack?
The linear approach: Draw up a R&D proposal for an innovative device, costing $10m in budget and just under $20m on final completion once regulatory approval has been achieved. This device can harvest for needles in any given haystack in any terrain at any time of the day or night, and be operated in-situ or remotely.
The non-linear approach: Faced with such a heterogeneous organisation of data you assemble a bunch of friends (size dependent on amount of free food & drink) and hold a wild party on the haystack. One of the partygoers will bound to find the needle simply by stepping, or sitting on it. Or if they don't something much more strange + interesting will appear, so that the needle is simply classified as a variant hay-straw and ignored. And the new discovery classified as the strange attractor - that "diamond in the rough" you refer to!