Feeds:
Posts
Comments

Archive for the ‘Data Science’ Category

 NewImageJeffrey Leek, Assistant Professor of Biostatistics at John Hopkins Bloomberg School of Public Health, has identified six(6) archetypical analyses. As presented, they range from the least to most complex, in terms of knowledge, costs, and time. In summary,

  • Descriptive
  • Exploratory
  • Inferential
  • Predictive
  • Causal
  • Mechanistic

1. Descriptive (least amount of effort):  The discipline of quantitatively describing the main features of a collection of data. In essence, it describes a set of data.

– Typically the first kind of data analysis performed on a data set

– Commonly applied to large volumes of data, such as census data

-The description and interpretation processes are different steps

– Univariate and Bivariate are two types of statistical descriptive analyses.

– Type of data set applied to: Census Data Set – a whole population

 Example: Census DataNewImage

2. Exploratory: An approach to analyzing data sets to find previously unknown relationships.

– Exploratory models are good for discovering new connections

– They are also useful for defining future studies/questions

– Exploratory analyses are usually not the definitive answer to the question at hand, but only the start

– Exploratory analyses alone should not be used for generalizing and/or predicting

– Remember: correlation does not imply causation

– Type of data set applied to: Census and Convenience Sample Data Set (typically non-uniform) – a random sample with many variables measured

Example: Microarray Data Analysis NewImage

3. Inferential: Aims to test theories about the nature of the world in general (or some part of it) based on samples of “subjects” taken from the world (or some part of it). That is, use a relatively small sample of data to say something about a bigger population.

– Inference is commonly the goal of statistical models

– Inference involves estimating both the quantity you care about and your uncertainty about your estimate

– Inference depends heavily on both the population and the sampling scheme

– Type of data set applied to: Observational, Cross Sectional Time Study, and Retrospective Data Set – the right, randomly sampled population

Example: Inferential Analysis NewImage

4. Predictive: The various types of methods that analyze current and historical facts to make predictions about future events. In essence, to use the data on some objects to predict values for another object.

– The models predicts, but it does not mean that the independent variables cause

– Accurate prediction depends heavily on measuring the right variables

– Although there are better and worse prediction models, more data and a simple model works really well

– Prediction is very hard, especially about the future references

– Type of data set applied to: Prediction Study Data Set – a training and test data set from the same population

Example: Predictive Analysis

NewImage

 

Another Example of Predictive Analysis

NewImage 

5. Causal: To find out what happens to one variable when you change another.

– Implementation usually requires randomized studies

– There are approaches to inferring causation in non-randomized studies

– Causal models are said to be the “gold standard” for data analysis

– Type of data set applied to: Randomized Trial Data Set – data from a randomized study

Example: Causal Analysis

NewImage 

6. Mechanistic (most amount of effort): Understand the exact changes in variables that lead to changes in other variables for individual objects.

– Incredibly hard to infer, except in simple situations

– Usually modeled by a deterministic set of equations (physical/engineering science)

– Generally the random component of the data is measurement error

– If the equations are known but the parameters are not, they may be inferred with data analysis

– Type of data set applied to: Randomized Trial Data Set – data about all components of the system

Example: Mechanistic Analysis

NewImage

Please also check out my dedicate blog on data science.

Read Full Post »

NewImage Data scientist recruiting can be a challenging task, but not an impossible one. Here are eleven tips that can get you going in the right recruiting direction:

1. Focus recruiting at the universities that have top notch computer programming, statistical, and advance sciences. For example, Stanford, MIT, Berkeley, and Harvard are some of the top schools in the world.  Also a few other schools with proven strengths in data analytics, such as: North Carolina State, UC Santa Cruz, University of Maryland, University of Washington, and UT Austin.

2. Look for recruits in the membership rolls of user groups devoted to data science tools. Two excellent places to start are The R User Group (for an open-souce statistical tool favored by data scientists) and Python Interest Groups (for PIGies). Revolutions provide a list of known R User Groups, as well as information around the R community.

3. Search for data scientists on LinkedIn, many of which have formed formal groups.

4. Hang out with data scientists at StrataStructure:Data, and Hadoop World conferences and similar gatherings or at inform data scientist “meet-ups” in your area. The R User Group Meetup Groups is an excellent source for finding meetings your a particular area.

5. Talk with local venture capitalist (Osage, NewSprings, etc.), who is likely to have gotten a variety of big data proposals over the past year.

6. Host a competition on Kaggle (online data science competitions) and/or TopCoder (online coding competitions), the analytical and coding competition websites. One of my favorite Kaggle competitions was the Heritage Provider Network Health Prize – Identified patients who will be admitted to a hospital within the next year using historical claims data.

7. Candidates need to code. Period. So don’t bother with any candidate that doesn’t understand some formal language (R, Python, Java, etc.). Coding skills don’t have to be at a world-class level, but they should be good enough to get by (hacker).

8. The old saying that “we start dying the day we stop learning” is so true of the data science space. Candidates need to have a demonstrable ability to learn about new technologies and methods, since the field of data science is exponentially changing. Have they gotten certificates from Coursa‘s Data Science or Machine Learning course; contributed to open-source projects; or built an online repository of code or data sets (e.g., Quandl) to share?

9. Make sure a candidate can tell a story in the data sets they are analyzing. It is one thing to do the hard analytical work, but another to provide a coherent narrative about a key insights (AKA they can tell a story). Test their ability to communicate with numbers, visually, and verbally. 

10. Candidates need to be able to work in the business world. Take a pass on those candidates that get stuck for answers on how their work might apply to your management challenges.

11. Ask candidates about their favorite analysis or insight. Every data scientist should have something in their insights portfolio, applied or academic. Have them break out the laptop (iPad) to walk through their data sets and analyses. It doesn’t matter what the subject is, just that they can walk through the complete data science value chain.

Please check out my dedicated Data Scientist Insights: Exploring The Darkest Places On Earth blog.

Read Full Post »

2013 01 23 10 26 42Moxie Group’s Creative Director Tina Chadwick makes the case that real-time data analytics “brings us tangible facts on how consumers actually react to almost anything.” She makes light of the “notion that 10 people in a room, who volunteered to be there because they got paid and fed,” could truly represent consumer behaviors (psychographics) is a thing of the past. Sadly though, for many advertising companies, this is still the mainstay of their advertising-oriented evaluative methodology.

New capabilities based on neuroscience, integrating machine learning with human intuition, and data science/big data is leading to a new creative processes, which many call NeuroMarketing, the direct measurement of consumer thoughts about advertising through neuroscience. The persuasive effects of an advertising campaign (psychographic response) are contingent upon the emotional alignment of the viewer (targeted demographic); that is, the campaigns buying call to action has a higher likelihood of succeeding when the viewer has a positive emotional response to the material. Through neuroscience we can not directly measure emotional alignment without inducing a Hawthorne Effect

This is new field of marketing research, founded in neuroscience, that studies consumers’ sensorimotor, cognitive, and affective response to marketing stimuli. It explores how consumer’s brain responses to ads (broadcast, print, digital) and measures how well and how often media engages the areas for attention/emotion/memory/and personal meaning – measures of emotional response. From data science-driven analyses, we can determine:

  • The effectiveness of the ad to cause a marketing call to action (e.g., buy product, inform, etc) 
  • Components of the ad that are most/least effective (Ad Component Analysis) – identifying what elements make an ad great or not so great.
  • Effectiveness of a transcreation process (language and culture migration) used to create adverting in different culturally centric markets.

One of the best and most entertaining case studies I have seen for NeuroMarketing was done by Neuro-Insight, a leader in the application of neuroscience for marketing and advertising. Top Gear used their technology to evaluate which cars attract women to which type of men. The results are pretty amazing.

While NeuroMarketing is an emergent field for advertising creation and evaluation, the fundamentals of neuroscience and data science make this an essential transformational capability.  For any advertising agency looking to leap frog those older, less agile companies that are stilled anchored in the practices of the 70s, neuromarketing might be the worth looking into.

POSTED FROM: Data Science Insights: Exploring The Darkest Places On Earth

Read Full Post »

NewImageData science is changing the way we look at business, innovation and intuition. It challenges our subconscious decisions, helps us find patterns and empowers us to ask better questions. Hear from thought leaders at the forefront including Growth Science, IBM, Intel, Inside-BigData.com and the National Center for Supercomputing Applications. This video is an excellent source of information for those that have struggled trying to understanding data science and its value.

Read Full Post »

NewImageData Visualization is a part, very significant part, of the big data story. The human brain, through its 100 billion neurons each interconnected 10,000 times, has an absolutely amazing visual processing capability, which is arguably surpassed by none. So leveraging this ability should not come as a surprise when thinking about the human component in the quadrication of big data.

Paul Butler, a FaceBook intern, discovered this earthly visualization using R, an open source statistical application. Based on around 10 million samples of Friends relationships taken from FaceBook’s Hadoop Apache Hive (data warehouse system for Hadoop), he was able to plot the weights for each pair of cities as a function of the Euclidean distance between them and the number of their respective friends. The result is this astonishing image of the earth. By the way, notice anything missing in the geographical location where China normally resides?

NewImage

Here is another awesome image, this time created by Eric Fischer using a heat map (a type of visualization tool) of places with Flickr photographs and Tweets.  While Twitter and Flickr may not individually have enough users to create a map so detailed as the FaceBook image, but put them together, and there’s a wealth of information that can be discovered in all of its glory.

NewImage

Data visualization is a powerful exploratory tool that should be exploited as early as possible when working to monetize your big data. Do not assume away discoveries just because you don’t know what you don’t know (third level of knowledge). Let the data take your brain on an unguided journey where discovery is the ultimate destination.

 

Read Full Post »

NewImageDefinition: “Extremely scalable analytics – analyzing petabytes of structured and unstructured data at high velocity.”

Definition: “Big data is data that exceeds the processing capacity of conventional data base systems.”

Big Data has three characteristics:

Variety – Structured and unstructured data

Velocity – Time sensitive data that should be used simultaneously with its enterprise data counterparts, in order to maximize value

Volume – Size of data exceeds the nominal storage capacity of the enterprise.

NewImageStatistics:

– In 2011, the global output of data was estimated to be 1.8 zettabytes (10^21 bytes)

– 90% of the world data has been created in the last 2 years.

– We create 2.5 quintillion (10^18) bytes of data per day (from sensors, social media posts, digital pictures, etc.)

– The digital world will increase in capacity 44 folds between 2009 and 2020.

– Only 5% of data is being created in structured forms, 95% is largely unstructured.

– 80% of the effort involved in dealing with unstructured data is reconditioning ill-formed data to well-formed data (cleaning it up).

Performance Statistics (I will start tracking more closely):

– Traditional data storage costs approximately $5/GB, but storing the same data using Hadoop only cost $0.25/GB – yep 25cents/GB. Hum!

– FaceBook stores more than 20Petabytes of data across 23,000 cores, with 50Terabytes of raw data being generated per day.

– eBay uses over 2,600 clustered Hadoop servers.

Read Full Post »

NewImageHere are some interesting statistics from IDC:

>> An organization employing 1,000 knowledge workers loses $5.7 million annually justin in time wasted having to reformat information as they move between applications.

>> The same organization will loose another %5.3 million from not finding information that is in the organization

>> Over 95% of the digital universe is “unstructured data,” that is, content that can not be represented by its field in a record (e.g., name, address, date, etc.).

>> In most organizations, unstructured data accounts for more than 80% of all information.

 

An
or
ga
niz
a
tion
em
pl
oying
1
,
000
kno
w
led
ge
w
orker
s
los
e
s
$
5
.
7
mi
llion
a
nnu
a
lly
j
u
s
t
in
tim
e
w
a
sted
havin
g
to
reform
a
t
inform
a
tion
as
the
y
mov
e
a
m
on
g
a
pp
lica
tion
s
.
No
t
find
ing
inf
orm
a
tion
cos
t
s
that
s
a
m
e
org
a
niz
a
tion
an
a
d
d
itional
$
5
.
3
m
a
y
ea
r
.

Read Full Post »