Wednesday, June 06, 2012

#iel12 7+/-2 Things ...Analytics

Entire title: 7+/-2 things learning professionals need to know about analytics
Speaker: Ellen Wagner
Analytics is hot.  Means a lot of different things

7 +/- 2 - capacity for people to remember things

Why do analytics matter so much?
- The storm is coming.  Will make our landscape look different
- The data that we leave behind - digital breadcrumbs.
  + Tell people LOTS about us
- This data can be mined to better "optimize online experiences"

Lots of data we leave behind all the time
- Likes on facebook, Purchasing, view, etc
- We are leaving this behind all the time

Focus / personalize / make more convenient our lives online

Major trends at play
- Data warehouses and "the cloud" affregates
- Sophisticated tech platforms provide computing power necessary for grinding through calculations and turning the mass of numbers into meaningful patterns
- Can use data mining - look inside those patterns for actionable info
- Predictive techniques help anticipate behaviors and events

The safety net - tons of data..  Looking for averages vs. individual transactions
- We now have so MUCH data that we may not understand what is going on anymore.

All you are doing is maximizing probability for success.
- You are increasing the odds of being right
- Trying to be in place to take advantage

It's about business intelligence
- Move from reacting to events to proactively seeking patterns
- Be ready for what you don't know what is coming at you.

We are trying to create action plans
- Business intelligence and research - related but separate

Try to be prepared for the next big wave

Pulls from lots of different sources
- If you look at all the different sources (web, mobile, blogs, apis, etc) - eventually will need to push into way of looking at this info (comprehensive analytics - dashboards, etc)
- This information then helps you optimize the business

Big data analytics - 5 things in the value change
- measure
- execute
- automate
- extend
- innovate

Learning and development cannot live outside the enterprise focus on measurable, tangible results.
- Learning professionals still believe that quantifying learning is Orwellian
- the reality is that we cannot live within our communities and not pay attention to tangible measurement and value add

It's about bringing change to practices (and our practice)

Assumption that digital breadcrumbs help us improve and personalize experience
- Learning pros are nervous
- Problem: no common definitions and frameworks

There is a period of innovation where we are going to spend a lot of time looking for the "one true answer"
- "I hope we succeed"

Where do you start?
- We have no established industry best practice about what to measure or methodology of measurement
- Enterprise concern about the data will show
- Competing priorities and lack of incentive for collaboration
- Siloed data across the enterprise doesn't help

CEO and CLO not so sure about what to do about learning analytics

What is making people nervous - we are not getting "answers" per se.  More probabilities.

Kris Rockwell - the analytics are really driven by the regulatory bodies

In higher ed - "Completion Agenda"

Vision vs. execution - we are in between

So what are your benchmarks?
- What story do you want to tell?
- The numbers don't really matter much

What's different for eLearning vs. Business - we measure slightly different things
- PLease stop measuring number of students served
- We are still looking at measuring and optimizing

We can look at
- Engagement
- Retention
- Learning Effectiveness - what does this mean for YOU?
- Completion - this is a nice benchmark

Every one of these words do not have a hard-core number

Where assessment gets measured
- remember: look at LOTS of inputs / data points
- These data points do not necessarily associate

Multiple levels of analysis
- Exploratory statistics - comprehensive, higher confidence levels for prediction, 1%
- Inferential - single-system. low confidence levels for prediction 9%
- Descriptive, subjective interpretation.  90%

Most of us are working at the descriptive level.

Inferences are fine - but we don't get the answer.

All of us needs to do more with less, areas we aren't familiar with.
How do we move people forward?  Can't do it our old ways.

Where data actually lives - enterprise systems, lms, end of course surveys

What are the 3-5 things in your organization that REALLY make a difference?

One way to approach - "what is being used the most" Do more of that (Craig Wiggins)

Click stream filled with data.
- Different levels of analysis
- Different levels of stakeholders (CEO looking for something different from Dept Chair)
- Engaging in overkill is the worst mistake you can make
  + We are staring at a few years of some really messy stuff

Most orgs - when talk about learning analytics, almost exclusively stuff out of the LMS. (Reubin)
- But what is happening on the consumer-facing side?
- There is almost 0% learning folks going out to organizations asking for THEIR information to serve THEM better.  (agreed)

The promise and direction is fabulous.  But the line-level instructors care about this too.

Lessons from Moneyball
- Where is the game actually being WON?
- We have not yet considered where the game is being won.  How can we TELL if we have won?
- Little money in competitive market

Predictive Analytics Reporting Framework
- Big Data project using predictive statistical analyses to identify factors affecting retention, progress and completion
- Came up with 33 variables.  But it meant something different to every school.  So had to standardize the definitions
- What types of analysis tells us more reliable things

Some Preliminary findings
- Students at risk - more courses at once = more likely to drop out.
- No apparent relationships between age, gender or ethnicity as a function of the student's risk profile
- Students not at risk - institution-specific factors predicted student success.
  + Could we do a for college admissions?  Would this be a good thing?

Putting Analytics to Work
- analytics are here today and here to stay.  We are NOT ahead of the curve.

- Someone on your team needs to know statistics and research techniques - you need data whisperers

- doing research on analytics is fundamentally different than applying analytics to make learning successful

- it's what we do with the analytical findings that matter

- we already have more data than we can handle

- more interesting data collection opportunities await

- we need to be prepared to live under the "sword of data"

- There is no such thing as "sort of" transparent

- we haven't begun to scratch the surface of the possibilities - because we don't know where it's going
  + we are in the middle of a gold rush in education

- Beware Bozosity - "the Bozo in all of us"
  + "I just need to figure this out..I just have to do this"
  + we make decisions based on what is immediate
  + Even more beware of the Bozo who comes with a "solution" to your problem

We have an exciting few years ahead of us.

No comments: