Inside the Community Health Index

Lithium Alumni (Retired)

Today we introduced Lithium Insights, a set of analytic solutions focused on enterprise online communities.    The first two solutions in the set are the Community Health Index (or CHI), and Lithium Lifecycle Benchmark Services.    I think these solutions represent a significant leap forward in the way online communities are measured, and I thought I'd give you a little insight into what they are, why they are important, and the road we took to developing them.   I'll also give you a peek at what's coming down the road from our analytics group at Lithium.


The Lithium platform has always had a pretty rich metrics tool inside.   There are currently more than 150 unique metrics in the system - everything from common metrics like posts and page views to relatively more arcane ones like wireless views, or net anonymous blog comments approved.  You can see these for the community as a whole, or for a unique category or element within a category (blog, forum, chat, or ideas).  With a single click you can see them by quarter, month, week, day, or hour, or you can choose custom ranges like last Tuesday to midnight last night.


So - lots of data.  Typically the way we've approached this data is to choose four or five high-level metrics indicative of community health - such as those Heidi Cohen shared in her column this week - and then used the remaining 145 metrics diagnostically.   That is to say, we watched for trends in high-level metrics, and then used more detailed metrics to better understand the trends we saw.   Here's a simple example I like:   In a forum, if you see post count dropping over time, you know you have a problem but you don't really know why, and you don't know what to do about it.  If you then pull data on threads and replies - essentially breaking post count down into those two different kinds of posts - you can see whether the drop is attributable to fewer questions (threads) or fewer answers (replies).   If the former, you probably have a problem with how you are promoting the community, since questions typically come from new users.  If the latter, you probably have a problem with how you are managing superusers, since they often create 30-40% or more of the replies in your community.   Pretty simple, pretty effective. 


But there are a couple of reasons why that breaks down over time.   First, almost all high-level metrics in a community are retrospective in nature.  Registrations, posts, page views, searches, visits, etc., all tell you a lot about yesterday not much about tomorrow.    So you can address problems when you see them, but you can't really predict them or prevent them.    Past performance is no guarantee of future results, as they used to say on Wall Street when there was a Wall Street.


Second, looking at high-level metrics in isolation can be very misleading.   I know from experience, for example, that page views are the most lagging of indicators - they can and do continue to scale long after your community begins to slide into failure. 


Third, high-level metrics become harder to read as a community matures.   Think about it - in the early weeks and months, a steep climb in page views and posts makes it pretty obvious you're succeeding.   But those growth curves tend to flatten over time, and in years 2, 3, 4, and 5, it can be harder to tell whether your community is really healthy or not.


At Lithium, we've been working with communities for more than 10 years.  As a software-as-a-service provider, we've followed all our communities continuously over time, and learned a lot about what a healthy community looks like.  Some of this is quantitative and some of it is frankly intuitive.   So how did we convert this knowledge into a quantitative index of health?


Here's how we did it:  we asked our statistician, Michael Wu, to create a model of community health.   He gathered the best of what we know from our most experienced community experts - I had a role in this - and he tested our knowledge and intuition against real data from our ten-year database of enterprise communities.    Essentially, we played hundreds of games of "Kasparov versus Deep Blue" over the past six months, testing our knowledge against Michael's model.


It wasn't always fun being a Kasparov - I think you can guess I didn't always win - but I learned some amazing things.   One thing I learned is that human beings always overestimate the health of large communities and underestimate the health of small ones.   Another thing I learned was that that quality we associate with communities that "feel like a community" versus pure question-and-answer actually has a statistical indicator.   After going through this exercise, it's clear to me that there's a vast amount yet to be learned - and that good tools for measuring communities are an absolute necessity for any of us working with communities today.


Today we published a whitepaper that shares our statistical model - we call it the Community Health Index, or CHI (rhymes with buy).   In the whitepaper you'll find a description of the model and all the formulas.   Like any statistical effort, it's a bit forbidding for the mathematically challenged (like me), so I'll be exploring simpler ways to describe it on this blog in the coming days and weeks.  But do check it out and let us know what you think.  To see how we're incorporating CHI in our reports, see this morning's blog post by Neil Beam, who led this effort at Lithium.  


We have created CHI as an open standard - the inputs are limited to data available from any community platform - both as an aid to anyone seeking to measure their online community and as a way to open the model up to exploration and critique by others in the field.   I'm sure we'll be tuning the model over time to increase its accuracy and relevance.   But I'm thrilled we've started on this journey, and I hope you'll join us, or simply follow along.


One final note:  community health is critically important, of course.   It is a measure of community success from the user's perspective.    But there are two large areas of measurement that go beyond community health.   One is return on investment - in other words, success from the business perspective.   The second is community insight and sentiment - what are users saying, and how do they feel?    Both are on the roadmap for our Lithium Insights program this year.   As with health, there are measures for both today but there is much, much work to be done to make them more reliable, more precise, and more useful.  You can trust that we will apply the same level of rigor to those efforts as we have with CHI.   

Message Edited by JoeC on 02-25-2009 10:08 AM
Tags (1)