Thanks for the reply. Your affirmation will ensure that this feature gets the attention it deserves. This is very good and I sincerely appreciate you taking the time to confirm that this is a good feature to have and that it is useful for you.
... View more
Thank you for the comment. I am truly honored to be invited to give a talk at the workshop on New Metrics for New Media: Analytics for Social Media and Virtual Worlds. I will try to cover some of my recent research along with the community health index during the workshop. And if time permits, I'd be happy to go over the art and science of designing an engaging ranking structure. I look forward to participating at the workshop.
... View more
Welcome back. I hope everyone had a relaxing Fourth of July holiday. Last week, Lithium delivered a new report, the community health report, to our clients. This report contains two important pieces of new information:
The community health index (CHI): a score that reflects the "health" of your community.
The community health compass: a radar chart that shows the relative value of the six health factors (traffic, content, members, liveliness, interaction, and responsiveness) that went into the computation of CHI.
Earlier this year we published a whitepaper on CHI. Since then, however, we've refined the CHI algorithm based on data from real-world use cases along with customer feedback. So for this post, I'd like to talk about the current state and future direction of CHI.
The Traffic Health Factor
Let me begin with the health factor you are most familiar with: Traffic. This health factor is most important when you are launching a community. If no one comes to your new community, the community will fail. As the community matures and acquires a large core of active users, the effect of traffic may become less important. This is because traffic reflects only the passive engagement of the visitors with the community. In a public community, visitors can accomplish quite a bit without any active contribution. For example, they can read messages, search for answers, navigate the community, checkout other members' public profiles, etc. These activities are considered "passive" because the visitors merely consume existing community content, but do not add content of their own.
Measuring Human Traffic is Not That Simple
In short, the traffic health factor is intended to measure the amount of human visits to the community. However, traditional page-view metrics also counts non-human visits by web robots and crawlers. Moreover, counting page views on a modern AJAX-filled website is not trivial. Because Web 2.0 technology furnishes users with a wide variety of interactions on a very dynamic webpage, users can potentially visit many places and perform many activities without refreshing the page. As a result, behind-the-scene REST API calls, RSS feeds, or any server-rendered pages go unnoticed to conventional page-view trackers such as Google Analytics.
The Current and Future of Traffic
In the current formulation of CHI, traffic is measured by Lithium's own PageView metric, which includes REST calls and any server-rendered pages. Although our preliminary study suggested that robots and crawlers do not significantly affect the final CHI score of the community, we intend to remove their contribution to our PageView metric in our next formulation of CHI. However, since robots do not always declare their identity (in fact some crawlers may even intentionally disguise themselves as human visitors), it is not possible to completely remove the effect of robots and crawlers. Despite this, through iterative reformulation, we hope to derive a traffic health factor that can track the passive engagement of human visitors more accurately.
I hope this post gave you a better understanding of the traffic health factor. Next time, let's talk about the content health factor. In the meantime, feel free to continue this conversation in the comments section below.
... View more
This week, I'm going to digress from the topic of flow and rank ladders in order to share an interesting talk I heard at a conference. Last week, I was at the C&T2009 meeting at Penn State University with our Chief Community Officer, Joe Cothrel. Although this meeting was rather relaxing for me, because I didn't have to present, I still can't believe it took me 12 hours to get there (9 hours of travel from SFO to State College connecting at DC, plus 3 hours lost from PST to EST). I stayed on campus at the Nittany Lion Inn that is a 5 minute walk to the IST building, the meeting venue location. I will not bother to recap the meeting, since a concise summary can be found in the conference program. However, one talk sparked some thoughts in my head that I'd like to share with you.
Day 2 of the conference opened with a keynote, titled "Knowledge Reuse and Novelty in Community Settings," delivered by Prof. Karim R. Lakhani from Harvard Business School. Prof. Lakhani presented an interesting experiment on collaborative innovation in the form of a MATLAB programming contest for solving an NP hard problem. Each code entry's performance is evaluated, scored, ranked, and displayed immediately with the contestant's name. Since this is a collaborative effort, any contestant may reuse and modify code submitted by others and then resubmit it as their own entry. The contest is closed after about a week and the top score at that time wins regardless of how many times one submits, how many lines of code one adds, or how much performance gain one contributes. The competition is all about reputation, collaboration, and learning; the winner only gets a T-shirt or a cap.
The results of this experiment are quite interesting!
Novelty per entry is quite low: 3.5% on average.
Borrowed code per entry is rather high: 71% on average.
Small chunks of novelty code are often reused, so they tend to have high social values. But code entries that are too novel (have too many novel blocks of code) are often not reused because they are too hard to understand. Therefore their social value decreases.
In contrary, small chunks of borrowed code are not often reused, so they tend to have lower social values. However, as the sizes (number of lines) of the borrowed blocks of code increase, they become reused more often, so their social value increases.
Winning entries tend to have few lines of novel code and many chunks of borrowed code. In fact, the amount of borrowed code is twice as predictive of top performance as novelty.
Finally, collaborative innovation almost always leads to a more optimal solution in shorter amount of time.
Since all communications in a community are persistent and are made available through the internet to the rest of the world, a community is a fertile ground for collaborative innovation. Although the amount of novelty per post is usually negligible, through many iterative refinements by many users from different backgrounds, the solution is often highly optimized and very innovative. This method of innovation and optimization is actually very similar to how evolution optimizes certain biological motifs through natural selection. Computer scientists have found this optimization method so effective that they invented the field of evolutionary computing through biomimicry.
Now, how would you like to run a similar type of collaborative innovation "contest" on your community? Lithium is geared up for a new product that will enable you to reuse the great content in your community, collaborate, innovate and produce highly valuable knowledge base articles. Watch out for our Tribal Knowledge Base (TKB) products announcement soon!
... View more
Previously, I have blogged on using the principle of flow to build ranking ladders and scale them to match the skill level of the superusers in your community. Because these blog articles are the foundation for this blog, I strongly recommend reading them first if you haven't. They can be found here:
1. Spacing the rungs of your ranking ladder
2. Know your superusers!
Flow with your most prolific superusers
In a benchmark study I conducted last year, we found that healthy and vibrant communities that have many superusers generally have a large number of ranks. In fact, the benchmark list of top communities has an average of 31 non-role-based ranks (ranks that are achievable through participation). If we include role-based ranks (that were assigned), the average is 59. Among these top communities, the number of ranks goes as high as 134 ranks. But does this apply to your community? The important questions are: How many ranks does YOUR community need, and how many is enough?
The answer is that you need as many ranks as necessary to keep your most active superuser engaged. The exact number will depend on how prolific your most active superuser is. I will illustrate this with the calculation on Lithosphere again. Last time we've designed an optimal ranking ladder for Lithosphere. The post requirement for each rank are: 3 posts, 9, 18, 30, 45, 63, 84, 108, 135, 165, 198, 234, 273, 315, 360, 408, 459, 513, 570, 630, 693, 759, 828, and finally 900 posts at the 24th rank. We've also calculated the post rate for all users who have been in the community for more than 2 weeks and sorted them. If you look at the previous blog, you will see that ScottD is the most active superuser on Lithosphere (he also happens to be our community admin, but let's ignore that for now). He has posted 451 messages in total and his post rate is 1.16 posts/day. If ScottD wasn't our admin, he would be on the 16th rank now. So the proposed rank ladder with 24 levels is definitely enough for now.
A natural question is when will this rank ladder become insufficient? If ScottD posts 450 more messages, his post count will exceed 900. After that, his contribution will no longer be rewarded by this ranking ladder. Soon after, he may become bored with the community. How long would that take? Since we have ScottD's post rate, 450 more posts would take him about (450 post)÷(1.16 posts/day)=388 days. So in a little more than a year, it will be time to adjust this ranking ladder. What do we need to do a year from now to keep ScottD engaged in his personal flow state? Add more ranks! But how should we set the rank criteria? Remember, if it's too easy, ScottD will be bored, and if too hard, he might become frustrated and leave.
Note that the gap between the 23rd and 24th rung of the ranking ladder is (900-828)=72 posts. For typical Lithosphere superusers with post rate of 0.851 posts/day, this will take them about (72 posts)÷(0.851 posts/day)=85 days, which is almost 3 months. Even for ScottD, the most prolific superusers on Lithosphere, it will take him about (72 posts)÷(1.16 posts/day)=62 days, which is about 2 month. This means the gaps between the top rungs of our existing ranking ladder is already challenging for our superusers. So after the 24 linearly incremental ranks that are designed to engaged the superusers of Lithosphere, it is a good time to switch to the arithmetic progression (a.k.a linear progression), which has a constant growth rate. The logical choice is to continue the rung spacing between the top ranks of the existing ranking ladder. How many ranks we add depends on how soon we want to adjust the ranking ladder again. If we want to challenge ScottD with the appropriate difficulty for 1 more year, we will need to add at least 6 ranks, starting with the 25th rank requiring 972 posts, and then spaced evenly every 72 posts thereafter.
Why wouldn't we just add 60 more ranks and be good for the next 10 years? You can, but I certainly would not recommend that because ScottD's capability may change. Perhaps another more prolific superuser will come along, or maybe some other superusers will surpass ScottD. So we may need to change the spacing of the linear progression again the following year. In general, it is a good idea to re-compute the post rate of your top superusers yearly (or semi annually) and adapt the post criteria to keep the "flow" with your superusers. Alternatively, if you have a superuser MVP program, you can also switch your top contributors to an assigned-rank system to build a more personal relationship with your superusers.
Next time we will add some mysteries and surprises to spice up your ranking ladder and explore what happens after we switched the ranking criteria over to the arithmetic progressions. Come and follow my update at mich8elwu. May the flow be with your superusers!
... View more