In my previous article, I spoke about the most common value levers that we can measure within an Online Community in order to evaluate what the Community is doing for the business. More often than not a stakeholder within an organization wants to know, “is this darned thing making me money or is it saving me money?” I can never begrudge anybody with such a question that spent money on Khoros software and allocated employees to their Community endeavor. In fact, I always want to do my best to answer these types of questions.
It turns out that a majority of our Community customers (around 60%) are primarily looking to save money, and so it is often the case that I do the calculations and subsequently explains that the Community is saving them money. There are, of course, exceptions. If the Community is not successful, then that is a bigger problem, and so I do not really talk about what the Community is doing for the business, but rather, what the business is leaving on the table when they do Community incorrectly. But even the Communities that do not have ‘saving money’ as their primary objective end up with a little bit of Savings anyway. Why? Because we are living in an era where people prefer to digitally self-serve. When somebody solves their own problem by reading a knowledge-base article or peer-to-peer discussion, then that self-service *potentially* deflects a contact to their Customer Support center.
There is nothing new about people resolving their own issues with a brand’s products or services. Have you ever heard of the team ‘shadetree mechanic’? Back in the day when cars were a lot more simple, a great number of regular people knew how to fix them, and those folks could do it in a few hours (or minutes!) under the shade of a tree. Time marches on though, and so much of the world has become more complex.
For me, the peer-to-peer Community has always represented a light in this complex darkness. No matter what I was interested in, there was a Community for it. And within those Communities were subject-matter-experts that could help answer my questions on any arcane subject. Better yet, I would commonly find that somebody else had already asked my question and that these subject-matter experts had already answered it. The treasured knowledge, which had been delivered over a year ago unto a poor sap with a problem identical to mine, was now sitting right there for me to learn from. Eureka!
But let us now make sure we really understand the mechanics of how a conversation saved money. If a dialogue around somebody asking a question on a peer-to-peer Community is answered by another more knowledgeable peer, that is great. Potentially, an inquiry to the brand’s Support team was avoided. But the real action is in how that particular answer lives on in perpetuity within the Online Community. In the field of self-service, the biggest cost savings are not found in the number of Accepted Solutions on a Community (i.e. - the Direct Contact Deflection of one person answering another), but rather in the subsequent views of those Accepted Solutions over time by large amounts of people that use the Community in an anonymous and passive manner (i.e. – Indirect Contact Deflection). So it is not ask-and-answer where the magic happens. It is in the ask-and-answer being viewed hundreds, thousands, or tens of thousands of times. Some have called this a ‘force multiplier effect’. Let us now examine it.
The formula to measure the mechanics of traffic, interactions, and your business, looks like this:
# of Visits to the Community during a given timeframe
% of Visits seeking support
% of Visits that are 'seeking support' who resolve their issue (aka - 'Resolution Rate')
% of those that were ‘seeking support’ and have successfully ‘resolved their issue’ which would have contacted Support if they had not solved their problem via the Community
Multiplied by the Average Cost-per-contact to the Support Center
(NOTE - use the 'Fully Burdened' cost per contact)
= Gross Savings
So a savings story might look something like this....
5,000,000 Visits to the Community in 1 year
70% of those Visits were seeking support
40% of those 'seeking support' Resolved their issue
50% of those that were 'seeking support' and have successfully 'resolved their issue' would have contacted Support if they had not solved their problem via the Community
X $12 cost per contact
= $8,400,000 in (gross) savings
For the more visually inclined, here is a picture of the same formula (with the same numbers) in action:
Measuring savings can be done on a weekly (or even daily!) basis, but our recommendation is to run the formula annually or, if it suits your business, quarterly
Though not required, it is recommended that the Khoros ‘Value Analytics’ survey be enabled on the Community. This survey asks the precise questions in the exact order of the formula. It is both included in the platform and an extremely non-fatiguing survey experience for the Community Visitor. Although the questions in our Value Analytics survey cannot be customized, our Value Analytics survey feature has the necessary flexibility to accommodate 3 important contexts; where it is served on the Community, when it is served on the Community, and to whom it is served on the Community
Though a brand may run their own survey (we have no problem with that!), be mindful of extraneous questions that are not relevant to the Community experience, much less the self-service experience. When left to their own devices, brands run surveys too quickly in the customer experience and/or they simply do not ask the right questions. The results are either an extremely low survey completion rate, or worse, a survey full of reactive (i.e. ‘mean spirited’) responses.
This is how you measure ‘Contact Deflection’ Savings in an Online Community. Easy, right?
We have struggled to effectively use metrics from the Khoros 'Value Analytics' survey responses to get a meaningful output using the contact avoidance formula described in this article: the raw data suggests a fantastically and unbelievably high number of contact avoidances in our case. They suggest we're deflecting far more contacts than we ever received before we had a community, and the community certainly hasn't dropped our contacts to 0.
Have other people had success in identifying concrete data that gives numbers to plug into this formula to get believable results?
This is great formula as it is but there is one problem - if you look how many visits per giving time there is it includes all the visits. I mean human visits and bot visits, visits what last only second or so.
So the ROI is way bigger than it actually is.
Or, is the bot visits excluded from the Community analytics and Visits metrics / Unique visitors?
From what I remember, bots are excluded from Community Analytics. From the Khoros blog post:
Page Views approximates user behavior and therefore excludes non-human requests (Web crawlers, robots, RSS feeds, REST API calls, ActiveCast calls, and LevelUp requests).
A visit is one or more page views over time by a uniquely identified client (visitor).
@karolinalinda yeah Page views metrics has bots excluded but how about with the visitors / unique visitors? I can't find any mention about those.
@cgrinton I agree that the calc won't necessarily reflect actual because customers often don't really call for support for questions they're asking. They have these other behaviors:
[Skip any and all of this if you've already done it.]
A better place to start IMHO would be:
Can you share what the % results you're getting for each?
Based on the sample data, only 14% are deflected to achieve the ROI. At $12 p/ call, you do need to deflect a lot, and if you had 5M customers that would still be a huge return, but most of us aren't serving 5M customers.
One last thing.
Be sure to subtract out your operating costs from the ROI calc, unless it is included in the fully burden cost already -- often it isn't.
Do if I had 50K customers * 70% seeking * 40% resolved * 50% would actually call * $25 burden = $175K
$175K - (2 mods) - (Khoros cost) = I'm losing money
Looking forward to hearing from others. And if you're not using this, how are you measuring your return?
We have spoken about this lately and choose not to use visits when calculating ROI. Because it includes all the visits, no matter how long visit lasted. 1-5 seconds visits, it is impossible to get info you were looking for, read and understand.
We choose to use Google´s new metric, Engaged Sessions. If visitor stays least 10 seconds in one page it is calculated as a session.
Calculating ROI using this metric it is much closer actual ROI. But just like @JCTatRedSail mentioned, you have to take note platform investment, peoples salary on so on.
I like it! -- I wonder how the ROI calculator would play out for people once they use Engagement Sessions instead of visits.
I'm going to have to go back and look what that does to my #s. @TeroRe - Can you share the % difference from visits to Engaged that you saw?
The example I posted previously might just change a bit if you dropped 20% of the visits.
Do if I had 40K customers * 70% seeking * 40% resolved * 50% would actually call * $25 burden = $140K
$140K - (2 mods) - (Khoros cost) = I'm losing money
@TeroRe - Thank you for the comments and observations. Indeed, one of the worst things that can happen when applying this formula is that it does not pass a gut-check with anybody within the business. And what @cgrinton mentioned is the biggest red flag of all (i.e. - when the number of deflections due to the power of Community eclipses the number of Support Tickets that the business receives each year).
One quick thing:
* Visits does exclude bots and API calls
Another quick tip to help with credibility:
Visits, by and large, is a good metric to start with. However, if you really want to tighten that metric up, try doing this when you go into Community Analytics:
Community Analytics -> Content -> All Apps -> Forums
You will see that the Visits number grows smaller, the deeper you go into that Community Analytics breadcrumb. Alternatively, if you have a Tribal Knowledge Base (TKB), you can substitute the TKB for 'Forums' in my little tip above.
'Visits' is, IMO, the best metric to use for the formula, but 'best' does not mean 'without flaws'. A number of brand Communities can (perhaps inadvertently so) funnel lots of people to the Community homepage or other pages that should not be counted. There is an immediate bounce once the person realizes they are in place that they have no interest in, but a 'Visit' is still counted.
A brand Community's homepage can sometimes have a critical solution spelled out front-and-center (because the brand realizes how important it is for that solution to get a spotlight), or sometimes it does not. There are just so many varieties, exceptions, and outliers to the minutia of what makes for a useful 'Visit' or not, and where the most valuable content can be found.
Is the great content all across the Community? Is it solely relegated to the conversations threads / topics themselves? The best place to start is with the widest possible top-of-the-funnel Visits count and refine that funnel as necessary depending on the Community's structure and design. The way I refine it, depending on the Community layout, is by using that little narrowing technique in Community Analytics.
As an aside, thank you @TeroRe for pointing out Google's new Engaged Sessions metric (although it seems more like a KPI than an actual metric). I like it for a number of reasons, but I am concerned for one specific reason:
For a session to be considered engaged, a visitor has to do one or more of the following:
If you consider the User Experience when somebody does a Google search, gets a result that leads them directly to a Community page that has their solution, the person digests that solution, and then leaves (happy and satisfied); that whole experience does not "have two or more screen or page views", and thus does not constitute an 'Engaged Session'.
But hey, I still like the spirit of where they're going with this new way of measuring engagement.
@JakeR yeah there isnt perfect way to measure ROI 🙂 Engaged session is still better than visit, though, because it is better to share forward smaller numbers than bigger (compared to the "actual" ones).
@TeroRe - there is indeed, no perfect way to measure ROI for Community. On that we agree. I am not ready to agree yet that 'Engaged Sessions' is a better metric (KPI?) to use than Visits though. The present definition of 'Engaged Sessions' has a substantial Achilles Heel at this point due to the '2 or more pageviews' rule, IMO.
However, I think we are at in strong disagreement when it comes to another point you made above. You said:
...because it is better to share forward smaller numbers than bigger (compared to the "actual" ones)
No way, my friend! Let us instead offer perspective to our engaged audience and share multiple numbers. It sounds so cliche at this point, but there is a 'teaching moment' for those that are willing to listen to what we are trying to calculate. So long as we are transparent with the numbers we use and how we arrive at our formulas, then there is no 'better' or 'actual' for people to get hung up on.
If senior leadership disagrees with our numbers, then at the very least, it becomes a lively discussion. And during that discussion they must (perhaps grudgingly so) admit that there is *some value*.
But better yet, maybe you can convince them to 'put their money where their skeptical mouths are' and turn off the darned Community for a month! Then, in 30 days, they can subsequently walk the smoldering remains of what was once your brand's functioning Support Center. I can almost practically guarantee that they will both hang their heads in shame for having doubted you, and then cry tears of joy for your triumphant return.
....Alternatively, maybe they can just read my latest blog article and discover what happened when another brand did just that.
Yeah @JakeR , I am not going to argue with this with my bad English 🙂
But still, I am not willing to accept visits as a metric when calculating ROI and this is because there are all visits included. As said before, visit what lasts 0,5 seconds (or so) cant be included my numbers what I present to my stakeholders.
Btw, Google Engaged Session:
Note those or's there. "Or had 2 or more page views".
Access Denied on the blog post.
I like the discussion and the idea of having a discussion with senior leadership. In short what I took away.
Let me conclude by saying we all have different seats in the room, i.e. different views, and therefore there is (likely) no one right answer.
@JCTatRedSail - There is absolutely no right answer. Of that, I am sure. My new mantra these days is Perspective, perspective, perspective. Oh, and thank you for the broken-link call-out. That new blog article link should now be fixed.
BTW - @TeroRe - your English is great! Thank you for sharing your thoughts. I appreciate hearing what has been going on when trying to make the calculations and then tell the story.
Engaged session is session lasted least 10 seconds or more
had 2 or more page views.
Any link to the forrester-tsia research?
@davidsand - I wish that Khoros was licensed to provide you with either (or both) of the whitepapers created by Forrester Research or the Technology Services Industry Association (TSIA), but sadly, we are not.
However, if your organization has memberships or subscriptions to Forrester Research or the TSIA, here are the whitepapers to request or download:
Measuring the Success of Customer Communities
Membership, Activity Levels, Return on Investment, Technology Integration - by John Ragsdale (TSIA-TI-14-030 | November 18, 2014)
The ROI Of Online Customer Service Communities (June 30th, 2009)
A Total Economic Impact Analysis uncovers Big Benefits From Social Technologies - by Natalie L. Petouhoff, Ph.D.
with Chip Gliedman, William Band, and Andrew Magarie
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.