@davidsand - I wish that Khoros was licensed to provide you with either (or both) of the whitepapers created by Forrester Research or the Technology Services Industry Association (TSIA), but sadly, we are not.
However, if your organization has memberships or subscriptions to Forrester Research or the TSIA, here are the whitepapers to request or download:
Measuring the Success of Customer Communities Membership, Activity Levels, Return on Investment, Technology Integration - by John Ragsdale (TSIA-TI-14-030 | November 18, 2014)
The ROI Of Online Customer Service Communities (June 30th, 2009)
A Total Economic Impact Analysis uncovers Big Benefits From Social Technologies - by Natalie L. Petouhoff, Ph.D. with Chip Gliedman, William Band, and Andrew Magarie
... View more
@JCTatRedSail - There is absolutely no right answer. Of that, I am sure. My new mantra these days is Perspective, perspective, perspective. Oh, and thank you for the broken-link call-out. That new blog article link should now be fixed.
BTW - @TeroRe - your English is great! Thank you for sharing your thoughts. I appreciate hearing what has been going on when trying to make the calculations and then tell the story.
... View more
@TeroRe - there is indeed, no perfect way to measure ROI for Community. On that we agree. I am not ready to agree yet that 'Engaged Sessions' is a better metric (KPI?) to use than Visits though. The present definition of 'Engaged Sessions' has a substantial Achilles Heel at this point due to the '2 or more pageviews' rule, IMO.
However, I think we are at in strong disagreement when it comes to another point you made above. You said:
...because it is better to share forward smaller numbers than bigger (compared to the "actual" ones)
No way, my friend! Let us instead offer perspective to our engaged audience and share multiple numbers. It sounds so cliche at this point, but there is a 'teaching moment' for those that are willing to listen to what we are trying to calculate. So long as we are transparent with the numbers we use and how we arrive at our formulas, then there is no 'better' or 'actual' for people to get hung up on.
If senior leadership disagrees with our numbers, then at the very least, it becomes a lively discussion. And during that discussion they must (perhaps grudgingly so) admit that there is *some value*.
But better yet, maybe you can convince them to 'put their money where their skeptical mouths are' and turn off the darned Community for a month! Then, in 30 days, they can subsequently walk the smoldering remains of what was once your brand's functioning Support Center. I can almost practically guarantee that they will both hang their heads in shame for having doubted you, and then cry tears of joy for your triumphant return.
....Alternatively, maybe they can just read my latest blog article and discover what happened when another brand did just that.
... View more
Tell the truth, continually look for perspective, and you’ll be alright.
Although my lot in life was never to be a Presidential Historian, I nonetheless continue to collect little tidbits of knowledge about those who have occupied the Oval Office. Some US Presidents have been loved and some have been reviled, but all hold a place in history. In some cases, aphorisms from past Presidents continue to linger in American culture; more often than not to reinforce morality (e.g. - a young George Washington’s statement, “I cannot tell a lie” after he chopped down a cherry tree), but sometimes to just pithily articulate some kind of gestalt.
It is of the latter variety of aphorisms described above which comes to mind when I consider the anonymous visitors to an Online Community. Our 37th President of the United States, Richard Nixon, described those who elected him as the ‘Silent Majority’. An appropriate term, if ever there was one, to describe the vitally important audience that large brand communities accommodate. Indeed, the ‘Silent Majority’ of passive viewers that read Online Community content to solve their own Support problems and to inform their purchase decisions become a critical component to discovering the Community’s business value.
And yet, questions and doubts still arise.
If we do not know who these people are, what they learned, or what they bought, then how can we possibly attribute value to their Visits? For years I have just taken it on faith that (some, not all) anonymous passive viewers of an Online Community benefited from their digestion of the peer-to-peer content contained within. However, my blind faith in the power of the ‘silent majority’ was recently tested by a business stakeholder during a large project I had been working on for her brand’s Community. She wanted to see my Contact Deflection story from a much more narrow-angle. Instead of using Visits (or even Unique Visitors) as the starting metric to evaluate Contact Deflection, her suggestion was to use only those members of the Community that were known to us (i.e. - ‘Authenticated Users’), and furthermore, she asked that any Accepted Solutions that were viewed to help avoid contact to Support be authored exclusively by another peer or customer and not an employee (contractor or otherwise) of the brand.
“Oh geez”, I thought to myself, “this simply will not do”. I understood the spirit of her request. The savings story is great! Both the Community and the brand are very large and extremely successful. And although there was every indication that she was a believer in the returns generated by the Community, she still had to internally explain the story to the brand’s senior leadership and perhaps an even tougher and more skeptical crowd, the brand’s Finance Department. So she was well within her rights to request a recalculation of the Savings story with those tighter parameters.
But under those more restrictive conditions and numbers, I knew the returns would be substantially lower, and even more importantly, I knew that those numbers would not be telling the truth. I would have to tell this stakeholder that she was wrong! So long as the Community content is able to be searched for, read, and potentially used by the ‘Silent Majority’ to stave off contacts to Support, then there is value being derived via cost savings. Now I found myself in little George Washington’s shoes. I could not tell a lie!
But I could tell a story. So that is what I did.
The story started with doing what she asked.
I ran the calculations using only authenticated Visits that viewed Solutions only authored by other customers of the brand and not employees or contractors. As expected, the Savings story went from being impressive to anemic. No surprise there. But then I backed up a little bit and explored what others have said over the last 25+ years about anonymous/passive Visits to websites. I figured there had to be some substantive industry research that could be boiled down to some kind of philosophical missive that I could use to gently convey that the ‘Silent Majority’ can not and will not be ignored.
As usual, I was right. But as a great man once said to me during my 6-month review at this company (nearly 15 years ago!):
“It is not Jake’s job to correct the customer”.
So forget the corrections. Consider the following instead:
The single largest audience who interact with your business is anonymous. Typically 97% or 98% of website visitors are anonymous. For most companies, only 2%-3% of visitors end up sharing their email addresses or otherwise identify themselves.
57%-70% of the purchase decision is complete before a potential customer is known to the business. These anonymous site visitors may be far in the purchase process and by the time a sales lead is generated the decision may already have been made.
And at the risk of putting too fine a point on the power of the ‘Silent Majority’, one of the greatest pieces of Online Community research (and certainly one that involved a tremendous amount of courage) dared to examine what would happen if anonymous Visitors could no longer easily find an already successful Online Community. A brand de-indexed their Online Community from Search Engine Optimization (SEO) and the results were nothing short of stunning. Richard Millington from FeverBee documented this story in the Harvard Business Review :
"The locals still know it’s there and can visit, but the visitors don’t and won’t. When we did this, the number of visitors plunged by an average of 83%. While some found the community through other channels (the website), most simply filed support tickets or called customer support instead. The number of tickets (calls/online ticket forms) to other support channels increased by 58%."
I may not agree with Millington on a number of Community-related things, but ladies and gentlemen, this article more than any other restored my faith in the power of the ‘Silent Majority’. Props to Millington and the brand he worked with for doing what I have boldly (but gently) suggested to the occasional non-believing customers that I have encountered during the last 7 years that I have been doing Online Community ROI as a full-time job.
So now that my story about the ‘Silent Majority’ started to develop some meat on the bone, I did what any and all of us do in 2021, I made a slide deck out of it.
Terrible, right? Nah. It actually kind of worked out.
At first, I showed it to the Community’s dedicated strategist (who is eager to please his boss and his boss’s boss) and he said, “So the true number is probably somewhere in between the original number that you shared and then this newly refined conservative number, right?” I responded with an, “absolutely not”. “The smaller number is the result of an exploration.”
I could hear his brow furrowing. This guy, good old Uncle Jake, is going to tell the business stakeholder that she is wrong. Nah. I am just going to illustrate what the industry has said, how the industry looks at such things, and then share an anecdote about what another brand did to test (after a fashion) her way of measuring the Contact Deflection Savings….and then I’m going to tell her that she is wrong.
Well, although I do admit to dusting off my resume before going into the meeting to share my presentation, I am happy to report that the business stakeholder had a reaction that really impressed me.
First, she thanked me for taking the time to slice and dice all the data to arrive at the much more conservative number. I liked that. Indeed, a critical part of the story that I am sparing all of you is how to arrive at the ‘Resolution Rate’ and ‘Solution Views’ generated by only non-employees of the brand. Those particular calculations involved some gymnastics.
But it is what she said next that really won me over (and I already liked her, btw):
It is good to have another number as a reference point
Measuring the Community in such a manner may not be ‘correct’, per se, but it can offer perspective
Yes! I cannot agree with those sentiments enough. In this day and age where everybody is so darned certain about everything that they believe in, perspective is invaluable.
A lot of people don’t know this, but...
President Richard Nixon was both a huge fan of American football (Washington D.C.'s team even allowed him to ‘call a play’ during their appearance in a Super Bowl one year), and he was also an avid bowler. If you watch the film, The Big Lebowski, you can see that the Jeffrey Lebowski character has a poster in his apartment of President Richard Nixon bowling. In all likelihood, the photograph was taken in the bowling lane that Nixon had installed inside the White House in 1969.
An Online Community has a ‘Silent Majority’ coursing through its veins. Ignore it at your own peril. Richard Nixon occupied the highest elected office in American politics because of the ‘Silent Majority’ that voted for him, but his reign ignominiously ended due to the proverbial gutterballs he threw.
So remember that if you tell the truth and continually look for perspective, you’ll be alright.
... View more
@TeroRe - Thank you for the comments and observations. Indeed, one of the worst things that can happen when applying this formula is that it does not pass a gut-check with anybody within the business. And what @cgrinton mentioned is the biggest red flag of all (i.e. - when the number of deflections due to the power of Community eclipses the number of Support Tickets that the business receives each year).
One quick thing:
* Visits does exclude bots and API calls
Another quick tip to help with credibility:
Visits, by and large, is a good metric to start with. However, if you really want to tighten that metric up, try doing this when you go into Community Analytics:
Community Analytics -> Content -> All Apps -> Forums
You will see that the Visits number grows smaller, the deeper you go into that Community Analytics breadcrumb. Alternatively, if you have a Tribal Knowledge Base (TKB), you can substitute the TKB for 'Forums' in my little tip above.
'Visits' is, IMO, the best metric to use for the formula, but 'best' does not mean 'without flaws'. A number of brand Communities can (perhaps inadvertently so) funnel lots of people to the Community homepage or other pages that should not be counted. There is an immediate bounce once the person realizes they are in place that they have no interest in, but a 'Visit' is still counted.
A brand Community's homepage can sometimes have a critical solution spelled out front-and-center (because the brand realizes how important it is for that solution to get a spotlight), or sometimes it does not. There are just so many varieties, exceptions, and outliers to the minutia of what makes for a useful 'Visit' or not, and where the most valuable content can be found.
Is the great content all across the Community? Is it solely relegated to the conversations threads / topics themselves? The best place to start is with the widest possible top-of-the-funnel Visits count and refine that funnel as necessary depending on the Community's structure and design. The way I refine it, depending on the Community layout, is by using that little narrowing technique in Community Analytics.
As an aside, thank you @TeroRe for pointing out Google's new Engaged Sessions metric (although it seems more like a KPI than an actual metric). I like it for a number of reasons, but I am concerned for one specific reason:
For a session to be considered engaged, a visitor has to do one or more of the following:
Engage actively on your website or mobile app for over 10 seconds.
Have two or more screen or page views
Fire a conversion event
If you consider the User Experience when somebody does a Google search, gets a result that leads them directly to a Community page that has their solution, the person digests that solution, and then leaves (happy and satisfied); that whole experience does not "have two or more screen or page views", and thus does not constitute an 'Engaged Session'.
But hey, I still like the spirit of where they're going with this new way of measuring engagement.
... View more