Margaret Francis is VP of Product, where she leads product development strategy and delivery for Lithium Social Media Monitoring.
She is a regular blogger for Lithium and in the Lithosphere you'll see her as MargaretF. You can follow Margaret on Twitter at @margaretfrancis
We've fielded a lot of great user questions since launch, and the number one area we've fielded them in is sentiment. This may be too much information for some of you, but if you really want the details, read on!
The sentiment feature in the Lithium Social Media Monitoring/Scout Labs application is the ability for the machine to judge whether or not the author of a story is expressing a positive or negative attitude towards a specific word or phrase. For those companies with only a few posts per day that they can judge for themselves, this feature is a nice to have. But for brand and product marketers looking at a significant volume of posts, this feature is essential to understanding changes in consumer opinion.
So how do we do it? How accurate is it? And how should you use it?
How we do it Our sentiment is "entity specific". What some products do when they produce "machine generated sentiment" is that they count happy words vs. sad words in a news article. The "tone" of the article is shown by the happy word count. Consider "I love baseball. My happiest memories in life are from sitting in the bleachers at Fenway. It's the greatest game on earth. But guys like Bonds and A-Rod are bringing it down." Despite the high "happy" word count, this does not express a positive opinion about Barry Bonds or Alex Rodriguez.
In the Lithium Social Media Monitoring application, we don't count happy words. We evaluate sentiment for each particular word or phrase you search for. We can tell that the sentiment for baseball is positive but negative for Bonds and Rodriguez. This is done via part of speech tagging: parsing the underlying semantic structure of a sentence and determining which emotion words apply to the key word. Emotion words come from dictionaries of standard English words and have been augmented with phrases and slang to better map to the world of social media. So Scout Labs' sentiment is entity-specific, which is very important.
Lithium SMM sentiment can be changed by users. We use confidence intervals to decide whether something is positive or negative, but if we get it wrong (more on that below), you can change the score, immediately updating that item for yourself and the rest of your team. Charts and graphs update immediately as well. And the really cool part is that every time a user changes a sentiment value, that item becomes a labeled piece of data that we can use to abstract out additional rules and add words and phrases for our dictionary. So our ability to detect sentiment just gets better over time.
Lithium SMM can "backfill" sentiment data for the previous 3 months in less than a day. We have 6 months of live data in our app for our users . We can go backward and score all the posts from the last 3 months in less than 24 hours. So you will have complete sentiment trend for everything going forward and going backward within less than a day from creating a search (All other graphs -- buzz, share of voice, etc.) are real time and have no lag time at all).
Does it work? Yes. We have done extensive human vs. machine testing and it's accurate in the 70-80% range, meaning our algorithm agrees with humans' scores 70-80% of the time. This is only slightly less than humans agree with each other. Some other insights and findings from our testing:
College educated people with business experience agree on the sentiment ratings for a blog post about 85% of the time. Using less qualified people, such as you might find in a random Mechanical Turk experiment, produces lower rates of agreement. We were surprised that we couldn't get that rate higher. Some of the discrepancy stems from the human tendency to equate negative opinions and negative information: "I hate Coke" is a negative opinion; "Merrill Lynch just downgraded Coca-Cola" is negative information.
The Lithium SMM sentiment feature agrees with college educated people about 75% of the time. We try to pad that a little by being conservative about what we call positive or negative -- we call things neutral if they're borderline.
The Lithium SMM sentiment feature sucks at detecting irony and sarcasm. Posts that are heavy on the irony often end up classed as "neutral" because the machine can't even guess. Consider "Another winner from the almighty Microsoft." That's a tough one.
Machines don't understand business context. Perhaps you work for Apple and every mention of an unlocked iPhone is negative because people shouldn't unlock their iPhones. An algorithm that uses grammar and vocabulary based rules cannot classify this post as negative about iPhone: "I love my iphone. My boyfriend unlocked it for me last night."
So the Sentiment feature produces a pretty good guess, about what you'd get using if you got a half dozen ratings from Mechanical Turk and chose the rating the most humans agreed on. (See this useful paper from the Dolores Labs blog about how to use Mechanical Turk to get reliable human judgments). And our best guess plus your teams' efforts to quickly change the things we miss or get wrong means really high accuracy levels for you and your team with a minimum amount of work and expense.
How you should use the sentiment feature:
To find the top positive and negative posts. Click on "Sentiment" and filter for positive or negative posts. You'll get immediate insight into some forceful opinions about what is wrong -- or what is right -- about the product or brand you are searching for.
As a starting point for your own sentiment analysis. Any user can change the sentiment rating for any post. If you work for Apple and you want all those unlocked iPhone posts marked "negative," you can do that. Just click on the sentiment icon and make the change. These changes will carry through to all graph data, so you can create accurate data sets to view in the application or export data for. We use your rating changes as machine learning inputs, but your specific ratings are proprietary and confidential to your workspace.
To get insight into consumer opinion via alerts. When you set up a daily, weekly or monthly alert for a search, you'll get buzz, top news, new words, recent tweets, and the top positive and negative posts pushed to your inbox via a text email. It's a great way to stay informed and know when to invest more attention.
To compare sentiment between brands or products. Do consumers like Symantec or Norton? The Lebron 6 or the KD1? Embarq or Comcast? Sentiment Trend graphs can help you see trends, spikes, and make comparisons.
We have heard over and over again from our users that an affordable, reliable way to assess sentiment, with user override built in, is critical to getting insight into social media, so we continue to work on this feature. We hope you'll let us know how you want it to evolve in the future.
We've already got a slew of new feature requests to work on, including more metrics, visualizations, and customizations. Get your ideas into the mix at the Social Media Monitoring product page at Lithium.com
... View more
Jenny, the CEO of Scout Labs, and I first met while working at an agency (er- it was called marchFIRST), and it was from an agency (Razorfish) that Jenny lured me when she started Scout Labs.
So many of the things I wanted to do for my clients I could not efficiently execute, due to lack of tools in market. As a former agency strategist, I have a special affinity for Scout Labs’ agency customers and the use cases they have for our platform. Here are some of the reasons agencies give me for needing a social listening platform:
We need to know what is happening with the client’s brand to be effective strategists. Agencies’ stock in trade, especially digital agencies and high end marketing agencies, is that they are better placed than their clients to anticipate trends and craft breakthrough marketing strategies because they have the benefit of cross-company and cross-brand experience. It isn’t possible to maintain that positioning if the account team isn’t up to speed on what’s happening with the client’s brand on Twitter and YouTube, much less their competitors’ brands.
We are monitoring a particular campaign. Agencies are often responsible for the design, execution and tracking of marketing campaigns. As the power of social media to reach consumers grows, agencies are more and more interested in tracking campaign performance in this channel- or even in defining campaign success by social media criteria. After all, if your big campaign idea is to let a consumer shoot your Superbowl ad, you might want to know how many people saw it, if the people who saw it liked it, and if so, why. Forwarding around a couple of Google Alerts is just not going to cut it.
We are engaged in a social media listening program on behalf of a client. More and more of the agencies I talk to- PR, traditional, digital, media only, creative, branding, SEM, etc- are engaged in retained listening programs on behalf of their clients. These range from pay per tweet to ad hoc studies delivered in PPT for $5-50K to inclusion of social media data in traditional brand health measures for $20K+ to outsourcing response to Twitter and blog mentions and so forth for $100K+ month.
In almost every case, the agencies that we talk to decide that they need to standardize on a single platform that can provide the backbone of their monitoring and measurement program. They may do some additional work to add proprietary or other data, or to produce custom analyses, but there is usually a core content aggregation, analysis, and metrics platform that the agency team can build on.
We need a common digital dashboard/ application to share across the company and the agency. A really good agency is often deeply embedded in a client organization, more like an extra arm of the clients’ organization than a “vendor” to it. It’s essential for the holistic team to be able to share a single app for content, analysis, and response.
We needed a tool for our own internal research and business development. Sometimes the agency just need to look really smart in a pitch, or to engage in a little research that the client isn’t exactly paying for- yet. Sometimes the agency is in the business of market research, creative development, or even product development- all of which are morphing to account for the rise of social media.
Of course none of those reasons is totally unique to Scout Labs, nor is Scout Labs the only solution in market. I see agencies using everything from free tools like Google Alerts to other agencies to help them address these use cases. Here are some of the reasons I see agencies embracing Scout Labs:
Easy to use for the whole team. We get consistent feedback that Scout Labs is the easiest, most intuitive, user friendly application out there, for newbies and veterans. If you’re trying to drive social media awareness throughout a Fortune 1000, you need an application that the masses can use. A lone specialist does not organizational change create.
Ad hoc search. None of this advance commitment to a lone word or phrase- “Neutrogena”- and only that word, for which data begins accumulating when you contract for it. Agencies not only need data now, they need some trend on it- and who wants to wait 6 months for 6 months back data? And what if that initial scan turns up issues around self-tanning or an ingredient like pomegranate? Agencies need to see trends and data on those, too.
Real time market intelligence. People are flabbergasted at how fast it is get data across across all social media channels. Especially people who have used other platforms that have a lag to their data collection. Being able to not just retrieve the data fast, but get speedy analysis of it- frequent words, back sentiment, 6 month graphs- likewise gratifies the time starved agency strategist.
Saved items, graphs, and exports. So your deliverable needs to be on the agency PPT template/ HTML email/ Flash presentation. We understand. Scout Labs make it easy to do things like export graph data so you can make the line in the graph your client’s exact brand color, or add your client’s logo and your agency’s logo to the application header.
ROI! Having relatively low price points and plans with unlimited users makes this the biggest no-brainer purchase of the year. Go check out prices and see. With Scout Labs you get the productivity boost of a dozen interns for the cost of less than one, just on the content aggregation side- before you even get into metrics and insight from our NLP driven features that no intern is going to provide.
This isn’t a complete list of either use cases or reasons to choose Scout Labs, but it’s getting to be a long post. I’m always open to hearing from agencies about what they need to be more effective on behalf of their clients. Please share it- you might get pleasantly surprised by what we have in development for you.
Previously posted on scoutlabs.com/blog on Feb 23, 2010
... View more
Here at Scout Labs, we know there are some big corporations out there that are still standardized on IE6. We work with some of them, which is why to date we’ve supported IE6.
We didn’t want to cut off IE6 users or subject them to a substandard application experience. We know most of them are stuck on IE6 because of IT admins who overinvested in proprietary apps that ONLY work on IE6, have NEVER been updated and never will be, are purposefully holding them back.
This is a decision point that old skool internet companies like Yahoo and Web 2.0 companies like Facebook and bellwethers in the SAAS space like Salesforce have already gone past. Hell, 37signals phased out IE6 support in October of 2008, which is the Internet equivalent of the Nixon era. Even Europe is following suit. But for those of you still using IE6, here are some options:
If you have the necessary permissions on your computer, install and use any browser more modern than Internet Explorer 6. You can download Firefox or Chrome for free. As of Feb 2010 Scout Labs officially supports IE7, IE8, Firefox, Safari and Chrome.
Upgrade to Internet Explorer 7 or 8. Even IE 7 is faster, more reliable, and better supported by Microsoft than IE 6. Though we’d pick IE8 over IE7 any day.
If you don’t have the necessary permissions on your computer, find the person who does. If they wont help you, send them this link: http://www.ie6nomore.com/ Or this one: http://www.stoplivinginthepast.com/ Or this one, from a Microsoft employee: http://www.hanselman.com/blog/IE6WarningStopLivingInThePastGetOffOfIE6.aspx Or….you get the picture.
If you are denied permission to upgrade past IE6, go find a company executive who believes in the future. This is a great way for some up-and-comer to make everyone in the company more productive via upgraded internet tools and experiences, and themselves wildly popular (with all non-IT personnel) in the process.
The cool part is, now we get to support Chrome- which is a fun browser, and great news for users of Microsoft OS products of a more recent vintage. And for all you network admins who just can’t seem to get everyone off IE6 and Win2000? Better hurry up, before every SAAS app your workforce relies on becomes standard equipment on the corporate smartphone- and no one gives a hoot about that big old box with a ten year old browser on it, anyway.
Previously posted on scoutlabs.com/blog on Jan 26, 2010
... View more
Here at Scout Labs we figured the best holiday gift we could give our users would be some new features. So here’s a quick recap of recently released features, including some we deployed just last night:
New search OVERVIEW page. Everyone wanted a single screen dashboard that would aggregate the most telling graphs, the leading indicators, and most important social media content. Welcome to the new OVERVIEW page. Instead of clicking from tab to tab within your search, you can now get a snapshot of buzz volume, sentiment trend and top stories from Twitter, Blogs, and everywhere else on a single page.
Interactive graphs. We were as disappointed as all of you when we had to pull back from our earlier interactive graphs implementation, which used Flash technologies not universally supported by corporate sanctioned browsers, and rely on an image based solution that was not clickable. But now interactive graphs are back, and they’re bigger and better than before. You can hover over a particular day to see the counts; click into spikes to read what happened; and of course still customize your date range within the last 6 months or export the data in a .csv.
One thing we did lose in the transition was the ability to export graphs as a .png. We’ll eventually bring it back for you, Steve Majewski, but in the meantime, take a screenshot- there new graphs are much better looking than their PNG predecessors!
Ability to sign up your colleagues up to receive email alerts. Many of you asked for this feature because you wanted us to send your favorite email alerts directly to other team members, instead of having to forward them yourselves. Now, instead of forwarding Scout Labs alerts, you can simply CC other users on your alerts. And opting out is as simple as clicking on a link within the email. So now you can sign your teammates up for alerts for your brand, a competitor’s campaign, or whatever else you might be tracking.
Links to source included in exports. Now the number of links to each source is included in the export files. Mike Arauz and Spencer Waldron, that one was for you guys.
Previously posted on scoutlabs.com/blog on Dec 21, 2009
... View more
Last Thursday I spoke on a panel at BlogWorldExpo (#bwe09) entitled “Social Secure-ity: Managing Your Brand’s Online Reputation.” Audience questions drove the discussion; I managed to jot down some of the audience questions/ panelist responses and have compiled them below.
In my role at Scout Labs, I have seen a wide range of corporate responses to these questions, as I’m sure had the other panel participants- Connie Bensen of Techrigy, Amber Naslund http://altitudebranding.com/ of Radian6, Melyssa Plunkett-Gomez of Crimson Hexagon, and Aubrey Podolsky of Sysomos.
Thinking back on it later in the day, a lot of this advice just isn’t useful with some resources and corporate commitment to social media- and from the tone of many of the questions, resources and commitment are still an issue. There’s no doubt that social media is here to stay whether or not companies want to “formulate a response.” The real question is which companies are going to recognize that better understanding their customers through social media is a source of competitive advantage- whether they decide to turn Twitter into a customer service channel or not.
From a travel industry representative from Las Vegas: How should we deal with negative criticism? How should we respond to it?
There will always be negative criticism. Develop a framework for what you need to respond to from a business perspective, and try not the take the rest too personally.
There are trolls and there are people with genuine issues. You can’t please the trolls, so don’t try.
Your social media presence can and should be about more than damage control to the brand.
Let the community respond on your behalf.
Get ahead of the negative criticism- change the business so that consumers aren’t complaining!
David Spinks, a community manager, asked about the importance of responding to content on sidewiki:
The sources of feedback are multiplying and will continue to do so. Very few brand manager have the time to respond to every single last comment anyone has ever made on any platform. If Sidewiki gets more traction in the market, vendors like the ones on the panel will eventually help marketers figure out how to track and respond to it. Right now it’s a small blip on a crowded radar screen.
A gentleman who works for a hedge fund asked what to when everyone hates you:
Having a social media presence can help humanize the brand or the industry, and demystify what it is that hedge funds do for the economy
Whose opinion are you trying to influence, and why? Make sure you know and have the resources to follow through before you set your social media strategy- there are some inconsistent, useless attempts at building a community out there, and they’re not helping the brands they represent
If you do something for your community that does not directly serve your own interest- for instance publishing free analyst repots- it build brand goodwill
If they care enough to hate you, they may care enough to love you. There are things the company does that would help them to love you. Make people aware of them.
Kat French asked for thoughts on what to do about clients who only focus on the negative:
Reframe the conversation through competitive analyses- what do people love or hate about your competitors? It’s myopic to only focus on the negative posts about your own brand
Another way to reframe is to focus on the positive- what is it that customers like about the brand? What strengths can the brand build on? This is the long term strategic response to negative feedback- not just a crisis communications plan
Tracy Schmidt from ChicagoNow asked a question about policies for employee participation in social media:
Have a policy in place helps guide employee behavior- and know that there will still be problems. Be prepared to deal with them as decisively as any other personnel problem
Employees are trusted to have email addresses and phones- they are already representing the company. Employee behavior is a hiring and training issue, not a social media issue. If an employee can’t be trusted, they can’t be trusted
Employee trust is an enormous issue for every organization. Social media is forcing massive organizational changes on companies, ones that are really challenging for managers and employees. This is a big change for everyone involved and training is crucial.
Last was a great question on how to deal with franchises, where much feedback comes about individual franchisees that impacts the overall brand:
Monitoring customer satisfaction/ ability to meet corporate requirements is already a part of franchising. Extending that to social media will become a corporate function.
And Lacey Kemp from Seattle, I got your name but not your question- ask it again here and I’ll do my best to answer you!
I found it encouraging that there were a fair number of B2B marketers at the session- leveraging social media is just as crucial for them as it is for the B2C brands, if not more so, and they rarely get the same level of media love.
Some other memorable sessions at BlogWorldExpo were the ROI smackdown- very smart thinking and good examples from Deb Micek, Rob Kay, Beth Harte, and Stephanie Agresta- and the Real Time Web session from the ever thoughtful Louis Grey. The sessions were all taped and will theoretically available online at the conference site but I don’t see the links up yet. Much thanks to Jason Falls for organizing.
Previously posted on scoutlabs.com/blog on Oct 19, 2009
... View more