Margaret Francis is VP of Product, where she leads product development strategy and delivery for Lithium Social Media Monitoring.
We've fielded a lot of great user questions since launch, and the number one area we've fielded them in is sentiment. This may be too much information for some of you, but if you really want the details, read on!
The sentiment feature in the Lithium Social Media Monitoring/Scout Labs application is the ability for the machine to judge whether or not the author of a story is expressing a positive or negative attitude towards a specific word or phrase. For those companies with only a few posts per day that they can judge for themselves, this feature is a nice to have. But for brand and product marketers looking at a significant volume of posts, this feature is essential to understanding changes in consumer opinion.
So how do we do it? How accurate is it? And how should you use it?
How we do it
Our sentiment is "entity specific". What some products do when they produce "machine generated sentiment" is that they count happy words vs. sad words in a news article. The "tone" of the article is shown by the happy word count. Consider "I love baseball. My happiest memories in life are from sitting in the bleachers at Fenway. It's the greatest game on earth. But guys like Bonds and A-Rod are bringing it down." Despite the high "happy" word count, this does not express a positive opinion about Barry Bonds or Alex Rodriguez.
In the Lithium Social Media Monitoring application, we don't count happy words. We evaluate sentiment for each particular word or phrase you search for. We can tell that the sentiment for baseball is positive but negative for Bonds and Rodriguez. This is done via part of speech tagging: parsing the underlying semantic structure of a sentence and determining which emotion words apply to the key word. Emotion words come from dictionaries of standard English words and have been augmented with phrases and slang to better map to the world of social media. So Scout Labs' sentiment is entity-specific, which is very important.
Lithium SMM sentiment can be changed by users. We use confidence intervals to decide whether something is positive or negative, but if we get it wrong (more on that below), you can change the score, immediately updating that item for yourself and the rest of your team. Charts and graphs update immediately as well. And the really cool part is that every time a user changes a sentiment value, that item becomes a labeled piece of data that we can use to abstract out additional rules and add words and phrases for our dictionary. So our ability to detect sentiment just gets better over time.
Lithium SMM can "backfill" sentiment data for the previous 3 months in less than a day. We have 6 months of live data in our app for our users . We can go backward and score all the posts from the last 3 months in less than 24 hours. So you will have complete sentiment trend for everything going forward and going backward within less than a day from creating a search (All other graphs -- buzz, share of voice, etc.) are real time and have no lag time at all).
Does it work?
Yes. We have done extensive human vs. machine testing and it's accurate in the 70-80% range, meaning our algorithm agrees with humans' scores 70-80% of the time. This is only slightly less than humans agree with each other. Some other insights and findings from our testing:
So the Sentiment feature produces a pretty good guess, about what you'd get using if you got a half dozen ratings from Mechanical Turk and chose the rating the most humans agreed on. (See this useful paper from the Dolores Labs blog about how to use Mechanical Turk to get reliable human judgments). And our best guess plus your teams' efforts to quickly change the things we miss or get wrong means really high accuracy levels for you and your team with a minimum amount of work and expense.
How you should use the sentiment feature:
We have heard over and over again from our users that an affordable, reliable way to assess sentiment, with user override built in, is critical to getting insight into social media, so we continue to work on this feature. We hope you'll let us know how you want it to evolve in the future.
We've already got a slew of new feature requests to work on, including more metrics, visualizations, and customizations. Get your ideas into the mix at the Social Media Monitoring product page at Lithium.com
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.