Posts about surveys :: Look deeper before you believe
Surveys are fun to read and fun to write about. Just as in traditional media, too often when the surveys are presented in blogs, a lot is left out. There is often little, if any, discussion of the background, possible motives for the survey, methodolgy and how the results are reported. This echo effect – just publishing the results – may lead to a misrepresentation of the true character of the survey. It could be taken as endorsement. It may also lead people to believe something is true when that truth has yet to be proven.
For instance, Steve Rubel is writing about a corporate blogging study from a competitor. I’ve got a few questions, so, I ask:
OK, Steve. Your cut-n-paste skills are up to par. But, what do you think of the conclusions of the survey? You post it, but do not discuss it. I’m often curious when people post things without any thoughts shared. You know, is there a motive involved? Or, is this just sharing little bits of information? Either way, with his post, I don’t know the answer.
My questions. First, do we have all the information we need to judge whether or not this is a valid survey? Second, as David Maister suggests, should “(these) numbers … be shouted by every PR firm from the rooftops”?
If so, why? I followed the single link on the post Steve offered. I followed links found there and read the posts and claims. There is nothing there, really. Well – except for some claims tied to broad statements citing vague (undefined) percentages. They do not reveal what those percentages represent. No identity of the number of respondents surveyed, the larger potential respondent pool, the types of corporations surveyed, etc. are revealed. So, what – if anything – can we derive from this survey?
Ann Handley wraps herself around “63 percent said…” while not knowing what that 63 percent represents. Is it 63 percent of 30 respondents or 300? Wouldn’t that make a difference as to how much authority I can apply to the survey results? What if it was just 63% of pet food corporations? There are a lot of corporations out there. What business are those corporations in – tech, consumer goods, services? We don’t know, do we.
Please, I beg you. People, do the right thing. The sane thing. Put down the koolaid, folks, until you know what is in it.
What Steve posted, it turns out, is essentially a pitch for a free webinar:
Cymfony and Porter Novelli Announce Free Webinar Highlighting Findings from Corporate Blog Survey to Help Companies Seeking to Capitalize on the Blog Channel.
The full details of the research (which might answer some of those questions above) are apparently going to be provided this week.
On Tuesday, July 25, 2006, at 12:00 p.m. EDT, Cymfony and Porter Novelli will host a free webinar entitled “Corporate Blog Learnings — The Discovery Age.” … A full report on the survey findings will also be available on the July 25 webinar, as well as on Cymfony’s and Porter Novelli’s Web sites.
Now, the examples above come from the PR / Marketing area. I think we’ve all seen examples of survey research reported in more traditional media and the details were brushed over, or entirely left out. This is bad practice, no matter where it occurs.
I’m thinking it would be wise to wait for the details to determine whether or not anything should be embraced as noteworthy and shouted from any rooftops. To their credit, Cymfony/Porter Novelli note:
All efforts were made to reach a broad cross-section of corporate bloggers; however, the corporate blog environment is less mature than the consumer blog environment, creating a challenge in finding corporate blog owners and getting them to answer questions about ROI, resources and strategy.
Now, the emphasis in that last quote is mine. Quite telling, isn’t it? This is a pitch to encourage participation in an immature market. See, now we can derive some idea of the possible motivation behind such research. They want to encourage more participation. They, perhaps, want to sell services to help clients accomplish this task.
So, for my students, I will suggest that all you read online (and everywhere, for that matter) should be investigated a bit before wrapping yourself around it as earth shattering news – or fact. Also, if you take the time to do that little bit of further research on your own part – your blog posts will likely provide a better service to your readers, and a worthwhile learning experience for yourself.
I’d rather read a news story or blog post about such survey research with some form of commentary included by the blogger. Hey, even if I don’t agree with their conclusions, it provides much more food for thought than a blanket cut-n-paste. It is even more important, I believe, for those claiming to be the “thought leaders” to actually include – well – some thought in their posts. What a concept.
OK, so i get over-enthusiastic sometimes!
OK, so I’m red with embarrassment – I used to teach statistics and should know better than to react without investigation. Whether or not the conclusions are or are not valid, your point is a crucial one.’
Thanks for helping us all remember to keep our perspective!
Thank you, David. No harm intended. I just look for posts online to share with the students. Then I try to write it in manner that will get their attention. Hey, I’ve been over-enthusiastic before, too.
Thanks for coming by and commenting. All the best.
I swear, Robert, this is the exact post I had in my head but hadn’t worked up the nerve to write. Good stuff. It’s important for someone to be providing this sort of context and perspective on issues like this.
Thanks, Chris. The treatment of surveys in the press, and in blogs, is sort of a pet peeve of mine. Too often we see numbers presented and – if only implied – the impression seems to be that we should take them as fact.
Apparently I’ve lost some of the filters that previously told me not to post these ideas, but I’m not sure that’s a bad thing. My intention is not to do harm, but rather to remind – especially my students – to be skeptical and dig further before buying into anything they may read or hear.
This just struck me as a good example to use.
This is another example of an issue I’ve written a lot about … consultants and consultancy companies touting “the power of blogs” because they sell blogging services.
In the rush to find evidence that will compel others to hire them, they create surveys that are in reality meaningless … but unfortunately, appeal to people who find comfort in numbers.
I hear ya, John. That’s been happening for quite some time now.
I received an email pitch for PR related online services recently. It touted their own survey. The survey said they were #1 in all sorts of areas.
What they did not share is how the respondents for the survey (the survey which claimed they were the #1 provider of those services) came from their own customer list. Yet, their pitch used phrases like “all PR practitioners” and such broad declarations.
I only learned of the true respondent pool when I wrote to the CEO of the company and asked. Something about the email pitch just didn’t seem right. Know what the CEO asked me before providing that information? If he/she provided the methodology – I could not reveal it.
True story. I have the emails. But, I promised – so I didn’t write about it. Did share it with my students, though (sans the company name).
Ah, methodology and transparency. If only some PR practitioners knew what they were – we’d all be better off.
Thank you, Robert! This is great advice for students. I try to limit immediate comments to posts or elaborate on my own blog until I have done my homework. I will have up to 50 potential blogs posts on my list at any given time because of the need for deeper thought and research.
Thanks, Lauren. I do the same thing with drafts. In fact, I have to go back in periodically and clear them out because I probably write 5 drafts for every 1 post I make.
Drafts / rewrites are important. Some things, upon further review, should probably not be posted. Sometimes I write something and don’t like it. There are many reasons to create drafts, rewrite them (then rewrite again) … it is all about creating readable and worthwhile writing for your audiences.
Sort of going off topic here, but … My favorite analogy for rewrites is Martin Luther King’s “I Have A Dream” speech. Somehow, I think the line “I have a dream” started out something like “I have this dream of a day when…” or something like that.
Wish I could find the article I read, a long time ago, about the writing of that speech. It was quite inspiring and enjoyable to read. Anyone know of such an article? I think it contained interviews with those familiar with the writing of the speech.
How to Lie with Statistics: Dissecting the Latest
First, let me say that I love research. But lately it seems as if there is a lot of it out there and few are offering critical thought around how to interpret all of these results. Here is my roundup of recent surveys with some critical commentary:
Robert:
Fantastic post. Toby Bloomberg and I just came out of an episode with JupiterKagan (JupiterResearch) where we were asking some basic questions about the survey and were initialy rebuffed by the company. Our experiences offer a great case study in why it is important to look beyond the numbers so that surveys can be placed into context. At the risk of being filtered by your spam eater, I’ll provide some key links here:
-JupiterResearch Passes Around The Kool-Aid (Diva Marketing Blog)
-Corporate Blogging and JupiterResearch: Survey Raises More Questions Than It Answers (HealthCareVox)
-The JupiterKagan Corporate Blogging Survey: Four Lessons I Learned From This Incident (HealthCareVox)
-JupiterResearch – End Story (Diva Marketing Blog)
I think students, bloggers and others can learn a lot from this incident.
Thank you, Fard.
Good points. Remembering that episode now, I wish I’d linked to it as an example.
As more and more people become involved in blogging – many want to make money off of the new adopters. With more businesses considering it – even more people see opportunities for profits. Some of these surveys are just one of the tactics used to drive business.
I’m not suggesting that ‘all’ the blogging surveys are created for these purposes. But, when surveys are created by an organization with an agenda, skepticism should be amply applied to any consideration of their claims, IMO.
You’d think that is common sense, but too often they are taken as fact.
Thanks.
Robert
You make many good and important points. Analysis is so important. When a study on business blogging recently came out from the University of Mass., I saw many bloggers linking to it, but no one analyzing it, so I shared this critique: http://inprogress.typepad.com/studio501c/2006/06/report_compiles.html .
Thanks for the post Robert. I wrote a response to it yesterday. With all of the studies I cover from now on, I am going to use the format I set out in my post: Survey title, with links to the report, if available; Findings, Commentary ; and the Bottom line (or summary)
Thanks, Kami. I saw the post and enjoyed it. Commented, too.
I like that format and think I’ll maybe copy it.
Thanks.
Customers Drive Corporate Blogging Growth
As the lead author of two corporate blogging survey’s over the last two years, one initiated on my blog PR Communications, and the other in my role as Director of Blogging Strategies at Backbone Media, I was very interested in
[...] InfOpinions: Posts about surveys :: Look deeper before you believe Bloggers love surveys. They tend to point to data that fits most bloggers personal agendas, and then bloggers run with those numbers. The problem is that most of those surveys are crap, and, well bloggers are self-serving and have their own agenda to push (mostly that everyone should blog). Take these with a grain of salt. [...]
This kind of stuff makes me crazy. I should know, I sell research, and in my younger days used to put out those sorts of press releases that would essentially promote the need for more of my services. But I realized that touting questionable data does more harm to our industry than you ever know. Every time we are revealedas fudgers of data, our crediblity goes down another ten points. See my rant on the subject.
Featured Video
Theme Switcher
Categories
Archives
Most Commented
AuburnMedia Sites Search
Most Viewed
Feedburner