One of the main bugaboos I’ve developed since ensconcing myself in academia is the drawing of overly-broad conclusions based on ungeneralizable samples or populations. This is often the fault of researchers themselves, but I’ve been unsurprised to learn (given my scepticism of media generally, from a previous life) that it’s perhaps even more often the fault of reporters or media outlets who either don’t care about placing research in its proper, provisional context, or who simply don’t understand that an N=20 self-report survey isn’t sufficient basis for making sweeping conclusions about human nature.
Of course, media reports often don’t bother to use actual data in the first place, “some have reported” being a pretty easy standard to satisfy, so perhaps I’m picking nits here.
But why blog if I’m not going to pick nits? So it was that I greeted James Fallows’ dismantling of some, er, questionable research on Internet usage patterns by Chinese vs. American teens.
The Economist.com takes at face value a silly speech by Barry Diller*, based on a silly survey, and draws silly sky-is-falling conclusions.
The headline on the Economist.com item was: “America’s emobyte** deficit: China’s youth surpass their American rivals online.” The story opened with a quote from Diller:
“THE Chinese people seem to be way ahead of Americans in living a digital life,” said Barry Diller, an American media mogul, last week in a speech to students in Beijing…[Diller’s data] revealed that in this arena as in so much else, China is surging ahead..
They “seem” to be way ahead? I suppose, in the same sense in which I “seem” to be way taller than Yao Ming. Both of these seem true only if you ignore the actual facts.
Which, as far as media goes, doesn’t tend to be the Economist‘s gig (centre-right and all, but mostly pretty reality-based). And once you take a look at the survey itself (really, just a press release from Barry Diller’s consultancy filtered through something called “PR Newswire” and then posted on CNN Money’s website – nice placement if you can get it), you can see why. Among other things, there’s this:
While the U.S. sample is representative of America’s youth, the Chinese sample is necessarily weighted toward the young elite. Only about 10 percent of the Chinese population is online, largely young, urban and educated males. All Chinese respondents had a monthly household income of at least RMB 1,500. (See appendix for more demographic data.)
That’s… problematic. As Jim notes, “this takes us back to blaming who ever swallowed the survey.” And that’d be the Economist reporter. But they’re swallowing something that’s already “out there” – vouched for by CNN, injected into respectable media discourse through PR Newswire, which describes its services as follows:
Whether your news has to go around the corner or around the globe, PR Newswire serves all of your information distribution needs. PR Newswire is the world leader in the electronic delivery of news releases and information directly from companies, institutions and agencies to the media, financial community and consumers.
Presumably, they aren’t doing this for charity. So what’s going on here? As anyone who’s worked in the PR-political-media-corporate food system could tell you, it’s pretty obvious: this story was “placed.” For those unfamiliar with the concept, that means “bought.” As mentioned in the “lede” of the “article,”
Millions of young Chinese are embracing the Internet as a discreet space for their thoughts and emotions, according to a survey of Chinese and American youth released today by IAC, which operates businesses in sectors being transformed by the Internet, and JWT, the fourth largest advertising agency network in the world.
This was a survey designed by a business technology firm – IAC – and an ad firm – JWT – and I’m going to go out on a limb here and say that one or both of these firms might have some sort of financial interest in a more widespread acceptance of of the narrative(s) they’re pushing, yes, even if they’re not really true. Shocking, I know.
I’m not so naïve as to be surprised by this sort of thing. For better or (mostly) worse, it’s the way our discourse works these days. But I’ve got a stake in this particular game, and if I’m so lucky as to do research that gets published in, say, JCMC, I’d like people to read it – scrutinize it – and if it withstands their examination, maybe agree with some of the conclusions I’ve reached. So it kind of chaps my hide that conclusions from poorly-wrought surveys, corporately funded for a particular-result, are able to make their way into wide circulation despite not being, you know… true. Or even pointing in some direction like truth.
This particular survey purported to show American teens “falling behind” their Chinese counterparts because, among other things, “almost five times as many Chinese as American respondents said they have a parallel life online (61 percent vs. 13 percent). And while fewer than half of the 1,079 American respondents agreed that “I live some of my life online” (42 percent), a sizable majority of the 1,104 Chinese respondents agreed with the statement (86 percent).” What this really should have said is “almost five times as many [upper-middle-to-upper-class, mostly male] Chinese as American respondents etc…”, but even leaving aside questions of generalizability, the survey totally dodges a fairly obvious (to me) alternate interpretation of the American results: American teens aren’t living “parallel” lives online, but rather ICT tools – including but not limited to the Internet (and including also mobile phones and mobile social apps) – are being utilized as an increasingly seamless part of their everyday social existences. Put into simpler terms, imagine asking a 14-year-old American, “Do you use MySpace/Facebook, IM, and your cell phone to stay in touch with your friends and live your life?” – and the look of withering contempt accompanying the “Well, duh” response (or perhaps simply a gaped-mouth, disbelieving stare at your incredible, dense lameness) such a question would inspire.
Of course, as researchers we need to quantify that response, even if we “know” it’s true. But that’s the point of research – getting a sense of what the world is like, then running studies to prove or disprove our operating theory. I claim no special nobility for academic research – we all have agendas, driven often by ego, desire for success in its many forms, and so on. But a business tech firm and an ad firm with pre-existing interests and goals designing “research” to “prove” that their product(s) are needed is the other end of the spectrum, here.