Feeds:
Posts
Comments

Archive for the ‘research’ Category

Look at your phone. Go on, look at it. What is it?

It’s a clock. It’s a text-messaging glass slab. It’s a dynamically updating map/tracking device. It’s a ticket. It’s a late-night magazine. It’s an alarm clock. It’s a camera, photo album and publishing platform. It’s a gaming device, newsfeed(s), and a tether keeping work with you 24 hours a day.

Your laptop: it’s forty tabs open at once, word processing documents, music libraries (if you’re old), an EVEN BETTER gaming device, a TV and movie-watching platform, an audio editing suite, and, uh, other forms of entertainment.

You use these devices for dozens of different purposes, out of convenience and functional capacities. What I want you to think about is who you are in each of those purposes, and for whom you are in those purposes.

One of the most intriguing findings from my dissertation research (read it! become a member of a tiny club!) lo these four years ago was the degree to which students segregated audiences by medium. As I put it, they “use different communications technologies in their interactions with social, familial and academic audiences, in part as a manner of combatting the context collapse taking place on social network sites and mediated communications generally.” More directly: they talked to their friends via text message and Facebook message, called their parents on the phone, and only and ever talked to their professors in person and via email. That was, as they say, interesting, and something worthy of further study.

Well: I didn’t. But while the particular practices have shifted in the intervening time, these behaviors are no less intriguing or worthy of study and contemplation.

Cross-medium behavioral research is rare for a number of reasons. It’s expensive, difficult, time-consuming, methodologically fraught, ethically fraught. But I think the main limiting factor is that in any given moment, the incentives for any organization or individual performing research is to answer their central questions, as quickly/cheaply as possible. For an advertising firm: how did a given campaign deliver on KPIs as promised to the client? For an academic researcher: how does X behavior impact on my hopefully-tenure-securing line of research? For a membership organization: what were the A/B test results on a fundraising solicitation?

And to be crystal clear, this is NOT a problem solved by “Big Data.” Few but the most world-spanning organizations have the capacity to iteratively formulate hypotheses, expand data collection across boundaries, and act on findings. And the evidence suggests that even those world-spanning organizations don’t really know what to do with their endless reams of data. But, really, that’s neither here nor there: if you aren’t inside one of the world’s larger walled gardens of behavioral data, you’re still left with the same question. Namely: just who are your users, and who (and when, and how) are you to your users?

One of the foremost issues is attention. There are two ways of looking at attention: as something to maintain, and as something to be acquired. From your perspective, dear reader, you of course want to maintain sustained attention – on relationships, on work, on engaging culture. An advertiser, on the other hand, wants to capture your attention. Chartbeat – which makes a fantastic suite of products for publishers, that I’ve used and enjoyed – is part of a tech vanguard that recognizes this. As they put it:

Online publishers know clicks don’t always reflect content quality.

But research shows more time spent paying attention to content does.

Advertisers know click-through rates don’t matter for display or paid content.

Research shows 2 things matter for getting a brand’s message across: the ad creative and the amount of time someone spends with it.

The Attention Web is about optimizing for your audience’s true attention.

From their perspective, attention equals quality, and a shift to focusing on quantifying attention means better quality content (oh and also more clients). It’s a compelling thesis – but then, it is your attention that they’re selling, to advertisers. Others are more interested in selling your attention to, well, you:

As our computing devices have become smaller, faster, and more pervasive, they have also become more distracting. The numbers are compelling: Americans spend 11 hours per day on digital devices, workers are digitally interrupted every 10.5 minutes, with interruptions costing the U.S. economy an estimated $650 Billion per year. That’s a lot of distraction.

Device makers have largely turned a blind eye to this issue, building distractions in to the very devices we need for work. We address this challenge with tools that simply and effectively reduce digital distractions. Our software interrupts the habitual cycle of distraction associated with social media, streaming sites, and games.

Attention is basically an adversarial dynamic: your devices and the advertiser-supported content therein yelling at you while you struggle to maintain concentration. Many or most of us are in this stage of managing our relationships with digital communicative prostheses – a struggle. It’s not a struggle without benefits, but nor is it one without costs – study after study shows the costs to both productivity and personal health and well-being of a consistently-interrupted existence.

A central part of this struggle is creating a hierarchy – either explicit or implicit – of attention. When do you respond to a text message? It depends when you receive it, and from whom. Do you return an email? Again: who sent it, work or personal, when did it get received? And then: what do you read, or listen to? That also depends – how did you get there? A link from a friend, an immediately-forgotten source on your social media timeline, through a series of unreproducible clicks? The depth, length, and quality of the attention devoted depends on all these factors and more – but I believe it’s impossible to understand the meaning of a given interaction without looking at how these hierarchies are created.

Read Full Post »

Earlier this week I went to an excellent discussion put on by danah boyd and her Data & Society Research Institute, entitled “Social, Cultural & Ethical Dimensions of ‘Big Data.’” Right off the top, I have to give major kudos to danah for organizing a fantastic panel that incorporated a great combination of voices – who, not for nothing (indeed, for a lot) were not just a bunch of white dudes (only one white dude, in fact) – from across different disciplines and perspectives. I’ll do a brief play-by-play to set the table for a couple of larger thoughts.

Following a rigorously on-message video from John Podesta and fairly anodyne talk (well, except for this) from Nicole Wong from the White House Office of Science and Technology Policy, danah led off with introductory remarks and passed off to Anil Dash, who served excellently as moderator (mostly by staying out of the way, as he made a point of noting). Alondra Nelson from Columbia University was first up, giving an account by turns moving, terrifying, and engaging on the state of play and human consequences flowing from DNA databases – both those managed by law enforcement and the loopholes that allow privately-managed data repositories to skirt privacy protections. She was followed by Shamina Singh from the MasterCard Center for Inclusive Growth, who provided several on-the-ground examples of working with governments, NGOs, and poor people to more efficiently deliver social benefits. In particular, she focused on a MasterCard program to provide direct transfers of cash to refugee populations, cutting out the vastly inefficient global aid infrastructure network.

Singh was followed by Steven Hodas from the New York City Department of Education, who laid out an illuminating picture of the lifecycle of data in education systems, the ways in which private actors subvert and undermine public privacy, and – not just a critic – offered a genuinely thought-provoking new way of thinking about how to regulate dissemination of private information. The excellent Kate Crawford batted cleanup, discussing predictive privacy harms and what she called “data due process.” Dash facilitated a very long and almost entirely productive audience question and discussion session (45 minutes, at the least), and I left with many more things on my mind than I entered with. I’d had the privilege of listening to eight different speakers, each from a background either subtly or radically different from one another. Not once did a speaker follow another just like them, and no small value came in the synthesis from those differing perspectives and those of the audience.

This week also saw the relaunch of FiveThirtyEight.com under its new ESPN/Disney instance. It was launched with a manifesto from founder Nate Silver, entitled “What the Fox Knows,” which is a bit meandering but generally comes down as setting FiveThirtyEight as opposed to both traditional journalism and science research, based on some fairly blithe generalizations of those fields. What it doesn’t quite do, oddly for a manifesto, is state just what FiveThirtyEight is for other than a sort of process and attitudinal approach. Marx (or even Levine/Locke/Searls/Weinberger) it ain’t.

Silver has come in for no small criticism, and not just from his normal antagonists. Emily Bell laid out the rather less-than-revolutionary staffing makeup of the current raft of new-media startups, led by Ezra Klein, Glenn Greenwald, and Silver. And Paul Krugman detailed some rather serious concerns about Silver’s approach:

you can’t be an effective fox just by letting the data speak for itself — because it never does. You use data to inform your analysis, you let it tell you that your pet hypothesis is wrong, but data are never a substitute for hard thinking. If you think the data are speaking for themselves, what you’re really doing is implicit theorizing, which is a really bad idea (because you can’t test your assumptions if you don’t even know what you’re assuming.)

These two critiques are not unrelated. Bell called out Silver for his desire for a “clubhouse,” and rightly so, because groupthink clubhouses – whether of insiders or outsiders – are the most fertile breeding grounds for implicit theorizing. Krugman revisited and expanded his critique, saying:

I hope that Nate Silver understands what it actually means to be a fox. The fox, according to Archilocus, knows many things. But he does know these things — he doesn’t approach each topic as a blank slate, or imagine that there are general-purpose data-analysis tools that absolve him from any need to understand the particular subject he’s tackling. Even the most basic question — where are the data I need? — often takes a fair bit of expertise.

Which brings me around to the beginning of this post. The value in Monday’s discussion flowed directly from both the diversity – in professional background, gender, ethnicity – and the expertise of the speakers present. They each spoke deeply from a particular perspective, and while “Big Data” was the through-line connecting them, the content which animated their discussion, approach, and theorizing was specific to their experience and expertise. The systems that create data have their own biases and agenda, which only discipline-specific knowledge can help untangle and correct for. There is still no Philosopher’s Stone, but base metals have their own stories. Knowing their essential properties isn’t easy or quick, but little is easy that’s of lasting and real value.

Read Full Post »

Having had a few weeks to digest both my initial thoughts on Google+ and the experience of actually using it, I thought I’d step back and offer a 30,000-foot-view of where I think things are, and are going, in this space.

First: Google+ is still pretty nice, even if it doesn’t quite know what it is. That’s fine! It’s been a month in psuedo-beta, and has 10M users. What I think the larger picture is here, is this:

Email is dying. Smart people are helping kill it. Google understands this, and that gMail is at this point a continually depreciating asset (something MSFT never recognized about Hotmail). Google+ is among other things, about providing gMail users a bridge to a post-email digital social space with minimal transition costs (always the biggest barrier to entry for a new social service). The social graph is already built into gMail users’ contact lists – Google+ is just about bootstrapping a different interface onto it.

There are real privacy concerns about Google+, and the mass deletion of accounts shows that Google is still struggling with psuedonymity. But long-run, the proposition is clear for both Google and its users.

Let me backtrack and say that I don’t think email will die, exactly, but rather become a mode of communication used for some things, sometimes, but not everything all the time. Physical mail, similarly, isn’t disappearing anytime soon (well not until 3D printing really scales up), and while telephony might be wearing different hats these days (i.e., mobile and digital rather than locked-position and analog), the fundamental dynamics are the same.

What we’re seeing is a rationalization of communications mediums, with people – young people especially, who don’t have the cruft of legacy communications patterns built on top – only using what makes sense for a given use, at a given time. For quick and short communications, this means text messaging – and given the shift in mobile plans towards offering unlimited texting, this doesn’t make anyone money, because save for the NSA and NewsCorp, nobody’s scanning and indexing your text message for relevant advertising content. Likewise, the efforts of both Facebook and Google to incorporate SMS into their user interfaces just haven’t caught on – why make the easiest way of communicating less easy?

Advertisers don’t know how good they’ve got it right now – despite click-through rates of at best 0.1% on banner ads, they’ve never known more about their target audiences for less money. From here on out, as more and more communication moves from the public social web and indexable email back to a range of peer-to-peer communications (TXT, telephony, video chats), it’s only going to get more and more expensive to know what people are thinking and talking about.  Because you’re going to have to actually ask the people.

This is why it’s a shame how comparatively little institutional support there’s been – and how slow IRBs have been in addressing the pace of online user research – for research around online sociability (no sour grapes here though! SILS is kicking butt!). Apart from the great work of a relatively small cohort of pioneering researchers who’ve been gathering data when and how they can this last decade, we’ve just lost a lot of data: those questions that went unasked, data went unscraped. And when perceptions of an interface aren’t asked in the moment, it’s not just gone, it’s gone-gone – who can remember what Facebook looked like 18 months ago?

But this is also an opportunity. Going forward into a world of multi-channel communications presents a fascinating set of new questions not just about why thus-and-so interface creates certain effects, but what people want to do, and how they realize those desires in different contexts. This is changing constantly, and shows no sign of slowing down. The real opportunity is to build on what we as researchers (both academic and market-oriented) already know about online sociability by asking questions focused not on the vagaries of changing interfaces and services, but rooted in the first principles of how people use communications technology. When we can keep asking those same questions over time – building real longitudinal data that can take into account the ebb and flow of seasons and services – we will build from knowledge to understanding. And when we understand, we can become the masters of our technologies, not the other way around.

Read Full Post »

When telephones were first invented, you didn’t just call a number and get a person on the other end: usually, first you’d talk to an operator, who would then connect you to a local loop where the desired party resided.  There were special rings within each loop to distinguish who was getting the call, but if someone else on your loop or the loop you were calling wanted to listen in, you couldn’t stop them.  This was a function of cost – it was pretty expensive to get a residential telephone line before the federal government guaranteed universal access and then deregulated the phone companies.

This was, it’s pretty well agreed, a bad system notwithstanding the excellent fodder it produced for light farce.  The residential system that replaced it was pretty problematic, too, leading as it did to:

a 70 year or so period where for some reason humans decided it was socially acceptable to ring a loud bell in someone else’s life and they were expected to come running, like dogs.

So, not the best either, even though it too did produce some great songs.

The growth of electronically-mediated and time-shifted communications may have a mixed record on a lot of issues, but it’s an unambiguous good in terms of individual control over their method, mode and timing of response to communications. I think this is a good thing. Communications where the sender is unsure of the extent of the audience, or the receiver potentially forced into confrontation, are not beneficial for either the clear conveyance of meaning or social cohesion.

Which is why Facebook’s recent actions are both troubling and perplexing.  By making all connections public for all users, they are ambiguating audience and forcing potential confrontations (between managed identities, work and personal lives, etc.) for all their users.  The shift in Facebook privacy settings takes as its central premise that the advances in telephone communications of the past century were a bad idea. It is forcing all of its users into an always-on global party line, where the conversations are transcribed and sold to all interested parties.  That’s not good.

Digital technologies allow us the ability to talk to basically whomever we want (and only them) whenever we want (and only then).  That Facebook would consider these to be bad things is deeply weird, and makes a compelling case against using it as a central mode of communication.

Read Full Post »

A few weeks back I took an all-day seminar with Don Dillman, “How Visual Design and Layout Influence Responses to Questionairres.” It was a great course and I definitely recommend taking the opportunity to do anything similar with Dillman or Odum if the opportunity presents itself.

In addition to some great walk-throughs on the power of design to elicit greater rates of survey response, and the importance of harmonizing design elements across multiple modes in survey designs (i.e., web, mail, phone), Dillman also made a pretty shocking (even to him!) point about what his latest research showed: namely, that mail surveys are (still?again?) the best method:

Postal delivery sequence file (DSF) provides all residential addresses and may now be our best household address frame.

When you give this a minute to think, it’s not all that outlandish. Despite huge increases in Internet connectivity – even among older and rural populations – it remains far from universal, and any given channel online (e-mail, SNS) is only going to present a relatively small and self-selecting share of the population. Further, there’s no centralized database of “online users”, and those with the biggest files (Facebook, MySpace, Google, Yahoo! and Microsoft) sure aren’t giving you access to them, Mr./Ms. Academic Researcher. Landline use continues to decline and cell-phone-only-households, with the protection of the Federal Do Not Call Registry, continue to move into a patchwork non-contactable space.

But nearly everyone still has a street address, and even if it’s not always correlated reliably with a person-name, it’s the best way to reach the biggest and most generalizable share of the population. So in a world where more and more of our interactions and identity are moving online and mobile, to spaces where we increasingly control access, how can researchers hope to build generalizable samples of the population?

Let’s step back for a minute and talk about the U.S. in five years. Just as most of the population now has a cell phone, most of the population will have a smart phone/iPhone-like device that will handle voice communications, e-mail, SNS, microblogging, etc. [A point for future discussion is just what this will do to the differential effects of media channel as observed in the media effects literature] It will be the pivot point for all of our communications and personal identity information – we’ll increasingly be using it as an identity storage and verification device for airport check-in, payment and receipt of payment, and a half-dozen other things that now seem outlandish and will soon seem mundane. It’ll be how we carry who we are, and how we tell others about that, for any manner of transactions and interactions.

But that identity will also be floating, a bit. There’ll be several big databases – the mobile companies, Apple, Google, Facebook, etc. – but, again, they won’t be distributing Yellow Pages. Your identity will be relational and transactional and contingent but always subject to change and shift depending on your satisfaction with service provided. Which is good, but presents an increasing challenge to anyone who wants to find some kind of “everyone” (e.g., Census takers, public opinion researchers, etc.). What’s needed is a tether, that also is contingent and user-controlled, but is based on a stable hook.

The DSF can provide that hook. Most people will continue to have a street address – indeed, even the homeless can provide some manner of address that would interface with the DSF – even as the majority of their communications are mediated through shifting electronic interface. A user-controlled and -verified system of tying your various communication methods – contingently – to a physical address could allow users the ability to better control access to all manner of modes of communication and contact. Physical address and solicitation could become tokens that would then be entered into whatever other interface you wished (Amazon for deliveries, Gallup for polls, IRS.gov for taxes), allowing the third-party only what permissions you desired but also providing the verification layer that you are indeed [a person]. Of course this would raise all sorts of new issues about interface and self-report data, but given Dillman’s very promising results – >50% response rates to online surveys via mail solicitation (and >70% via mail) – this is certainly worth thinking about more extensively.

Read Full Post »

I had a great time presenting on my work with the Bot2.0 project earlier today at ASIS&T; you can find a copy of the talk here [.pdf]. Thanks to Miguel Ruiz for organizing the panel, moderating and presenting, to my co-panelists Bryan Heidorn and Nathan Hall, and everyone who came out to listen and ask some great questions.

Read Full Post »

A Future Academic Discourse

Phil Edwards pointed this morning to an excellent article from Inside Higher Ed on the impossibility of keeping up with current scholarship in academia, and asking what the way forward might be:

We have collectively created the equivalent of an academic monsoon over the past three decades, with no change in the forecast for the coming years. Without a major reconsideration of how we share and use information, how we keep up with the field, and how we recognize academic accomplishment, we will continue to add to the floodwaters, all the while spending less attention on whether or not anyone reads our work, listens to our presentations, or appreciates our professional contributions. Academe 2.0 offers tools to build more effective dikes and even to regulate the flow. But we need to realize that the lakes at the end of the bloated academic rivers – our faculty, researchers and students – have finite capacity, in terms of time and ability to assimilate information. Controlling the scholarly input is crucial to ensuring that we actually learn from and about each other, and ensuring that our academic work truly makes a difference.

Hill Taylor notes that the University of Michigan Press is moving away from monograph publishing and towards a digital approach, and that

uses and practices of literacy will change because of this too. Preferences for consumption and organization of such information will drive these new literacies. Of course, policy and pedagogy must recognize this change, driven by digital literacy, and accommodate accordingly.

While I’m not sure if this is a generalizable example, research into online activities does point out one way that we might square these circles. In my current research into tagging and folksonomies, many of the seminal piece of research and commentary – including the coining of the term “folksonomy” itself – occurred not in the pages of a peer-reviewed journal but in self-published and mediated online discussion: blogs, forums, mailing lists. Most of this was inherently dispensible, but some of it has stood the test of time, and what distinguishes the memorable from the forgettable is not the imprimatur of a journal’s nameplate but the usefulness of the information and analysis.

This is not a plea to abandon the peer-reviewed journal process – for the highest quality research, I believe it can and should serve a valuable purpose in disseminating knowledge. Rather, I would suggest that digital monographs and online self-publishing present a potentially better model for the actual exchange and construction of knowledge than a massive conference with an unreadable proceedings – the program referenced in the first piece above ran to 180 pages, never mind any of the papers presented. I am more likely to read, cite, and comment on a piece of scholarship if it’s actually available to me, and pushing out digital monographs via non-DRM’d .pdfs is a better model of accessibility than far-flung conferences with dozens or hundreds of unattendable sessions.

Read Full Post »

On Writing

One of my favorite authors, Jonathan Carroll, maintains a decidedly non-techy blog (Bill Gibson and Charlie Stross, not surprisingly, tend to be much more webizens in their online writings). He’ll sometimes share links but usually it’s small observations of the wonder of the mundane (his or his readers’), or excerpts from his writing that are appropriate for his mood. It’s a very nice blog, actually, rarely more than a graf or two a day but always worthwhile.

Today’s entry is more involved and descriptive than most, and quite helpful. I always enjoy reading writers writing about writing (or any practitioner writing about what they do, but given that I spend quite a bit of time fiddling with words, writers’ perspectives are especially useful to me). Not necessarily because their approaches are always applicable – neither profile of creativity/productivity Gladwell explored in “Late Bloomers” is anywhere near how I work – but just to see how all sorts of people approach the same issues.

Along those lines, here is some of what Carroll discusses:

One of the questions people frequently ask is do I ever get writer’s block and if so, what do I do about it? Luckily I’ve never had that gruesome beast but I do have some thoughts about it, and those thoughts run into the idea of creativity in general.

I like to write. I always have. I consider writing my friend. We sit down together in the morning and do our job. But (and this is a big but) if my friend Writing (notice the capital W) says not today because I’d rather goof off, or drink coffee, or nothing at all, I say fine—no work today. If that extends to a week, then so be it. Like the wild animals living so incongruously but comfortably in a gamekeeper’s house on the Serengeti Plain, Writing stays friendly so long as I let him come and go as he pleases. If he doesn’t want to stay in the house and walks out for a while, I simply do something else like read a book or go to the movies. I never, ever grab Writing by the neck and say you sit back down here and go to work. I’d never treat a friend like that, nor would I treat a tiger like that. So why treat the thing I love as much as my creative ability like that?

I believe people get writer’s block a lot of the time because they panic when the flow stops. Then they run around the house shutting the ‘doors and windows,’ trying to trap their creativity inside. Bad idea. I do think that if they were to just move away from the work for however long, many of their problems would solve themselves. Some of you could say yeah but I’ve been blocked for six months—what about that? I’d posit it’s likely some of your block, perhaps not all, is because you are frightened and trying to close all your windows. Which in turn has scared your Writing and made IT panicky. You get my drift. Of course there are exceptions. But I really do believe the greatest trick to either get going in the morning, or after a long dry spell, or even trying to conquer the fearsome mountain of ‘I don’t know where to go from here’….is to get up and walk away. At least until you feel comfortable. Or in the best-case scenario, until you are eager to get back to work again. Because at that point your friend Writing or Creativity says okay, I’m rested and ready to go. I’m so happy you left me alone to go out into the world a while to recharge my batteries.

Obviously it’s easy to take this advice too far, and just endlessly postpone dealing with the problem of Writing the Thing. There’s probably a contrast here between professional writers – for whom eventually finishing whatever they’re working on results in a paycheck and promise of another – and graduate students, for whom delaying completion of the dissertation can (to a point, after which the “Done yet?” questions feel like lead basketballs [or so I’ve heard]) extend a very pleasant life of inquiry and intellectual discussion in a nice place.

As I write this, I’ve just received the first feedback from my advisor on a very preliminary draft of one of the several chapters of my literature review. It’s a very incremental movement of the status bar across the screen, but it’s movement, and for me that’s one of the most important parts of the writing process. I tend to view writing as a problem-solving process: solving the problem of writing [WRITING OBJECT X]. This has been true for as long as I can remember, whether it’s a research paper, a newspaper article, a two-pager, an agenda, or what have you.

The first part of this process is understanding what the parameters are of [WRITING OBJECT X] – what are the variables I need to be aware of, what does it look like when it’s completed, what am I trying do here, etc.? Once I can quantify what needs to happen, I can start assembling the various elements and putting them together, and have at least a fair idea of how it’s going. The status bar is not a perfect analogy – at least for me, there’s no explicit “XX% complete” flashing – but useful nonetheless in that as I’m working I can definitely see/feel the movement towards completion. Iterations, drafts, processing of notes and citations are of course as important (moreso, actually) for me as the actual laying of words on the screen.

I’m lucky enough that this is all a pretty enjoyable process for me, which makes graduate school both a great place and something that is necessarily going to end as part of its nature. If there’s one central takeaway that I have found as a key to this whole process for me, one piece of advice that people seem to find useful, it’s this: words are not precious. Conceiving of the writing process as a problem-solving exercise allows me to let go of attachment to any particular set of words, phrases, or formulations. Revisions become not an ego-wounding and heart-wrenching process of letting go and moving on, but rather something more like adding a second (third, fourth…, nth) coat of paint to a room – something that is simply part of the project. And just as with any project that’s a problem-solving exercise – tilling and planting a garden, putting together a bed from Ikea, etc. – at the end of writing [WRITING OBJECT X] I tend to feel not the disgust and loathing that many writers describe with their just-completed works, but simply the satisfaction of a job done.

Speaking of which – back to work, now.

Read Full Post »

Real Long Distance“/Josh Ritter

What is the lag between proposal of new theories in a field and their widespread adoption (if they are adopted at all)? Is there a consistent model by field for this lag, or for adoption? What would be the important factors to examine in a cross-field metanalysis of new theory adoption?

Read Full Post »

Black Swan“/Thom Yorke

What are the factors that lead to rationalization of markets – in goods, services, thoughts, artistic expressions – online? How does this compare with rationalization of markets in a city

  • pre-newspaper
  • in the newspaper era
  • in the phone book era

Read Full Post »

Older Posts »