Andy Carvin's Waste of Bandwidth

« Chirac Declares War on Google Library Project | Main | Last Night's Boston Digital Divide Meetup »

March 22, 2005

Local Languages, RSS and the Digital Divide

Susan Mernit recently posted a blog entry about the issue of RSS and its ability to display non-Latinate languages. Susan's blog was in turn inspired by postings on Rebecca MacKinnon's blog about diversity in the blogosphere. Both blogs include a quote from Richard Sambrook of the BBC:

I was speaking to one of our BBC World Service software engineers yesterday who made a point I hadn't appreciated but which potentially has a hugely negative effect on diversity: The issue is RSS does not have a way to display right to left languages correctly and is not very compatible with non Latin languages. I believe it just was not thought about deeply by the people and development effort behind RSS.

This slows down the growth of non Latin RSS adoption. We need to develop multiple language RSS and hopefully redefine standards and approaches.

I've spent a lot of time in the last few years noting the importance of producing local-language content on the Internet. According to various surveys, it's believed that around two-thirds of all Internet content is produced in English, even though English speakers make up less than 10% of the world's population. Some languages, such as Spanish and Chinese, are finally beginning to blossom online, though they still trail behind English, the lingua franca of online discourse.

Unfortunately, there haven't been many global surveys regarding language and Internet content. One important study came from the Barcelona media company Vilaweb in 2001, which found around 68% percent of all websites to be in English. When the study came out, I decided to make a quick chart that compared the number of Web pages found in a given language with the number of people worldwide who spoke that language. Here's what I found. (Again, please not that this data is several years old, so take it with a grain of salt...)

Web Pages and Languages, ranked by the number of speakers per web page:

As you can see here, there were about one and a half English speakers for every Web page in English. Interestingly, the next highest ranking came from Iceland; while they're aren't many Icelandic speakers, they've produced a lot of online content, so the ratio of speakers to Web pages is close as well. But compare this to Arabic-language content: there were so few Web pages in Arabic at the time compared to the large population of Arabic speakers, you end up with more than 1,500 people per Arabic Web page. Of course, this data is a few years old, and I'd love to update this chart, but so far I haven't seen a recent study that tabulates the number of pages for each of these languages.

So how of all of this relate to RSS? Well, RSS has become the de-facto way to syndicate content on the Internet. Blogs and news services rely on RSS, as do a growing number of blog consumers. But like the recent Pew study demonstrated, the average blogger is white, well-educated, well-off and English speaking. There's no way we can seriously bridge the digital divide as long as people can't create or access knowledge in their native language. If somehow we managed to bring Internet access to every village in the developing world, it won't mean much if those villagers are stuck using the Net only in English.

Fortunately, the UNICODE project has helped bring local languages to the Internet by providing a universal scheme for displaying tens of thousands of non-Latin characters. This means that I can go to an Israeli newspaper and read it in Hebrew, or an Iranian blog in Farsi. Slowly but surely, language is becoming democratized online, even if the amount of content or readers for a particular language leaves much to be desired.

But what if RSS isn't capable of handling all of these UNICODE languages? Will a Gujarati family in New Jersey be able to read news feeds coming from Gujarati bloggers in India? Honestly, I'm not sure. Technically, it should work: if you take a look at Hoder's blog, written in Farsi (Persian), you can subscribe to his Farsi RSS feed. I don't speak Farsi, but when I tried to subscribe to his feed using Mozilla Thunderbird, I received tons of blog entries, all of which were indeed written in Farsi. So in this particular case, the system apparently works.

But does this apply to all languages supported by UNICODE? Frankly, I have no idea, so I'm hoping some UNICODE techie will jump in and set the record straight. But I certainly hope it can work for all UNICODE-supported languages. Otherwise, we'll see a new facet of the linguistic digital divide. For languages with RSS support, knowledge gets produced and disseminated at a rapid pace, allowing more online knowledge to be produced, and an expanding community of people able to talk about this knowledge and contribute even further to it. But for languages that can't be transmitted via RSS, they'll be stuck sharing content at a much slower pace, to smaller, less-connected audience. Internet users shouldn't be penalized just because their language isn't on a global top 10 list. So let's make sure that people can blog and publish in the language of their choice -- and that their RSS feeds will support them, 100 percent. -andy

Posted by acarvin at March 22, 2005 2:29 PM

Listen to a computer-generated podcast of this article