Facebook Threatens To Sue Researcher For Crawling Their Site

Despite the fact that the fundamental nature of the web is that public pages are open for indexing and analysis — that’s how search engines and other fundamental tools of the web work — Pete Warden recently discovered that Facebook doesn’t agree.

Pete is the guy that analyzed Facebook data by building his own web crawler, and then published some results of his analysis of social connections. I was one of the many folks that linked to his post, How To Split Up The US, that included this map:

In the post, Pete offered to share his dataset with others, which might have been the point that led to his getting a phone call from Facebook’s chief counsel, demanding that he destroy the data. Or else.

Pete Warden, How I Got Sued By Facebook

On Sunday around 25,000 people read the article, via YCombinator and Reddit. After that a whole bunch of mainstream news sites picked it up, and over 150,000 people visited it on Monday. On Tuesday I was hanging out with my friends at Gnip trying to make sense of it all when my cell phone rang. It was Facebook’s attorney.

He was with the head of their security team, who I knew slightly because I’d reported several security holes to Facebook over the years. The attorney said that they were just about to sue me into oblivion, but in light of my previous good relationship with their security team, they’d give me one chance to stop the process. They asked and received a verbal assurance from me that I wouldn’t publish the data, and sent me on a letter to sign confirming that. Their contention was robots.txt had no legal force and they could sue anyone for accessing their site even if they scrupulously obeyed the instructions it contained. The only legal way to access any web site with a crawler was to obtain prior written permission.

Obviously this isn’t the way the web has worked for the last 16 years since robots.txt was introduced, but my lawyer advised me that it had never been tested in court, and the legal costs alone of being a test case would bankrupt me. With that in mind, I spent the next few weeks negotiating a final agreement with their attorney. They were quite accommodating on the details, such as allowing my blog post to remain up, and initially I was hopeful that they were interested in a supervised release of the data set with privacy safeguards. Unfortunately it became clear towards the end that they wanted the whole set destroyed. That meant I had to persuade the other startups I’d shared samples with to remove their copies, but finally in mid-March I was able to sign the final agreement.

I’m just glad that the whole process is over. I’m bummed that Facebook are taking a legal position that would cripple the web if it was adopted (how many people would Google need to hire to write letters to every single website they crawled?), and a bit frustrated that people don’t understand that the data I was planning to release is already in the hands of lots of commercial marketing firms, but mostly I’m just looking forward to leaving the massive distraction of a legal threat behind and getting on with building my startup.

So: Frankenberg believes that we are moving toward a world where people will be sharing more and more information about themselves, freely. Facebook is merely on a mission to make that happen. People just don’t understand that Facebook isn’t evil, and doesn’t have ulterior motives at heart, or so Zuckerberg says. At least until someone else wants to gather that ‘open’ information, which is published publicly on users’ Facebook pages. Then they want it totally locked up, or else they will sue you.

3 June 2010 Permalink

Zuckerberg Takes Off Hoodie, But Doesn’t Clarify Privacy Issues

I have yet to take the time to exhaustively review the new Facebook privacy settings, as well as various people’s musings about them, but I plan to do so this weekend. In the meantime, Miguel Helft recounts Mark Zuckerberg’s discomfort at the D8 conference yesterday:

Miguel Helft, Zuckerberg On The Hot Seat

About 20 minutes into his on-stage interview at the D8 conference, Mark Zuckerberg had to take off his hoodie, the black sweatshirt that has become his trademark and that he said he never took off.

Mr. Zuckerberg was being grilled about Facebook’s latest privacy flap, and he was visibly uncomfortable and sweating profusely. Mr. Zuckerberg was on the proverbial hot seat and he seemed to know it.

“There have been misperceptions that we are trying to make all information open,” Mr. Zuckerberg said at one point. “That’s completely false.”

The short, crisp statement contrasted with most of his other answers, which were long and rambling, prompting even more questions from Walt Mossberg, the 63-year-old conference co-host.

Mr. Zuckerberg, 26, appeared ill-at-ease with questions that he had answered deftly a week earlier when he admitted that Facebook had made mistakes by letting its privacy settings grow too complicated. At the time, Mr. Zuckerberg announced simplified controls and appeared contrite.

This time, Mr. Zuckerberg was on the defensive for much of the time, but he appeared to get some sympathy from his audience.

When Mr. Mossberg said he would move on from the privacy grilling to other topics applause broke out in the room. But others were less forgiving.

The veteran technology journalist Dan Gillmor, for instance, wrote on Twitter: “Walt Mossberg insists on an answer re FB’s unilateral privacy changes; nope, still no answer.”

I’m with Dan Gillmor on this one.

3 June 2010 Permalink

Facebook Apologists Are Missing The Point: Facebook Isn’t The Future

As the Facebook ‘privacygate’ affair swells and swells, most recently fed by the leaking of Zuckerberg instant messages from years ago, various members of the tech commentariat are starting to come forward to defend Zuckerberg and suggest that the media have gone too far.

My sense is that these apologists are going too far in supporting Zuckerberg and the actions that Facebook has taken; for example, Michael Arrington [my comments are italicized.]

Michael Arrington, The Media Attacks On Facebook And Mark Zuckerberg Are Getting Out Of Hand

Friday is Facebook CEO Mark Zuckerberg’s 26th birthday. My guess is he’s won’t be enjoying it as much as he should, given that the top tech story of the day is a look at a private instant message exchange he supposedly had six or seven years ago at Harvard. The messages show a callous disregard for personal information added by early Facebook users. Given that Facebook is in one of its regularly scheduled privacy scuffles right now, the connection is just too juicy. The press has gone wild.

It’s completely out of hand, and it’s just another example of an online mob getting out of control. I’m embarrassed to see people I respect stopping one step short of calling for physical violence against Zuckerberg. And they certainly aren’t stopping short of calling him every nasty thing they can think of. The Huffington Post actually compared Facebook’s privacy issues to the BP oil spill. Shameful.

Why are you ashamed of your colleagues, who are justifiably incensed by Facebook’s actions and their ham-fisted response to the controversy? Just because a large group of people share similar concerns about Facebook’s policies doesn’t mean that this is ‘mob justice’ — some unthinking swarm of torch-bearing Jacobins hoping to murder anyone better off than them. It could simply be a growing awareness of serious problems; ones that need careful reflection and discussion by our tech pundits.

The Facebook privacy issue is a reasonable thing to debate. Whether or not Vice President of Communications and Public Policy Elliot Schrage gave a reasonable defense of the company’s privacy policies to the New York Times is also a reasonable thing to debate. Even a high profile person saying they’re going to close their Facebook account, obviously for competitive or for promotional purposes, isn’t going too far.

A more investigative analysis of Schrage’s Q&A on the NY Times by Dan Tynan shows that he is either misinformed as to how Facebook’s privacy system works, or he is intentionally misinforming us about it (“lying”). Even leaving Schrage’s Q&A aside, Facebook has clearly not done as much as it could to clarify their privacy position and what it means for users. And considering their market position, they have an obligation to do so.

This is one of the major issues: that Facebook seems determined to not be open and honest about privacy, and they are obviously not making it easy for users to understand the privacy system and the changes they are making, and most importantly, how a user should proceed to get they privacy they want.

One simple observation is that users will not be able to get the privacy they want (or think they already have, or at least had in the past) in today’s Facebook.

But what Mark Zuckerberg said or didn’t say six years ago isn’t relevant to anything. It isn’t an indication of his character, or how he views privacy today. It’s nothing, a snip of a private conversation without context and certainly without the benefit of knowing more about him as a person.

Who here hasn’t said something stupid when they were 19? Who here hasn’t done something dumb when they were 19? None of you. If you’re getting all self righteous, you’re lying to yourself.

On the other hand, Zuckerberg’s nefarious dealings with his former partners at UConnect, the actions he took to squelch the discourse about that, and so on — all actions that took place at the founding of Facebook — do reflect on his character and the company’s DNA.

Six years ago Zuckerberg had no idea what Facebook would become, or how much he’d have to change and mature to handle it. He’s the CEO of one of the most powerful corporations on the planet. He is leading a team that is recreating and redefining our culture as a society.

This line of argument just doesn’t make sense. On one hand, he was a young kid, who didn’t know he would one day be a powerful CEO, and he was unaware of how much he had to grow to handle that responsibility. Ok, granted. But now he is that CEO, and he must be judged on the actions he has taken as CEO of Facebook, even going back in time.

And frankly, none of what Facebook is doing privacy-wise should be a surprise to anyone. At a high level anyway. Facebook is trying to invent, on the fly, an entirely new way or organizing the Internet. 500 million people a month visit the site. They can’t do anything at all without angering some portion of them. And since the service is growing and evolving so fast there’s no way change won’t happen.

Facebook’s privacy misadventure may not be much of a surprise to market-watchers like you and me, Mike, but it is a surprise of some 17-year old in Poughkeepsie or a 35-year old secretary in Los Angeles who still don’t know about the privacy changes, and who are operating under the assumptions they had last year. And while those of us in the bubblicious tech world accept the nosebleed-inducing future shock of incessant and radical change on the web, most people do not. It is completely inadequate to say something like ‘you can’t make an omelette without breaking eggs’ when the eggs in question are people’s lives.

I am no fan of threatened violence (although I have yet to see any of that in this Facebook flare-up), but I believe that Facebook’s users have a solid basis for being seriously pissed off. Even though I am an advocate for publicy — living life in the open on the web — I am by no means an advocate for having it jammed down our throats by a unilateral change in the Terms Of Service agreement by a powerful corporation.

Facebook has shown a studious and callous disregard for the impacts that the company’s decisions are having on people’s lives. What Arrington never really addresses is the fact that Facebook is obviously involved in creating a business model that is strip mining information about users who believed that info would be kept ‘private’, where ‘private’ is a very slippery concept. The only recourse for users at this time is to stop using Facebook altogether, if they want to live a truly private life. This is exactly the fear that Zuckerberg and company are banking on: that users get so much from socializing on Facebook that they will not quit the service, even if they feel that they are being exploited.

Others (like Scoble and Venturebeat ) are softsoaping the flare-up and offering advice to Zuckerberg to get over this public relations hiccup and get back to conquering the world. Meanwhile, aside from the stories about the old IMs, new figures about dropping growth rates by Danny Sullivan suggest that Facebook defection is rising and that the furor may be stopping newbies from signing up.

It’s clearly a turning point for Facebook and the social revolution on the web.

I am an outspoken advocate for social connection and the rise of social tools to help us accomplish that. On the other hand, I am concerned about the centralization of too much control in the hands of a single company; and most especially, in the hands of a single company that seems to be uninterested in the needs of users, and completely motivated by a corporate and financial agenda.

At this point, I would suggest that Facebook’s management and Zuckerberg in particular are not equal to the challenges that confront them, and that even if they get this particular mess behind them, things will start to unwind. Large corporate partners who may have been heading down the road to integrate Facebook into their websites or applications will start to reconsider. Users will opt to spend more time in smaller, more specialized social networks, rather than a single, all-encompassing social context. Application developers will want to create more distance between themselves and Facebook, which increasingly looks like a competitor, not a platform.

And in the final analysis, the next generation of operating environments may turn Facebook into a quaint oddity (and tools like Twitter, as well), because the next generation operating platforms from Google, Apple, Microsoft and others will have sociality built in a fundamental level.

We will be able to ‘follow’ friends — where they are, what they are watching on TV, and what they think we should be reading — across all devices, applications, and contexts — obviously, subject to our own notions of privacy and publicy controls. But this advance — which will be as fundamental as the rise of the web has been to date — cannot be sparked by a player like Facebook. This will come from those who are busy on the foundations of the next generation web, which is not Facebook, despite its dizzying market valuation. Look to interoperable social standards — the future equivalent of HTTP, XMPP, and email protocols — to be forged by competition between Google, Apple, and other foundational players.

We are headed for a time when files and directories are all tidied up, and buried in the gearbox of operating platforms, but where social connection and social networking will be treated as a first class element of the web. This is the social revolution, at last. And Facebook will become a footnote in that history, like SixDegrees.com, Friendster, MySpace and, yes, even Twitter.

14 May 2010 3 Permalink

Facebook Fail: Publicy Backlash

The recent furor about Facebook’s privacy policy changes is the outcome of several factors, the most glaring of which is Facebook’s apparent venality. They seem to consider the formerly private information that users squirrel away on their servers the way the energy industry looks at oil underground: a resource to be mined and exploited for their personal benefit, with little regard for others.

I think that efforts by apologists like Mark Ingram fail: trying to assert a balance between the aspirations of an aggressive company and the need for users to remain aware of changing privacy policies is pseudo rational:

Matthew Ingram, The Relationship Between Facebook and Privacy: It’s Really Complicated

The tension between Facebook and its users — and governments, and advocacy groups — over privacy is one of the biggest thorns in the company’s side right now, as it tries to balance the demands of the network (and of advertisers) with the desires of users, and with the law. And all of this is taking place in an environment where the very meaning of what is “private” and what is “public” is being redefined, by Facebook and other online giants such as Google, and even users themselves sometimes can’t decide what information they want to share with the world and what they don’t.

Over the past few weeks, the social network has been caught at the center of a privacy maelstrom, with consumer groups attacking it — 15 of them filed a formal letter of complaint with the Federal Trade Commission late yesterday — senators sending threatening letters, and growing numbers of users canceling or deactivating their accounts over privacy concerns. The company has been struggling to respond to security holes that expose private data such as chats, and a survey released yesterday by Consumer Reports says that more than 50 percent of people engage in what it calls “risky behavior” on the social network. Another survey of Facebook users finds that their use of the network is inherently shallow and largely unfulfilling.

Even as he is being hailed as a billionaire genius akin to Microsoft founder Bill Gates, the empire Mark Zuckerberg has built seems to be taking fire from critics on all sides. But is all of this criticism fair? Probably not. It’s true that Facebook’s launch of recent changes involving “instant personalization” and the creation of community pages related to users’ profile interests has been badly handled. And it doesn’t help that many people are confused by how to adjust their privacy settings, how to control what information is displayed, and how to disable applications (we put together a comprehensive guide to the new changes and how to disable them if you want to).

But it’s also true that Facebook exists, and has accumulated almost half a billion users worldwide, because it makes it easy for people to connect with their friends and family and to share things with them: photos, thoughts, social games, goofy gifts and yes, even birth dates. Plenty of people clearly want to do this, even after they have been repeatedly warned about the risks, because they believe the trade-off is worth it. And perhaps Facebook doesn’t make it as clear as it could what is involved, or how to fine-tune its privacy controls — but at the same time, some of the onus for doing these things has to fall to users.

Except that the behemoth in this case specifically doesn’t care about the implications of privacy policy changes in the lives of individuals: they have a mass relationship with the millions of users who are being treated like cattle.

So Matthew’s arguments — and others — just don’t cast a lot of light on the issues here, because they are too narrow and too specific to Facebook.

Let me lay out a few threads that I think frame this situation.

The Rise Of Publicy

Facebook’s shifting policy from private as default to public as default is a reflection of the open web. Twitter, in particular, has always been based on a public model, where the default modality is that all information is public unless you go to great lengths to conceal it. Executives of Twitter have gone so far as to say they wanted to publish a publicy policy instead of a privacy policy, but couldn’t because of legal requirements (see A Publicy Policy, Not A Privacy Policy).

So, we have seen a very rapid change in people’s thinking about how and how much to share with others. Perhaps that’s why Matthew Ingram thinks we have to lay some of the ‘blame’ on the users.

However, because Facebook is in a sense trying to track a general shift in the web, it is such a large player — and with so many users that aren’t at the forefront of this trend — a lot of innocent fingers are getting crushed in the machinery.

I think Jarvis is onto something when he says that Facebook has lost the difference between the Public and Publics:

Jeff Jarvis, Confusing *a* public with *the* public

Facebook and Mark Zuckerberg seem to assume that once something is public, it’s public. They confused sharing with publishing. They conflate the public sphere with the making of a public. That is, when I blog something, I am publishing it to the world for anyone and everyone to see: the more the better, is the assumption. But when I put something on Facebook my assumption had been that I was sharing it just with the public I created and control there. That public is private. Therein lies the confusion. Making that public public is what disturbs people. It robs them of their sense of control—and their actual control—of what they were sharing and with whom (no matter how many preferences we can set). On top of that, collecting our actions elsewhere on the net—our browsing and our likes—and making that public, too, through Facebook, disturbed people even more. Where does it end?

Facebook has been playing this tension since its early days. Remember the hubbub over News Feed: When Facebook aggregated our updates into feeds, it freaked users, even though Mark Zuckerberg pointed out that all these updates were already visible to us among our friends on their pages. Zuckerberg’s vision was right in the end; the News Feed is critical to Facebook’s utility, value, and growth and it presaged the appeal of Twitter. But even in the public Twitter, even though we are publishing to the world, we still have a measure of control; we decide whom to follow—that is, which publics to join.

So let me repeat: In Facebook, we get to create our publics. In Twitter, we decide which publics to join. But neither is the public sphere; neither entails publishing to everyone. Yet Facebook is pushing us more and more to publish to everyone and when it does, we lose control of our publics. That, I think, is the line it crossed.

Jarvis confuses things a bit with the various uses of ‘public’, but to restate: when I share things with connections on Facebook I don’t think of it as publishing the NYTimes. It is a social sharing with a specific social network of friends. But Zuckerberg and Co want to imagine everything going to everyone, thus obliterating social scale and tearing down any notion of socially afforded privacy.

Facebook As A Tonedeaf Velociraptor

The comparisons of Zuckerberg with Bill Gates are apt in one regard: he could care less that he is taking policy steps that benefit him to the detriment of users. Gates and Microsoft fought court cases for decades about monopolistic and illegal practices, and we can expect the same to be the case with Facebook, starting with the 15+ suits brought against them in recent weeks.

Zuckerberg wanted to buy Twitter because he was convinced that the open follower model was better than the Facebook architecture in the long run, and he has been pushing to rework Facebook into a system based on publicy rather than privacy ever since. To Zuckerberg, the users aren’t even pieces on the chess board, they are dust underneath the pawns. He’s playing against Ev Williams and Google: he doesn’t give a fig what other people think of him.

As a result, he will continue to delight with Asberger-ish interviews and one liners, like ‘Privacy is dead.’

Facebook Is Not The New Microsoft

So, Facebook might be the new Microsoft, except they don’t have a monopoly on something we absolutely need, like Windows or Office. (Well, we actually didn’t need those either, but it took a long time to get there.)

So a lot of people are simply bailing out. I stopped using Facebook for all intents and purposes several years ago, although I still have an account. My social needs are met by Twitter and blogging, so I haven’t gone through the disruptions of the past few turns of the wheel at Facebook.

We can simply say no. Facebook isn’t essential to life. It’s not even the same Facebook you were using a few months ago. Facebook is betting on people not leaving. Well, that’s what MySpace thought, too.

Open and Governance

There has been no credible open source alternatives to any serious social tool. There has been no open source, wikipedia-like music site, photo sharing site, or video sharing site: the bandwidth and legal issues are too large. There have been attempts for open source movements — like Identica — but they have made little progress in a market dominated by for-profit players. So I don’t think calls for an open Facebook (like Ryan Singel’s) will catch fire.

Perhaps the real question isn’t about ownership, or even privacy, per se: perhaps the real issue is governance. What rights do users have when they are using a service that is owned and operated by a private company? Can the company make policy changes at will? Do the users have recourse if such changes are perceived as harming the users?

Consider the unilateral decision of Twitter to change the sematics of @mentions or the way that retweet (RT) works: many were upset by these changes, and Twitter responded with more grace than Facebook, but that’s a low bar to set.

I think it would behoove any major player to create a user oversight board, made up of users, to help thrash out these sort of issues in advance of any launch of controversial features on an unprepared user community. I don’t believe Facebook will do that, but Twitter and other players might.

Last Word

Publicy is here to stay. It’s just too bad that the fall of privacy will forever be associated with Facebook’s maladroit market moves instead of the benefits of an open web largely built on publicy.

8 May 2010 Permalink

Facebook Wants To Be Twitter

Marshall KirkPatrick www.readwriteweb.com

Facebook wants you to make the status messages you post visible to the entire internet.

[…]

A substantial backlash has already begun in comments on the Facebook blog post about the announcement. Previous moves by the company, like the introduction of the news feed, have seen user resistance as well - but this move cuts against the fundamental proposition of Facebook: that your status updates are only visible to those you opt-in to exposing them to. You’ll now have to opt-out of being public and opt-in to communicating only with people you’ve given permission to see your content.

Will users go for it? If Facebook becomes a lot more like Twitter, will users stick around? The network of friends you’ve created on Facebook can’t be taken anywhere else - access to those people off-site is limited due to “privacy concerns.”

People are spiraling around the real issue here. All the purportedly Machiavellian reasons for Facebook to flipflop on its privacy orientation — to get more page views, or to make activities more visible — miss the real driver. Zuckerberg is obsessed with Twitter. He tried to acquire it once, was spurned, and now he is morphing Facebook into Twitter, which is more or less what he had hoped to do through acquisition.

So, once the pain has abated from the most recent yank in the chain between Zuckerberg’s hand and the user’s neck we will have traveled one giant step closer to a open, streaming son of Facebook.

Remember that Zuckerberg didn’t dream up the design of the original Facebook: he stole the whole thing while working with the Brothers Winklevoss and others on ConnectU (and settled for $65M). His scheme with Twitter was to buy what he sees as the superior model to Facebook Classic. That acquisition failed, so he has acquired Friendfeed’s brain trust, and they are busy rewiring the DNA of the platform.

Don’t get me wrong: he may become a very serious challenge to Twitter. He has serious bank. I think the Twitter guys should do that IPO and hire an additional 100 people to keep innovating faster than he can keep up. And not by rejiggering things like retweet — we need some more groundbreaking advances.

10 December 2009 Permalink