Looks like the battle lines have been drawn! in sand ofcourse . Eric Schmidt and Chris DeWolfe announced earlier today that Myspace is going to support the OpenSocial API being furthered by Google, Ning, and a bunch of other social networks relevant not in the US, but elsewhere in the world. For the time being atleast, facebook has not committed to supporting the API.

This is simply a brilliant move by Google. I admit I was thinking just this morning that this effort wouldn’t amount to much, as long as myspace and facebook both refuse to join it. It seems obvious in hindsight, and probably needed some chess-like thinking ahead, but myspace simply had no choice but to join the effort. Their platform effort was always going to be perceived as an also-ran; their network is being perceived as a boring, old, spammy place to be. facebook *is* the new hotness, and in this game, perception is reality.

By supporting the OpenSocial effort, myspace instantly lends credibility to OpenSocial, and at the same time, gets to reap the benefit of being perceived as visionary. And the developers win too – this is therefore a win-win-win negotiation, as Michael Scott would say.

Facebook’s secret sauce all along has been the “social graph”. It opened it up to outside developers by letting them write apps to harness this information about the users’ real-life connections, but was always super protective of the data itself, not allowing application developers to store any of this information outside of its own servers.

In one move, Google demolished this competitive advantage, making an API that’ll let all sorts of web-based services expose the social graph information they’re aware of. The API is not yet out, but hopefully (and I expect) there will be ways to stitch together connection information from multiple sources into one, and use that information in your app. Imagine the information from GMail, myspace, linkedin and Plaxo, all being utilized by one app, which renders on all four networks, giving you an appropriate (and perhaps tailored) experience on all four!

Without the competitive advantage of this connection information, facebook’s impending SocialAds-network is going to be just another ad network that tracks your interests. This new network was supposed to be facebook’s coming out party to the money making game, but Google nullified the threat to its advertising business (which really is what Google’s core business is). Brilliant by Google..

Another winner in all of this is the developer community. I was dreading the prospect of having to write and maintain N front-ends to my application. Now, I only have to write two, and hopefully soon, just one. There are going to be specializations, because each social network container is going to extend the API by supporting additional functionality, but that’s definitely more manageable than having to learn FBML, FQL, MySpaceHTML etc..

As a developer, my humble request to facebook – Please support OpenSocial! I will love you more if you do. And as your dear friend Mr. Ballmer would say, its all about the “developers, developers, developers, developers, developers, developers, ……, developers“.

For an exposition of how OpenSocial’s APIs work in practice, and what kinds of things are enabled by it, check out Marc Andreessen’s blog – Open Social: a new universe of social applications all over the web and Open Social: a screencast and screen shots

Facebook’s team of management and investors have done an amazingly good job of maneuvering themselves into this cushy position. Have to commend them on a job well done. Regardless of what the current revenues, expected revenues, expect growth or expected growth rate of facebook are, facebook is worth what someone in the market is willing to pay for facebook. And Microsoft, and reportedly two hedge funds from NY, think facebook is worth $15 billion.

Nick Carr makes the connection with “mental anchors”, but says it is ridiculous to extrapolate Microsoft’s $240 million investment in facebook for a 1.6% stake, to its valuation being $15 billion. He says it is a strategic investment on Microsoft’s part, rather than a financial one. While this does sound credible, if the reports of two hedge funds investing at the same valuation turn out to be true, it would appear there are people who do value facebook at $15 billion, no matter what the bloggers say . Or for that matter, what Steve Ballmer’s professed views are about the faddishness of social networks, and the number of man-months it would take to build a social network .

Like Marc says, “You have to love this industry“.. Endlessly fascinating!

Now, if facebook had actually taken Yahoo’s offer a year or so ago, would YHOO be worth $56 billion now ? Something to ponder upon, no?

Paul Graham recently had an essay on the Future of Web Startups, which had a bunch of similar thoughts.

I am one of those folks starting up from scratch without wanting to take venture money (not yet anyway). A large part of these hypotheses are playing out in practice already.

To Mark’s point though, I wonder if there is any way to make the incremental costs scalable and efficient in the enterprise space as well. There are nascent efforts in this space, such as faceforce – a facebook app that bridges facebook and salesforce. This is a huge industry, and it will definitely be interesting to watch how it plays out.

Given the nature of the search problem – indexing and querying the entire web – search companies have been at the bleeding edge of the research in this field.

• Inktomi’s search product (now part of Yahoo!) emerged out of Berkeley’s Network of Workstations (NOW) project back in the late 90′s.
• More recently, Google has been at the forefront of innovation in this field, publishing papers about their internal systems such as the ones on Google File System, Map-Reduce framework, BigTable, Chubby etc. They are also said to have amazing systems to manage these deployments of hundreds of thousands of computers in datacenters distributed around the world. For some interesting papers out of Google in these fields and others, check out the Google Research Publications page.
• Amazon’s Werner Vogels and others have recently published a paper on an internal Amazon system called Dynamo. Dynamo is a scalable, very highly-available key-value store. It is engineered to deliver amazing SLAs such as “99.9% of the read or write requests to the system will execute within 300ms”.

There have been significant efforts in the open source community to solve these problems as well. The Apache Nutch project, started by Doug Cutting (author of Lucene), was one of the first open source efforts to solve the web search problem. Hadoop is an offshoot of the Nutch project, focusing on the cluster / utility / grid framework aspect alone. It has components such as a distributed file system (HDFS) and a Map-Reduce like data processing framework. Yahoo had started to get involved with Hadoop in a big way towards the tail-end of my stint there (late 2005), and have been contributing in a big way to the Hadoop project.

Other notable efforts in this area include:

• Kosmos File System (KFS) by Kosmix: similar to the Google File system. I haven’t had the chance to look through KFS’ sources yet, but I love the fact that it is written in C++.
• Hypertable by Zvents: similar to Google’s BigTable, providing a table-with-rows-and-columns abstraction for very large datasets. My friends Doug Judd and Luke Lu are the lead developers on this project. It uses HDFS or KFS as the backstore, and is written in C++ as well. Being written in C++, with the concomitant advantage in memory performance, is significant in this project. This is because the more efficient use you make of the available memory, the more table data you can keep in memory – this causes the performance of the system to increase dramatically.
• HBase: another project providing BigTable like functionality – it is a sub-project of Hadoop. Powerset had a blog post earlier today announcing the first release of HBase in Hadoop. I know how passionate my friend Barney is about open source projects, and it is great to see that they’re giving back to the community. It’ll also be interesting to see how HBase and Hypertable match up, given HBase is written in java.

It is great to see folks contributing to these open source projects – systems to process very large www-scale datasets are a necessity at an increasing number of startups. Building them in-house is an expensive, time-intensive process. Open source efforts such as Hadoop raise the baseline from which startups can build on, and are very commendable. While Google hasn’t been open sourcing their systems, their contributions to the field (by publishing details of the systems they have engineered) have been fantastic – I still remember the buzz around the publication of the fantastic Map-Reduce framework. They’re also leading an effort to educate computer science students at universities about the challenges inherent in building these kinds of systems. [For some tongue-in-cheek comments from Jeremy Zawodny about how Yahoo is helping Google out - check out this post. Quite funny ]. Yahoo has contributed to the Hadoop project in a big way, and others such as Powerset, Kosmix, Zvents have been making substantial contributions too.

If you’re an engineer who’s bored with the work you’re doing at your job, I would highly recommend that you consider becoming part of any of these open source efforts. I can guarantee you that you’ll have a fun time architecting these systems, and your skill set will be greatly enhanced. And the sense of achievement you get when your system is running clusters of thousands of computers across datacenters, not missing a beat during catastrophic failures like datacenter failures, power outages etc – it’s geek heaven I tell you!

Are there other open source projects that I have missed in this space ?