This Blog has Moved to NovaSpivack.com

Republished from March 11: Twine CEO Comments about Evri Acquisition

Please see this article -- my comments on the Evri/Twine deal, as CEO of Twine. This provides more details about the history of Twine and what led to the acquisition.

What's After the Real Time Web?

In typical Web-industry style we're all focused minutely on the leading trend-of-the-year, the real-time Web. But in this obsession we have become a bit myopic. The real-time Web, or what some of us call "The Stream," is not an end in itself, it's a means to an end. So what will it enable, where is it headed, and what's it going to look like when we look back at this trend in 10 or 20 years?

In the next 10 years, The Stream is going to go through two big phases, focused on two problems, as it evolves:

1. Web Attention Deficit Disorder. The first problem with the real-time Web that is becoming increasingly evident is that it has a bad case of ADD. There is so much information streaming in from so many places at once that it's simply impossible to focus on anything for very long, and a lot of important things are missed in the chaos. The first generation of tools for the Stream are going to need to address this problem.
2. Web Intention Deficit Disorder. The second problem with the real-time Web will emerge after we have made some real headway in solving Web attention deficit disorder. This second problem is about how to get large numbers of people to focus their intention not just their attention. It's not just difficult to get people to notice something, it's even more difficult to get them to do something. Attending to something is simply noticing it. Intending to do something is actually taking action, expending some energy or effort to do something. Intending is a lot more expensive, cognitively speaking, than merely attending. The power of collective intention is literally what changes the world, but we don't have the tools to direct it yet.

The Stream is not the only big trend taking place right now. In fact, it's just a strand that is being braided together with several other trends, as part of a larger pattern. Here are some of the other strands I'm tracking:

• Messaging. The real-time Web aka The Stream is really about messaging in essence. It's a subset of the global trend towards building a better messaging layer for the Web. Multiple forms of messaging are emerging, from the publish-and-subscribe nature of Twitter and RSS, to things like Google Wave, Pubsubhubub, and broadcast style messaging or multicasting via screencast, conferencing and media streaming and events in virtual worlds. The effect of these tools is that the speed and interactivity of the Web areincreasing -- the Web is getting faster. Information spreads more virally, more rapidly -- in other words, "memes" (which we can think of as collective thoughts) are getting more sophisticated and gaining more mobility.
• Semantics. The Web becomes more like a database. The resolution of search, ad targeting, and publishing increases. In other words, it's a higher-resolution Web. Search will be able to target not just keywords but specific meaning. For example, you will be able to search precisely for products or content that meet certain constraints. Multiple approaches from natural language search to the metadata of the Semantic Web will contribute to increased semantic understanding and representation of the Web.
• Attenuation. As information moves faster, and our networks get broader, information overload gets worse in multiple dimensions. This creates a need for tools to help people filter the firehose. Filtering in its essence is a process of attenuation -- a way to focus attention more efficiently on signal versus noise. Broadly speaking there are many forms of filtering from automated filtering, to social filtering, to personalization, but they all come down to helping someone focus their finite attention more efficiently on the things they care about most.
• The WebOS.  As cloud computing resources, mashups, open linked data, and open API's proliferate, a new level of aggregator is emerging. These aggregators may focus on one of these areas or may cut across them. Ultimately they are the beginning of true cross-service WebOS's. I predict this is going to be a big trend in the future -- for example instead of writing Web apps directly to various data and API's in dozens of places, just write to a single WebOS aggregator that acts as middleware between your app and all these choices. It's much less complicated for developers. The winning WebOS is probably not going to come from Google, Microsoft or Amazon -- rather it will probably come from someone neutral, with the best interests of developers as the primary goal.
• Decentralization. As the semantics of the Web get richer, and the WebOS really emerges it will finally be possible for applications to leverage federated, Web-scale computing. This is when intelligent agents will actually emerge and be practical. By this time the Web will be far too vast and complex and rapidly changing for any centralized system to index and search it. Only massively federated swarms of intelligent agents, or extremely dynamic distributed computing tools, that can spread around the Web as they work, will be able to keep up with the Web.
• Socialization. Our interactions and activities on the Web are increasingly socially networked, whether individual, group or involving large networks or crowds. Content is both shared and discovered socially through our circles of friends and contacts. In addition, new technologies like Google Social Search enable search results to be filtered by social distance or social relevancy. In other words, things that people you follow like get higher visibility in your search results. Socialization is a trend towards making previously non-social activities more social, and towards making already-social activites more efficient and broader. Ultimately this process leads to wider collaboration and higher levels of collective intelligence.
• Augmentation. Increasingly we will see a trend towards augmenting things with other things. For example, augmenting a Web page or data set with links or notes from another Web page or data set. Or augmenting reality by superimposing video and data onto a live video image on a mobile phone. Or augmenting our bodies with direct connections to computers and the Web.

If these are all strands in a larger pattern, then what is the megatrend they are all contributing to? I think ultimately it's collective intelligence -- not just of humans, but also our computing systems, working in concert.

Collective Intelligence

I think that these trends are all combining, and going real-time. Effectively what we're seeing is the evolution of a global collective mind, a theme I keep coming back to again and again. This collective mind is not just comprised of humans, but also of software and computers and information, all interlinked into one unimaginably complex system: A system that senses the universe and itself, that thinks, feels, and does things, on a planetary scale. And as humanity spreads out around the solar system and eventually the galaxy, this system will spread as well, and at times splinter and reproduce.

But that's in the very distant future still. In the nearer term -- the next 100 years or so -- we're going to go through some enormous changes. As the world becomes increasingly networked and social the way collective thinking and decision making take place is going to be radically restructured. 

Social Evolution

Existing and established social, political and economic structures are going to either evolve or be overturned and replaced. Everything from the way news and entertainment are created and consumed, to how companies, cities and governments are managed will change radically. Top-down beaurocratic control systems are simply not going to be able to keep up or function effectively in this new world of distributed, omnidirectional collective intelligence.

Physical Evolution

As humanity and our Web of information and computatoins begins to function as a single organism, we will evolve literally, into a new species: Whatever is after the homo sapien. The environment we will live in will be a constantly changing sea of collective thought in which nothing and nobody will be isolated. We will be more interdependent than ever before. Interdependence leads to symbiosis, and eventually to the loss of generality and increasing specialization. As each of us is able to draw on the collective mind, the global brain, there may be less pressure on us to do things on our own that used to be solitary. What changes to our bodies, minds and organizations may result from these selective evolutionary pressures? I think we'll see several, over multi-thousand year timescales, or perhaps faster if we start to genetically engineer ourselves:

• Individual brains will get less good at things like memorization and recall, calculation, reasoning, and long-term planning and action.
• Individual brains will get better at multi-tasking, information filtering, trend detection, and social communication. The parts of the nervous system involved in processing live information will increase disproportionately to other parts.
• Our bodies may actually improve in certain areas. We will become more, not less, mobile, as computation and the Web become increasingly embedded into our surroundings, and into augmented views of our environments. This may cause our bodies to get into better health and shape since we will be less sedentary, less at our desks, less in front of TV's. We'll be moving around in the world, connected to everything and everyone no matter where we are. Physical strength will probably decrease overall as we will need to do less manual labor of any kind.

These are just some of the changes that are likely to occur as a result of the things we're working on today. The Web and the emerging Real-Time Web are just a prelude of things to come.

The Future of the Web: BBC Interview

The BBC World Service's Business Daily show interviewed the CTO of Xerox and me, about the future of the Web, printing, newspapers, search, personalization, the real-time Web. Listen to the audio stream here. I hear this will only be online at this location for 6 more days. If anyone finds it again after that let me know and I'll update the link here.

The Next Generation of Web Search -- Search 3.0

The next generation of Web search is coming sooner than expected. And with it we will see several shifts in the way people search, and the way major search engines provide search functionality to consumers.

Web 1.0, the first decade of the Web (1989 - 1999), was characterized by a distinctly desktop-like search paradigm. The overriding idea was that the Web is a collection of documents, not unlike the folder tree on the desktop, that must be searched and ranked hierarchically. Relevancy was considered to be how closely a document matched a given query string.

Web 2.0, the second decade of the Web (1999 - 2009), ushered in the beginnings of a shift towards social search. In particular blogging tools, social bookmarking tools, social networks, social media sites, and microblogging services began to organize the Web around people and their relationships. This added the beginnings of a primitive "web of trust" to the search repertoire, enabling search engines to begin to take the social value of content (as evidences by discussions, ratings, sharing, linking, referrals, etc.) as an additional measurment in the relevancy equation. Those items which were both most relevant on a keyword level, and most relevant in the social graph (closer and/or more popular in the graph), were considered to be more relevant. Thus results could be ranked according to their social value -- how many people in the community liked them and current activity level -- as well as by semantic relevancy measures.

In the coming third decade of the Web, Web 3.0 (2009 - 2019), there will be another shift in the search paradigm. This is a shift to from the past to the present, and from the social to the personal.

Established search engines like Google rank results primarily by keyword (semantic) relevancy. Social search engines rank results primarily by activity and social value (Digg, Twine 1.0, etc.). But the new search engines of the Web 3.0 era will also take into account two additional factors when determining relevancy: timeliness, and personalization.

Google returns the same results for everyone. But why should that be the case? In fact, when two different people search for the same information, they may want to get very different kinds of results. Someone who is a novice in a field may want beginner-level information to rank higher in the results than someone who is an expert. There may be a desire to emphasize things that are novel over things that have been seen before, or that have happened in the past -- the more timely something is the more relevant it may be as well.

These two themes -- present and personal -- will define the next great search experience.

To accomplish this, we need to make progress on a number of fronts.

First of all, search engines need better ways to understand what content is, without having to do extensive computation. The best solution for this is to utilize metadata and the methods of the emerging semantic web.

Metadata reduces the need for computation in order to determine what content is about -- it makes that explicit and machine-understandable. To the extent that machine-understandable metadata is added or generated for the Web, it will become more precisely searchable and productive for searchers.

This applies especially to the area of the real-time Web, where for example short "tweets" of content contain very little context to support good natural-language processing. There a little metadata can go a long way. In addition, of course metadata makes a dramatic difference in search of the larger non-real-time Web as well.

In addition to metadata, search engines need to modify their algorithms to be more personalized. Instead of a "one-size fits all" ranking for each query, the ranking may differ for different people depending on their varying interests and search histories.

Finally, to provide better search of the present, search has to become more realtime. To this end, rankings need to be developed that surface not only what just happened now, but what happened recently and is also trending upwards and/or of note. Realtime search has to be more than merely listing search results chronologically. There must be effective ways to filter the noise and surface what's most important effectively. Social graph analysis is a key tool for doing this, but in addition, powerful statistical analysis and new visualizations may also be required to make a compelling experience.

Sneak Peak - Siri -- Interview with Tom Gruber

Sneak Preview of Siri – The Virtual Assistant that will Make Everyone Love the iPhone, Part 2: The Technical Stuff

 

In Part-One of this article on TechCrunch, I covered the emerging paradigm of Virtual Assistants and explored a first look at a new product in this category called Siri. In this article, Part-Two, I interview Tom Gruber, CTO of Siri, about the history, key ideas, and technical foundations of the product:

Nova Spivack: Can you give me a more precise definition of a Virtual Assistant?

Tom Gruber: A virtual personal assistant is a software system that

• Helps the user find or do something (focus on tasks, rather than information)
• Understands the user's intent (interpreting language) and context (location, schedule, history)
• Works on the user's behalf, orchestrating multiple services and information sources to help complete the task

In other words, an assistant helps me do things by understanding me and working for me.    This may seem quite general, but it is a fundamental shift from the way the Internet works today.  Portals, search engines, and web sites are helpful but they don't do things for me - I have to use them as tools to do something, and I have to adapt to their ways of taking input.

Nova Spivack: Siri is hoping to kick-start the revival of the Virtual Assistant category, for the Web. This is an idea which has a rich history. What are some of the past examples that have influenced your thinking?

Tom Gruber: The idea of interacting with a computer via a conversational interface with an assistant has excited the imagination for some time.  Apple's famous Knowledge Navigator video offered a compelling vision, in which a talking head agent helped a professional deal with schedules and access information on the net.  The late Michael Dertouzos, head of MIT's Computer Science Lab, wrote convincingly about the assistant metaphor as the natural way to interact with computers in his book "The Unfinished Revolution: Human-Centered Computers and What They Can Do For Us".  These accounts of the future say that you should be able to talk to your computer in your own words, saying what you want to do, with the computer talking back to ask clarifying questions and explain results.  These are hallmarks of the Siri assistant.  Some of the elements of these visions are beyond what Siri does, such as general reasoning about science in the Knowledge Navigator.  Or self-awareness a la Singularity.  But Siri is the real thing, using real AI technology, just made very practical on a small set of domains. The breakthrough is to bring this vision to a mainstream market, taking maximum advantage of the mobile context and internet service ecosystems.  

Nova Spivack: Tell me about the CALO project, that Siri spun out from. (Disclosure: my company, Radar Networks, consulted to SRI in the early days on the CALO project, to provide assistance with Semantic Web development)

Tom Gruber: Siri has its roots in the DARPA CALO project (“Cognitive Agent that Learns and Organizes”) which was led by SRI.  The goal of CALO was to develop AI technologies (dialog and natural language understanding,s understanding, machine learning, evidential and probabilistic reasoning, ontology and knowledge representation, planning, reasoning, service delegation) all integrated into a virtual assistant that helps people do things.  It pushed the limits on machine learning and speech, and also showed the technical feasibility of a task-focused virtual assistant that uses knowledge of user context and multiple sources to help solve problems. 

Siri is integrating, commercializing, scaling, and applying these technologies to a consumer-focused virtual assistant.  Siri was under development for several years during and after the CALO project at SRI. It was designed as an independent architecture, tightly integrating the best ideas from CALO but free of the constraints of a national distributed research project. The Siri.com team has been evolving and hardening the technology since January 2008.

Nova Spivack: What are primary aspects of Siri that you would say are “novel”?

Tom Gruber: The demands of the consumer internet focus -- instant usability and robust interaction with the evolving web -- has driven us to come up with some new innovations:

• A conversational interface that combines the best of speech and semantic language understanding with an interactive dialog that helps guide people toward saying what they want to do and getting it done. The conversational interface allows for much more interactivity that one-shot search style interfaces, which aids usability and improves intent understanding.  For example, if Siri didn't quite hear what you said, or isn't sure what you meant, it can ask for clarifying information.   For example, it can prompt on ambiguity: did you mean pizza restaurants in Chicago or Chicago-style pizza places near you?  It can also make reasonable guesses based on context.  Walking around with the phone at lunchtime, if the speech interpretation comes back with something garbled about food you probably meant "places to eat near my current location".  If this assumption isn't right, it is easy to correct in a conversation.  
• Semantic auto-complete - a combination of the familiar "autocomplete" interface of search boxes with a semantic and linguistic model of what might be worth saying.  The so-called "semantic completion" makes it possible to rapidly state complex requests (Italian restaurants in the SOMA neighborhood of San Francisco that have tables available tonight) with just a few clicks.  It's sort of like the power of faceted search a la Kayak, but packaged in a clever command line style interface that works in small form factor and low bandwidth environments. 
• Service delegation - Siri is particularly deep in technology for operationalizing a user's intent into computational form, dispatching to multiple, heterogeneous services, gathering and integrating results, and presenting them back to the user as a set of solutions to their request.  In a restaurant selection task, for instance, Siri combines information from many different sources (local business directories, geospatial databases, restaurant guides, restaurant review sources, online reservation services, and the user's own favorites) to show a set of candidates that meet the intent expressed in the user's natural language request.  

Nova Spivack: Why do you think Siri will succeed when other AI-inspired projects have failed to meet expectations?

Tom Gruber: In general my answer is that Siri is more focused. We can break this down into three areas of focus:

• Task focus. Siri is very focused on a bounded set of specific human tasks, like finding something to do, going out with friends, and getting around town.  This task focus allows it to have a very rich model of its domain of competence, which makes everything more tractable from language understanding to reasoning to service invocation and results presentation
  
• Structured data focus. The kinds of tasks that Siri is particularly good at involve semistructured data, usually on tasks involving multiple criteria and drawing from multiple sources.  For example, to help find a place to eat, user preferences for cuisine, price range, location, or even specific food items come into play.  Combining results from multiple sources requires reasoning about domain entity identity and the relative capabilities of different information providers.  These are hard problems of semantic information processing and integration that are difficult but feasible today using the latest AI technologies. 
• Architecture focus.  Siri is built from deep experience in integrating multiple advanced technologies into a platform designed expressly for virtual assistants.  Siri co-founder Adam Cheyer was chief architect of the CALO project, and has applied a career of experience to design the platform of the Siri product.  Leading the CALO project taught him a lot about what works and doesn't when applying AI to build a virtual assistant.   Adam and I also have rather unique experience in combining AI with intelligent interfaces and web-scale knowledge integration.  The result is a "pure  play" dedicated architecture for virtual assistants, integrating all the components of intent understanding, service delegation, and dialog flow management.  We have avoided the need to solve general AI problems by concentrating on only what is needed for a virtual assistant, and have chosen to begin with a finite set of vertical domains serving mobile use cases.

Nova Spivack: Why did you design Siri primarily for mobile devices, rather than Web browsers in general?

Tom Gruber: Rather than trying to be like a search engine to all the world's information, Siri is going after mobile use cases where deep models of context (place, time, personal history) and limited form factors magnify the power of an intelligent interface.  The smaller the form factor, the more mobile the context, the more limited the bandwidth : the more it is important that the interface make intelligent use of the user's attention and the resources at hand.  In other words, "smaller needs to be smarter."  And the benefits of being offered just the right level of detail or being prompted with just the right questions can make the difference between task completion or failure.  When you are on the go, you just don't have time to wade through pages of links and disjoint interfaces, many of which are not suitable to mobile at all.  

Nova Spivack: What language and platform is Siri written in?

Tom Gruber: Java, Javascript, and Objective C (for the iPhone)

Nova Spivack: What about the Semantic Web? Is Siri built with Semantic Web open-standards such as RDF and OWL, Sparql?

Tom Gruber: No, we connect to partners on the web using structured APIs, some of which do use the Semantic Web standards.  A site that exposes RDF usually has an API that is easy to deal with, which makes our life easier.  For instance, we use geonames.org as one of our geospatial information sources. It is a full-on Semantic Web endpoint, and that makes it easy to deal with.  The more the API declares its data model, the more automated we can make our coupling to it.

Nova Spivack: Siri seems smart, at least about the kinds of tasks it was designed for. How is the knowledge represented in Siri – is it an ontology or something else?

Tom Gruber: Siri's knowledge is represented in a unified modeling system that combines ontologies, inference networks, pattern matching agents, dictionaries, and dialog models.  As much as possible we represent things declaratively (i.e., as data in models, not lines of code).  This is a tried and true best practice for complex AI systems.  This makes the whole system more robust and scalable, and the development process more agile.  It also helps with reasoning and learning, since Siri can look at what it knows and think about similarities and generalizations at a semantic level.

Nova Spivack: Will Siri be part of the Semantic Web, or at least the open linked data Web (by making open API’s, sharing of linked data, RDF, available, etc.)?

Tom Gruber: Siri isn't a source of data, so it doesn't expose data using Semantic Web standards.  In the Semantic Web ecosystem, it is doing something like the vision of a semantic desktop - an intelligent interface that knows about user needs and sources of information to meet those needs, and intermediates.  The original Semantic Web article in Scientific American included use cases that an assistant would do (check calendars, look for things based on multiple structured criteria, route planning, etc.).  The Semantic Web vision focused on exposing the structured data, but it assumes APIs that can do transactions on the data.  For example, if a virtual assistant wants to schedule a dinner it needs more than the information about the free/busy schedules of participants, it needs API access to their calendars with appropriate credentials, ways of communicating with the participants via APIs to their email/sms/phone, and so forth.  Siri is building on the ecosystem of APIs, which are better if they declare the meaning of the data in and out via ontologies.  That is the original purpose of ontologies-as-specification that I promoted in the 1990s - to help specify how to interact with these agents via knowledge-level APIs.

Siri does, however, benefit greatly from standards for talking about space and time, identity (of people, places, and things), and authentication.  As I called for in my Semantic Web talk in 2007, there is no reason we should be string matching on city names, business names, user names, etc.  

All players near the user in the ecommerce value chain get better when the information that the users need can be unambiguously identified, compared, and combined.  Legitimate service providers on the supply end of the value chain also benefit, because structured data is harder to scam than text.  So if some service provider offers a multi-criteria decision making service, say, to help make a product purchase in some domain, it is much easier to do fraud detection when the product instances, features, prices, and transaction availability information are all structured data.

Nova Spivack: Siri appears to be able to handle requests in natural language. How good is the natural language processing (NLP) behind it? How have you made it better than other NLP?

Tom Gruber: Siri's top line measure of success is task completion (not relevance).  A subtask is intent recognition, and subtask of that is NLP.  Speech is another element, which couples to NLP and adds its own issues.  In this context, Siri's NLP is "pretty darn good" -- if the user is talking about something in Siri's domains of competence, its intent understanding is right the vast majority of the time, even in the face of noise from speech, single finger typing, and bad habits from too much keywordese.  All NLP is tuned for some class of natural language, and Siri's is tuned for things that people might want to say when talking to a virtual assistant on their phone.  We evaluate against a corpus, but I don't know how it would compare to standard message and news corpuses using by the NLP research community.

Nova Spivack: Did you develop your own speech interface, or are you using third-party system for that? How good is it? Is it battle-tested?

Tom Gruber: We use third party speech systems, and are architected so we can swap them out and experiment.  The one we are currently using has millions of users and continuously updates its models based on usage.

Nova Spivack: Will Siri be able to talk back to users at any point?

Tom Gruber: It could use speech synthesis for output, for the appropriate contexts.  I have a long standing interest in this, as my early graduate work was in communication prosthesis.  In the current mobile internet world, however, iPhone-sized screens and 3G networks make it possible to do so more much than read menu items over the phone.  For the blind, embedded appliances, and other applications it would make sense to give Siri voice output.

Nova Spivack: Can you give me more examples of how the NLP in Siri works?

Tom Gruber: Sure, here’s an example, published in the Technology Review, that illustrates what’s going on in a typical dialogue with Siri. (Click link to view the table)

Nova Spivack: How personalized does Siri get – will it recommend different things to me depending on where I am when I ask, and/or what I’ve done in the past? Does it learn?

Tom Gruber: Siri does learn in simple ways today, and it will get more sophisticated with time.  As you said, Siri is already personalized based on immediate context, conversational history, and personal information such as where you live.  Siri doesn't forget things from request to request, as do stateless systems like search engines. It always considers the user model along with the domain and task models when coming up with results.  The evolution in learning comes as users have a history with Siri, which gives it a chance to make some generalizations about preferences.  There is a natural progression with virtual assistants from doing exactly what they are asked, to making recommendations based on assumptions about intent and preference. That is the curve we will explore with experience.

Nova Spivack: How does Siri know what is in various external services – are you mining and doing extraction on their data, or is it all just real-time API calls?

Tom Gruber: For its current domains Siri uses dozens of APIs, and connects to them in both realtime access and batch data synchronization modes.  Siri knows about the data because we (humans) explicitly model what is in those sources.  With declarative representations of data and API capabilities, Siri can reason about the various capabilities of its sources at run time to figure out which combination would best serve the current user request.  For sources that do not have nice APIs or expose data using standards like the Semantic Web, we can draw on a value chain of players that do extract structure by data mining and exposing APIs via scraping.

Nova Spivack: Thank you for the information, Siri might actually make me like the iPhone enough to start using one again.

Tom Gruber:   Thank you, Nova, it's a pleasure to discuss this with someone who really gets the technology and larger issues.  I hope Siri does get you to use that iPhone again.  But remember, Siri is just starting out and will sometimes say silly things.  It's easy to project intelligence onto an assistant, but Siri isn't going to pass the Turing Test.  It's just a simpler, smarter way to do what you already want to do.  It will be interesting to see how this space evolves, how people will come to understand what to expect from the little personal assistant in their pocket.

How Social Media Changes Content Distribution from Web Sites to People to Software

I've written a new article about how content distribution has evolved, and where it is heading. It's published here: http://www.siliconangle.com/social-media/content-distribution-is-changing-again/.

Video: My Talk on The Future of Libraries -- "Library 3.0"

If you are interested in semantics, taxonomies, education, information overload and how libraries are evolving, you may enjoy this video of my talk on the Semantic Web and the Future of Libraries at the OCLC Symposium at the American Library Association Midwinter 2009 Conference. This event focused around a dialogue between David Weinberger and myself, moderated by Roy Tennant. We were forutnate to have an audience of about 500 very vocal library directors in the audience and it was an intensive day of thinking together. Thanks to the folks at OCLC for a terrific and really engaging event!

Twine's Explosive Growth

Twine has been growing at 50% per month since launch in October. We've been keeping that quiet while we wait to see if it holds. VentureBeat just noticed and did an article about it. It turns out our January numbers are higher than Compete.com estimates and February is looking strong too. We have a slew of cool viral features coming out in the next few months too as we start to integrate with other social networks. Should be an interesting season.

Fast Company Interview -- "Connective Intelligence"

In this interview with Fast Company, I discuss my concept of "connective intelligence." Intelligence is really in the connections between things, not the things themselves. Twine facilitates smarter connections between content, and between people. This facilitates the emergence of higher levels of collective intelligence.

Interest Networks are at a Tipping Point

UPDATE: There's already a lot of good discussion going on around this post in my public twine.

I’ve been writing about a new trend that I call “interest networking” for a while now. But I wanted to take the opportunity before the public launch of Twine on Tuesday (tomorrow) to reflect on the state of this new category of applications, which I think is quickly reaching its tipping point. The concept is starting to catch on as people reach for more depth around their online interactions.

In fact – that’s the ultimate value proposition of interest networks – they move us beyond the super poke and towards something more meaningful. In the long-term view, interest networks are about building a global knowledge commons. But in the short term, the difference between social networks and interest networks is a lot like the difference between fast food and a home-cooked meal – interest networks are all about substance.

At a time when social media fatigue is setting in, the news cycle is growing shorter and shorter, and the world is delivered to us in soundbytes and catchphrases, we crave substance. We go to great lengths in pursuit of substance. Interest networks solve this problem – they deliver substance.

So, what is an interest network?

In short, if a social network is about who you are interested in, an interest network is about what you are interested in. It’s the logical next step.

Twine for example, is an interest network that helps you share information with friends, family, colleagues and groups, based on mutual interests. Individual “twines” are created for content around specific subjects. This content might include bookmarks, videos, photos, articles, e-mails, notes or even documents. Twines may be public or private and can serve individuals, small groups or even very large groups of members.

I have also written quite a bit about the Semantic Web and the Semantic Graph, and Tim Berners-Lee has recently started talking about what he calls the GGG (Giant Global Graph). Tim and I are in agreement that social networks merely articulate the relationships between people. Social networks do not surface the equally, if not more important, relationships between people and places, places and organizations, places and other places, organization and other organizations, organization and events, documents and documents, and so on.

This is where interest networks come in. It’s still early days to be clear, but interest networks are operating on the premise of tapping into a multi--dimensional graph that manifests the complexity and substance of our world, and delivers the best of that world to you, every day.

We’re seeing more and more companies think about how to capitalize on this trend. There are suddenly (it seems, but this category has been building for many months) lots of different services that can be viewed as interest networks in one way or another, and here are some examples:

• Twine (my site)
• Friendfeed
• Strands
• Intuu
• SocialMedian
• Zimesh
• Popego
• YourVersion

What all of these interest networks have in common is some sort of a bottom-up, user-driven crawl of the Web, which is the way that I’ve described Twine when we get the question about how we propose to index the entire Web (the answer: we don’t. We let our users tell us what they’re most interested in, and we follow their lead).

Most interest networks exhibit the following characteristics as well:

• They have some sort of bookmarking/submission/markup function to store and map data (often using existing metaphors, even if what’s under the hood is new)
• They also have some sort of social sharing function to provide the network benefit (this isn’t exclusive to interest networks, obviously, but it is characteristic)
• And in most cases, interest networks look to add some sort of “smarts” or “recommendations” capability to the mix (that is, you get more out than you put in)

This last bullet point is where I see next-generation interest networks really providing the most benefit over social bookmarking tools, wikis, collaboration suites and pure social networks of one kind or another.

To that end, we think that Twine is the first of a new breed of intelligent applications that really get to know you better and better over time – and that the more you use Twine, the more useful it will become. Adding your content to Twine is an investment in the future of your data, and in the future of your interests.

At first Twine begins to enrich your data with semantic tags and links to related content via our recommendations engine that learns over time. Twine also crawls any links it sees in your content and gathers related content for you automatically – adding it to your personal or group search engine for you, and further fleshing out the semantic graph of your interests which in turn results in even more relevant recommendations.

The point here is that adding content to Twine, or other next-generation interest networks, should result in increasing returns. That’s a key characteristic, in fact, of the interest networks of the future – the idea that the ratio of work (input) to utility (output) has no established ceiling.

Another key characteristic of interest networks may be in how they monetize. Instead of being advertising-driven, I think they will focus more on a marketing paradigm. They will be to marketing what search engines were to advertising. For example, Twine will be monetizing our rich model of individual and group interests, using our recommendation engine. When we roll this capability out in 2009, we will deliver extremely relevant, useful content, products and offers directly to users who have demonstrated they are really interested in such information, according to their established and ongoing preferences.

6 months ago, you could not really prove that “interest networking” was a trend, and certainly it wasn’t a clearly defined space. It was just an idea, and a goal. But like I said, I think that we’re at a tipping point, where the technology is getting to a point at which we can deliver greater substance to the user, and where the culture is starting to crave exactly this kind of service as a way of making the Web meaningful again.

I think that interest networks are a huge market opportunity for many startups thinking about what the future of the Web will be like, and I think that we’ll start to see the term used more and more widely. We may even start to see some attention from analysts -- Carla, Jeremiah, and others, are you listening?

Now, I obviously think that Twine is THE interest network of choice. After all we helped to define the category, and we’re using the Semantic Web to do it. There’s a lot of potential in our engine and our application, and the growing community of passionate users we’ve attracted.

Our 1.0 release really focuses on UE/usability, which was a huge goal for us based on user feedback from our private beta, which began in March of this year. I’ll do another post soon talking about what’s new in Twine. But our TOS (time on site) at 6 minutes/user (all time) and 12 minutes/user (over the last month) is something that the team here is most proud of – it tells us that Twine is sticky, and that “the dogs are eating the dog food.”

Now that anyone can join, it will be fun and gratifying to watch Twine grow.

Still, there is a lot more to come, and in 2009 our focus is going to shift back to extending our Semantic Web platform and turning on more of the next-generation intelligence that we’ve been building along the way. We’re going to take interest networking to a whole new level.

Stay tuned!

Watch My best Talk: The Global Brain is Coming

I've posted a link to a video of my best talk -- given at the GRID '08 Conference in Stockholm this summer. It's about the growth of collective intelligence and the Semantic Web, and the future and role the media. Read more and get the video here. Enjoy!

The World is the Web

I've posted a new article in my public twine about how we are moving from the World Wide Web to the Web Wide World. It's about how the Web is spreading into the physical world, and what this means.

New Video: Leading Minds from Google, Yahoo, and Microsoft talk about their Visions for Future of The Web

Video from my panel at DEMO Fall '08 on the Future of the Web is now available.

I moderated the panel, and our panelists were:

Howard Bloom, Author, The Evolution of Mass Mind from the Big Bang to the 21st Century

Peter Norvig, Director of Research, Google Inc.

Jon Udell, Evangelist, Microsoft Corporation

Prabhakar Raghavan, PhD, Head of Research and Search Strategy, Yahoo! Inc.

The panel was excellent, with many DEMO attendees saying it was the best panel they had ever seen at DEMO.

Many new and revealing insights were provided by our excellent panelists. I was particularly interested in the different ways that Google and Yahoo describe what they are working on. They covered lots of new and interesting information about their thinking. Howard Bloom added fascinating comments about the big picture and John Udell helped to speak about Microsoft's longer-term views as well.

Enjoy!!!

The Future of the Desktop

(Brief excerpt from a new post on my Public Twine -- Go there to read the whole thing and comment on it with me and others...).

I have spent the last year really thinking about the future of the Web. But lately I have been thinking more about the future of the desktop. In particular, here are some questions I am thinking about and some answers I've come up so far.

This is a raw, first-draft of what I think it will be like.

Is the desktop of the future going to just be a web-hosted version of the same old-fashioned desktop metaphors we have today?

No. We've already seen several attempts at doing that -- and they never catch on. People don't want to manage all their information on the Web in the same interface they use to manage data and apps on their local PC.

Partly this is due to the difference in user experience between using real live folders, windows and menus on a local machine and doing that in "simulated" fashion via some Flash-based or HTML-based imitation of a desktop.

Web desktops to-date have simply have been clunky and slow imitations of the real-thing at best. Others have been overly slick. But one thing they all have in common: None of them have nailed it.

Whoever does succeed in nailing this opportunity will have a real shot at becoming a very important player in the next-generation of the Web, Web 3.0.

From the points above it should be clear that I think the future of the desktop is going to be significantly different from what our desktops are like today.

It's going to be a hosted web service

Is the desktop even going to exist anymore as the Web becomes increasingly important? Yes, there is going to be some kind of interface that we consider to be our personal "home" and "workspace" -- but it will become unified across devices.

Currently we have different spaces on different devices (laptop, mobile device, PC). These will merge. In order for that to happen they will ultimately have to be provided as a service via the Web. Local clients may be created for various devices, but ultimately the most logical choice is to just use the browser as the client.

Our desktop will not come from any local device and will always be available to us on all our devices.

 

The skin of your desktop will probably appear within your local device's browser as a completely dynamically hosted web application coming from a remote server. It will load like a Web page, on-demand from a URL.

This new desktop will provide an interface both to your local device, applications and information, as well as to your online life and information.

Instead of the browser running inside, or being launched from, some kind of next-generation desktop web interface technology, it's will be the other way around: The browser will be the shell and the desktop application will run within it either as a browser add-in, or as a web-based application.

The Web 3.0 desktop is going to be completely merged with the Web -- it is going to be part of the Web. There will be no distinction between the desktop and the Web anymore.

Today we think of our Web browser running inside our desktop as an applicaiton. But actually it will be the other way around in the future: Our desktop will run inside our browser as an application.

The focus shifts from information to attention

As our digital lives shift from being focused on the old fashioned desktop (space-based metaphor) to the Web environment we will see a shift from organizing information spatially (directories, folders, desktops, etc.) to organizing information temporally (river of news, feeds, blogs, lifestreaming, microblogging).

Instead of being a big directory, the desktop of the future is going to be more like a Feed reader or social news site. The focus will be on keep up with all the stuff flowing through and what the trends are, rather than on all the stuff that is stored there already.

The focus will be on helping the user to manage their attention rather than just their information.

This is a leap to the meta-level. A second-order desktop. Instead of just being about the information (the first-order), it is going to be about what is happening with the information (the second-order).

It's going to shift us from acting as librarians to acting as daytraders.

Our digital roles are already shifting from effectively acting as "librarians" to becoming more like "daytraders." We are all focusing more on keep up with change than on organizing information today. This will continue to eat up more of our attention...

Read the rest of this on my public Twine! http://www.twine.com/item/11bshgkbr-1k5/the-future-of-the-desktop

Peace in the Middle East: Could Alternative Energy Be the Solution?

I have been thinking about the situation in the Middle East and also the rise of oil prices, peak oil, and the problem of a world economy based on energy scarcity rather than abundance. There is, I believe, a way to solve the problems in the Middle East, and the energy problems facing the world, at the same time. But it requires thinking "outside the box."

Middle Eastern nations must take the lead in freeing the world from dependence on their oil. This is not only their best strategy for the future of their nations and their people, but also it is what will ultimately be best for the region and the whole world.

It is inevitable that someone is going to invent a new technology that frees the world from dependence on fossil fuels. When that happens all oil empires will suddenly collapse. Far-sighted, visionary leaders in oil-producing nations must ensure that their nations are in position to lead the coming non-fossil-fuel energy revolution. This is the wisdom of "cannibalize yourself before someone else does."

Middle Eastern nations should invest more heavily than any other nations in inventing and supplying new alternative energy technologies. For example: hydrogen, solar, biofuels, zero point energy, magnetic power, and the many new emerging alternatives to fossil fuels. This is a huge opportunity for the Middle East not only for economic reasons, but also because it may just be the key to bringing about long-term sustainable peace in the region.

There is a finite supply of oil in the Middle East -- the game will and must eventually end. Are Middle Eastern nations thinking far enough ahead about this or not? There is a tremendous opportunity for them if they can take the initiative on this front and there is an equally tremendous risk if they do not. If they do not have a major stake in whatever comes after fossil fuels, they will be left with nothing when whatever is next inevitably happens (which might be very soon).

Any Middle Eastern leader who is not thinking very seriously about this issue right now is selling their people short. I sincerely advise them to make this a major focus going forward. Not only will this help them to improve quality of life for their people now and in the future, but it is the best way to help bring about world peace. The Middle East has the potential to lead a huge and lucrative global energy Renaissance. All it takes is vision and courage to push the frontier and to think outside of the box.

Video of my Presentation at The Next Web 2008 Conference

Here is the full video of my talk on the Semantic Web at The Next Web 2008 Conference. Thanks to Boris and the NextWeb gang!

Tagging and the Semantic Web: Tags as Objects

John Mills, one of the engineers behind Twine, recently wrote up an interesting article discussing our approach to semantic tags. It's a good read for folks who think about the Semantic Web and tags.

If Social Networks Were Like Cars...

I have been thinking a lot about social networks lately, and why there are so many of them, and what will happen in that space.

Today I had what I think is a "big realization" about this.

Everyone, including myself, seems to think that there is only room for one big social network, and it looks like Facebook is winning that race. But what if that assumption is simply wrong from the start?

What if social networks are more like automobile brands? In other words, there can, will and should be many competing brands in the space?

Social networks no longer compete on terms of who has what members. All my friends are in pretty much every major social network.

I also don't need more than one social network, for the same reason -- my friends are all in all of them. How many different ways do I need to reach the same set of people? I only need one.

But the Big Realization is that no social network satisfies all types of users. Some people are more at home in a place like LinkedIn than they are in Facebook, for example. Others prefer MySpace.  There are always going to be different social networks catering to the common types of people (different age groups, different personalities, different industries, different lifestyles, etc.).

The Big Realization implies that all the social networks are going to be able to interoperate eventually, just like almost all email clients and servers do today. Email didn't begin this way. There were different networks, different servers and different clients, and they didn't all speak to each other. To communicate with certain people you had to use a certain email network, and/or a certain email program. Today almost all email systems interoperate directly or at least indirectly. The same thing is going to happen in the social networking space.

Today we see the first signs of this interoperability emerging as social networks open their APIs and enable increasing integration. Currently there is a competition going on to see which "open" social network can get the most people and sites to use it. But this is an illusion. It doesn't matter who is dominant, there are always going to be alternative social networks, and the pressure to interoperate will grow until it happens. It is only a matter of time before they connect together.

I think this should be the greatest fear at companies like Facebook. For when it inevitably happens they will be on a level playing field competing for members with a lot of other companies large and small. Today Facebook and Google's scale are advantages, but in a world of interoperability they may actually be disadvantages -- they cannot adapt, change or innovate as fast as smaller, nimbler startups.

Thinking of social networks as if they were automotive brands also reveals interesting business opportunities. There are still several unowned opportunities in the space.

Myspace is like the car you have in high school. Probably not very expensive, probably used, probably a bit clunky. It's fine if you are a kid driving around your hometown.

Facebook is more like the car you have in college. It has a lot of your junk in it, it is probably still not cutting edge, but its cooler and more powerful.

LinkedIn kind of feels like a commuter car to me. It's just for business, not for pleasure or entertainment.

So who owns the "adult luxury sedan" category? Which one is the BMW of social networks?

Who owns the sportscar category? Which one is the Ferrari of social networks?

Who owns the entry-level commuter car category?

Who owns equivalent of the "family stationwagon or minivan" category?

Who owns the SUV and offroad category?

You see my point. There are a number of big segments that are not owned yet, and it is really unlikely that any one company can win them all.

If all social networks are converging on the same set of features, then eventually they will be close to equal in function. The only way to differentiate them will be in terms of the brands they build and the audience segments they focus on. These in turn will cause them to emphasize certain features more than others.

In the future the question for consumers will be "Which social network is most like me? Which social network is the place for me to base my online presence?"

Sue may connect to Bob who is in a different social network -- his account is hosted in a different social network. Sue will not be a member of Bob's service, and Bob will not be a member of Sue's, yet they will be able to form a social relationship and communication channel. This is like email. I may use Outlook and you may use Gmail, but we can still send messages to each other.

Although all social networks will interoperate eventually, depending on each person's unique identity they may choose to be based in -- to live and surf in -- a particular social network that expresses their identity, and caters to it. For example, I would probably want to be surfing in the luxury SUV of social networks at this point in my life, not in the luxury sedan, not the racecar, not in the family car, not the dune-buggy. Someone else might much prefer an open source, home-built social network account running on a server they host. It shouldn't matter -- we should still be able to connect, share stuff, get notified of each other's posts, etc. It should feel like we are in a unified social networking fabric, even though our accounts live in different services with different brands, different interfaces, and different features.

I think this is where social networks are heading. If it's true then there are still many big business opportunities in this space.

Associative Search and the Semantic Web: The Next Step Beyond Natural Language Search

Our present day search engines are a poor match for the way that our brains actually think and search for answers. Our brains search associatively along networks of relationships. We search for things that are related to things we know, and things that are related to those things. Our brains not only search along these networks, they sense when networks intersect, and that is how we find things. I call this associative search, because we search along networks of associations between things.

Human memory -- in other words, human search -- is associative. It works by "homing in" on what we are looking for, rather than finding exact matches. Compare this to the the keyword search that is so popular on the Web today and there are obvious differences. Keyword searching provides a very weak form of "homing in" -- by choosing our keywords carefully we can limit the set of things which match. But the problem is we can only find things which contain those literal keywords.

There is no actual use of associations in keyword search, it is just literal matching to keywords. Our brains on the other hand use a much more sophisticated form of "homing in" on answers. Instead of literal matches, our brains look for things things which are associatively connected to things we remember, in order to find what we are ultimately looking for.

For example, consider the case where you cannot remember someone's name. How do you remember it? Usually we start by trying to remember various facts about that person. By doing this our brains then start networking from those facts to other facts and finally to other memories that they intersect.  Ultimately through this process of "free association" or "associative memory" we home in on things which eventually trigger a memory of the person's name.

Both forms of search make use of the intersections of sets, but the associative search model is exponentially more powerful because for every additional search term in your query, an entire network of concepts, and relationships between them, is implied. One additional term can result in an entire network of related queries, and when you begin to intersect the different networks that result from multiple terms in the query, you quickly home in on only those results that make sense. In keyword search on the other hand, each additional search term only provides a linear benefit -- there is no exponential amplification using networks.

Keyword search is a very weak approximation of associative search because there really is no concept of a relationship at all. By entering keywords into a search engine like Google we are simulating an associative search, but without the real power of actual relationships between things to help us. Google does not know how various concepts are related and it doesn't take that into account when helping us find things. Instead, Google just looks for documents that contain exact matches to the terms we are looking for and weights them statistically. It makes some use of relationships between Web pages to rank the results, but it does not actually search along relationships to find new results.

Basically the problem today is that Google does not work the way our brains think. This difference creates an inefficiency for searchers: We have to do the work of translating our associative way of thinking into "keywordese" that is likely to return results we want. Often this requires a bit of trial and error and reiteration of our searches before we get result sets that match our needs.

A recently proposed solution to the problem of "keywordese" is natural language search (or NLP search), such as what is being proposed by companies like Powerset and Hakia. Natural language search engines are slightly closer to the way we actually think because they at least attempt to understand ordinary language instead of requiring keywords. You can ask a question and get answers to that question that make sense.

Natural language search engines are able to understand the language of a query and the language in the result documents in order to make a better match between the question and potential answers. But this is still not true associative search. Although these systems bear a closer resemblance to the way we think, they still do not actually leverage the power of networks -- they are still not as powerful as associative search.

A Few Predictions for the Near Future

This is a five minute video in which I was asked to make some predictions for the next decade about the Semantic Web, search and artificial intelligence. It was done at the NextWeb conference and was a fun interview.

Learning from the Future with Nova Spivack from Maarten on Vimeo.

My Visit to DERI -- World's Premier Semantic Web Research Institute

Earlier this month I had the opportunity to visit, and speak at, the Digital Enterprise Research Institute (DERI), located in Galway, Ireland. My hosts were Stefan Decker, the director of the lab, and John Breslin who is heading the SIOC project.

DERI has become the world's premier research institute for the Semantic Web. Everyone working in the field should know about them, and if you can, you should visit the lab to see what's happening there.

Part of the National University of Ireland, Galway. With over 100 researchers focused solely on the Semantic Web, and very significant financial backing, DERI has, to my knowledge, the highest concentration of Semantic Web expertise on the planet today. Needless to say, I was very impressed with what I saw there. Here is a brief synopsis of some of the projects that I was introduced to:

• Semantic Web Search Engine (SWSE) and YARS, a massively scalable triplestore.  These projects are concerned with crawling and indexing the information on the Semantic Web so that end-users can find it. They have done good work on consolidating data and also on building a highly scalable triplestore architecture.
• Sindice -- An API and search infrastructure for the Semantic Web. This project is focused on providing a rapid indexing API that apps can use to get their semantic content indexed, and that can also be used by apps to do semantic searches and retrieve semantic content from the rest of the Semantic Web. Sindice provides Web-scale semantic search capabilities to any semantic application or service.
• SIOC -- Semantically Interlinked Online Communities. This is an ontology for linking and sharing data across online communities in an open manner, that is getting a lot of traction. SIOC is on its way to becoming a standard and may play a big role in enabling portability and interoperability of social Web data.
• JeromeDL is developing technology for semantically enabled digital libraries. I was impressed with the powerful faceted navigation and search capabilities they demonstrated.
  
• notitio.us. is a project for personal knowledge management of bookmarks and unstructured data.
• SCOT, OpenTagging and Int.ere.st.  These projects are focused on making tags more interoperable, and for generating social networks and communities from tags. They provide a richer tag ontology and framework for representing, connecting and sharing tags across applications.
• Semantic Web Services.  One of the big opportunities for the Semantic Web that is often overlooked by the media is Web services. Semantics can be used to describe Web services so they can find one another and connect, and even to compose and orchestrate transactions and other solutions across networks of Web services, using rules and reasoning capabilities. Think of this as dynamic semantic middleware, with reasoning built-in.
• eLite. I was introduced to the eLite project, a large e-learning initiative that is applying the Semantic Web.
• Nepomuk.  Nepomuk is a large effort supported by many big industry players. They are making a social semantic desktop and a set of developer tools and libraries for semantic applications that are being shipped in the Linux KDE distribution. This is a big step for the Semantic Web!
• Semantic Reality. Last but not least, and perhaps one of the most eye-opening demos I saw at DERI, is the Semantic Reality project. They are using semantics to integrate sensors with the real world. They are creating an infrastructure that can scale to handle trillions of sensors eventually. Among other things I saw, you can ask things like "where are my keys?" and the system will search a network of sensors and show you a live image of your keys on the desk where you left them, and even give you a map showing the exact location. The service can also email you or phone you when things happen in the real world that you care about -- for example, if someone opens the door to your office, or a file cabinet, or your car, etc. Very groundbreaking research that could seed an entire new industry.

In summary, my visit to DERI was really eye-opening and impressive. I recommend that major organizations that want to really see the potential of the Semantic Web, and get involved on a research and development level, should consider a relationship with DERI -- they are clearly the leader in the space.

Insightful Article About Twine

Carla Thompson, an analyst for Guidewire Group, has written what I think is a very insightful article about her experience participating in the early-access wave of the Twine beta.

We are now starting to let the press in and next week we will begin to let waves of people in from our over 30,000 user wait list. We will be letting people into the beta in waves every week going forward.

As Carla notes, Twine is a work in progress and we are mainly focused on learning from our users now. We have lots more to do, but we're very excited about the direction Twine is headed in, and it's really great to see Twine getting so much active use.

How about Web 3G?

I'm here at the BlogTalk conference in Cork, Ireland with a range of bloggers and technologists discussing the emerging social Web. Including myself, Ian Davis and Paul Miller from Talis, there are also a bunch of other Semantic Web folks including Dan Brickley, and a group from DERI Galway.

Over dinner a few of us were discussing the terms "Semantic Web" versus "Web 3.0" and we all felt a better term was needed. After some thinking, Ian Davis suggested "Web 3G." I like this term better than Web 3.0 because it loses the "version number" aspect that so many objected to. It has a familiar ring to it as well, reminding me of the 3G wireless phone initiative. It also suggests Tim Berners-Lee's "Giant Global Graph" or GGG -- a synonym for the Semantic Web. Ian stayed up late and put together a nice blog post about the term, echoing many of my own sentiments about how this term should apply to a decade (the third decade of the Web), rather than to a particular technology.

My Commentary: Radar Networks Raises $13M for Twine

I am pleased to announce that my company Radar Networks, has raised a $13M Series B investment round to grow our product, Twine. The investment comes from Velocity Interactive Group, DFJ, and Vulcan. Ross Levinsohn -- the man who acquired and ran MySpace for Fox Interactive -- will be joining our board. I'm very excited to be working with Ross and to have his help guiding Twine as it grows.

We are planning to use these funds to begin rolling Twine out to broader consumer markets as part of our multi-year plan to build Twine into the leading service for organizing, sharing and discovering information around interests. One of the key themes of Web 3.0 is to be help people make sense of the overwhelming amount of information and change in the online world, and at Twine, we think interests are going to play a key organizing role in that process.

Your interests comprise the portion of your information and relationships that are actually important enough that you want to keep track of them and share them with others. The question that Twine addresses is how to help individuals and groups more efficiently locate, manage and communicate around their interests in the onslaught of online information they have to cope with. The solution to information overload is not to organize all the information in the world (an impossible task), it is to help individuals and groups organize THEIR information (a much more feasible goal).

In March we are going to expand the Twine beta to begin letting more people in. Currently we have around 30,000 people on the wait-list and more coming in steadily. In March we will start letting all of these people in, gradually in waves of a few thousand at a time, and letting them invite their friends in. So to get into Twine you need to sign up on the list on the Twine site, or have a friend who is already in the service invite you in. I look forward to seeing you in Twine!

The last few months of closed beta have been very helpful in getting a lot of useful feedback and testing that has helped us improve the product in many ways. This next wave will be an exciting phase for Twine as we begin to really grow the service with more users. I am sure there will be a lot of great feedback and improvements that result from this.

However, even though we will be letting more people in soon, we are still very much in beta and will be for quite some time to come -- There will still be things that aren't finished, aren't perfect, or aren't there yet -- so your patience will be appreciated as we continue to work on Twine over the coming year. We are letting people in to help us guide the service in the right direction, and to learn from our users. Today Twine is about 10% of what we have planned for it. First we have to get the basics right -- then, in the coming year, we will really start to surface more of the power of the underlying semantic platform. We're psyched to get all this built -- what we have planned is truly exciting!

Artificial Stupidity: The Next Big Thing

There has been a lot of hype about artificial intelligence over the years. And recently it seems there has been a resurgence in interest in this topic in the media. But artificial intelligence scares me. And frankly, I don't need it. My human intelligence is quite good, thank you very much. And as far as trusting computers to make intelligent decisions on my behalf, I'm skeptical to say the least. I don't need or want artificial intelligence.

No, what I really need is artificial stupidity.

I need software that will automate all the stupid things I presently have to waste far too much of my valuable time on. I need something to do all the stupid tasks -- like organizing email, filing documents, organizing folders, remembering things, coordinating schedules, finding things that are of interest, filtering out things that are not of interest, responding to routine messages, re-organizing things, linking things, tracking things, researching prices and deals, and the many other rote information tasks I deal with every day.

The human brain is the result of millions of years of evolution. It's already the most intelligent thing on this planet. Why are we wasting so much of our brainpower on tasks that don't require intelligence? The next revolution in software and the Web is not going to be artificial intelligence, it's going to be creating artificial stupidity: systems that can do a really good job at the stupid stuff, so we have more time to use our intelligence for higher level thinking.

The next wave of software and the Web will be about making software and the Web smarter. But when we say "smarter" we don't mean smart like a human is smart, we mean "smarter at doing the stupid things that humans aren't good at." In fact humans are really bad at doing relatively simple, "stupid" things -- tasks that don't require much intelligence at all.

For example, organizing. We are terrible organizers. We are lazy, messy, inconsistent, and we make all kinds of errors by accident. We are terrible at tagging and linking as well, it turns out. We are terrible at coordinating or tracking multiple things at once because we are easily overloaded and we can really only do one thing well at a time. These kinds of tasks are just not what our brains are good at. That's what computers are for - or should be for at least.

Humans are really good at higher level cognition: complex thinking, decisionmaking, learning, teaching, inventing, expressing, exploring, planning, reasoning, sensemaking, and problem solving -- but we are just terrible at managing email, or making sense of the Web. Let's play to our strengths and use computers to compensate for our weaknesses.

I think it's time we stop talking about artificial intelligence -- which nobody really needs, and fewer will ever trust. Instead we should be working on artificial stupidity. Sometimes the less lofty goals are the ones that turn out to be most useful in the end.

Help us Win! Twine is a Finalist in the Crunchies!

My company's product, Twine.com, has made it to the finalist round in the Crunchies, a new annual tech industry awards competition, under the Best Technical Achievement category. Please help us win by casting your vote for Twine here. Thanks!

UPDATE: It turns out, that for some odd reason the Crunchies allows each voter to vote once per day per category -- in other words, you can vote multiple times in the same category -- one vote per user per day -- so please vote for Twine again if you can.

Jim Hendler on the Chaotic Semantic Web

Jim Hendler, who co-created the OWL language of the Semantic Web has written an interesting post where he comes out with his refutation against criticisms of the Semantic Web.

Powerpoint Deck: Making Sense of the Semantic Web, and Twine

Now that I have been asked by several dozen people for the slides from my talk on "Making Sense of the Semantic Web," I guess it's time to put them online. So here they are, under the Creative Commons Attribution License (you can share it with attribution this site).

You can download the Powerpoint file at the link below:

Download nova_spivack_semantic_web_talk.ppt

Or you can view it right here:

| View | Upload your own

Enjoy! And I look forward to your thoughts and comments.

Quick Video Preview of Twine

The New Scientist just posted a quick video preview of Twine to YouTube. It only shows a tiny bit of the functionality, but it's a sneak peak.

We've been letting early beta testers into Twine and we're learning a lot from all the great feedback, and also starting to see some cool new uses of Twine. There are around 20,000 people on the wait-list already, and more joining every day. We're letting testers in slowly, focusing mainly on people who can really help us beta test the software at this early stage, as we go through iterations on the app. We're getting some very helpful user feedback to make Twine better before we open it up the world.

For now, here's a quick video preview:

True Knowledge is Cool

The most interesting and exciting new app I've seen this month (other than Twine of course!) is a new semantic search engine called True Knowledge. Go to their site and watch their screencast to see what the next generation of search is really going to look like.

True Knowledge is doing something very different from Twine -- whereas Twine is about helping individuals, groups and teams manage their private and shared knowledge, True Knowledge is about making a better public knowledgebase on the Web -- in a sense they are a better search engine combined with a better Wikipedia. They seem to overlap more with what is being done by natural language search companies like Powerset and companies working on public databases, such as Metaweb and Wikia.

I don't yet know whether True Knowledge is supporting W3C open-standards for the Semantic Web, but if they do, they will be well-positioned to become a very central service in the next phase of the Web. If they don't they will just be yet another silo of data -- but a very useful one at least. I personally hope they provide SPARQL API access at the very least. Congratulations to the team at True Knowledge! This is a very impressive piece of work.

Is Google Making Social Networking Middleware?

Google's recent announcement of their OpenSocial API's appears to be a new form of middleware for connecting social networks together. But it's too early to tell, since the technical details are not available yet. The notion of a middleware service for connecting social networks and sharing data between them makes a lot of sense, and if Google has really made it "open" then it could be very useful. The question remains of course, why would Google do this unless there is some way they have a unique benefit from it? My guess is that they will run advertising through this system, and will have unique advantages in their ability to target ads to people based on the social network profiles they can see via this system. We'll have to wait and see what happens, but it is interesting.

From the perspective of Radar Networks and Twine.com, this is a trend we are watching closely. It could be something to integrate with, but until we really see the technical details we'll reserve judgement.

A Video and an Audio Cast About Twine

Last night I saw that the video of my presentation of Twine at the Web 2.0 Summit is online. My session, "The Semantic Edge," featured Danny Hillis of Metaweb demoing Freebase, Barney Pell demoing Powerset, and myself Demoing Twine, followed by a brief panel discussion with Tim O'Reilly (in that order). It's a good panel and I recommend the video, however, the folks at Web 2.0 only filmed the presenters; they didn't capture what we were showing on our screens, so you have to use your imagination as we describe our demos.

An audio cast of one of my presentations about Twine to a reporter was also put online recently, for a more in-depth description.

What a Week!

What a week it has been for Radar Networks. We have worked so hard these last few days to get ready to unveil Twine, and it has been a real thrill to show our work and get such positive feedback and support from the industry, bloggers, the media and potential users.

We really didn't expect so much excitement and interest. In fact we've been totally overwhelmed by the response as thousands upon thousands of people have contacted us in the last 24 hours asking to join our beta, telling us how they would use Twine for their personal information management, their collaboration, their organizations, and their communities. Clearly there is such a strong and growing need out there for the kind of Knowledge Networking capabilities that Twine provides, and it's been great to hear the stories and make new connections with so many people who want our product. We love hearing about your interest in Twine, what you would use it for, what you want it to do, and why you need it! Keep those stories coming. We read them all and we really listen to them.

Today, in unveiling Twine, over five years of R&D, and contributions from dozens of core contributors, a dedicated group of founders and investors, and hundreds of supporters, advisors, friends and family, all came to fruition. As a company, and a team, we achieved an important milestone and we should all take some time to really appreciate what we have accomplished so far. Twine is a truly ambitious and pardigm-shifting product, that is not only technically profound but visually stunning -- There has been so much love and attention to detail in this product.

In the last 6 months, Twine has really matured into a product, a product that solves real and growing needs (for a detailed use-case see this post). And just as our product has matured, so has our organization: As we doubled in size, our corporate culture has become tremendously more interesting, innovative and fun. I could go on and on about the cool things we do as a company and the interesting people who work here. But it's the passion, dedication and talent of this team that is most inspiring. We are creating a team and a culture that truly has the potential to become a great Silicon Valley company: The kind of company that I've always wanted to build.

Although we launched today, this is really just the beginning of the real adventure. There is still much for us to build, learn about, and improve before Twine will really accomplish all the goals we have set out  for it. We have a five-year roadmap. We know this is a marathon, not a sprint and that "slow and steady wins the race." As an organization we also have much learning and growing to do. But this really doesn't feel like work -- it feels like fun -- because we all love this product and this company. We all wake up every day totally psyched to work on this.

It's been an intense, challenging, and rewarding week. Everyone on my team has impressed me and really been at the top of their game. Very few of us got any real sleep, and most of us went far beyond the call of duty. But we did it, and we did it well. As a company we have never cut corners, and we have always preferred to do things the right way, even if the right way is the hard way. But that pays off in the end. That is how great products are built. I really want to thank my co-founders, my team, my investors, advisors, friends, and family, for all their dedication and support.

Today, we showed our smiling new baby to the world, and the world smiled back.

And tonight , we partied!!!

Radar Networks Announces Twine.com

My company, Radar Networks, has just come out of stealth. We've announced what we've been working on all these years: It's called Twine.com. We're going to be showing Twine publicly for the first time at the Web 2.0 Summit tomorrow. There's lot's of press coming out where you can read about what we're doing in more detail. The team is extremely psyched and we're all working really hard right now so I'll be brief for now. I'll write a lot more about this later.

Radar Networks Coming Out of Stealth - Friday, October 19

News Flash!

My company, Radar Networks, is coming out of stealth this Friday, October 19, 2007 at the Web 2.0 Summit, in San Francisco. I'll be speaking on "The Semantic Edge Panel" at 4:10 PM, and publicly showing our Semantic Web online service for the first time. If you are planning to come to Web 2.0, I hope to see you at my panel.

Here's the official Media Alert below:

               

(PRWEB) October 15, 2007 -- At the Web2.0 Summit on October 19th, Radar Networks will announce a revolutionary new service that uses the power of the emerging Semantic Web to enable a smarter way of sharing, organizing and finding information. Founder and CEO Nova Spivack will also give the first public preview of Radar’s application, which is one of the first examples of “Web 3.0” – the next-generation of the Web, in which the Web begins to function more like a database, and software grows more intelligent and helpful.

Join Nova as he participates in “The Semantic Edge” panel discussion with esteemed colleagues including Powerset’s Barney Pell and Metaweb’s Daniel Hillis, moderated by Tim O’Reilly.

Who:   
Radar Networks Founder and CEO Nova Spivack

When:   
Friday, October 19, 2007
4:10 – 4:55 p.m.
   
Where: 
Web2.0 Summit
Palace Hotel
Grand Ballroom
2 New Montgomery Street
San Francisco,  California  94105
   

Understanding The Semantic Web: A Response to Tim O'Reilly's Recent Defense of Web 2.0

Tim O'Reilly, recently blogged another article about Web 2.0 Versus Web 3.0 in which he responded to some of my points about what Web 3.0 is and is not. There are several points in his post that I need to respond to. Here is what I am going to cover in this article:

• Correcting some factual errors in Tim's post about Web 3.0
• Web 2.0 = Industry Renaissance + Marketing Hype
• Web 2.0 was NOT mainly about back-end innovation
• Web 2.0 = The Social Web
• What's After Web 2.0?
• The Semantic Web = The Data Web
• The Value of Open Data
• Key Points of Differentiation

FACTUAL ERRORS THAT NEED TO BE CLARIFIED

Before I address where I agree/disagree with his article, there are some factual errors that should be corrected. Contrary to what Tim states, the term "Web 3.0" was NOT originated by me, and I've never claimed it was. Nor have I designed a definition for it that was tailor-made for what my startup, Radar Networks, is up to.

In fact, the term, Web 3.0, was  independently originated by Jeffrey Zeldman, Tim Berners-Lee, Reed Hastings, John Markoff, and Dan Gillmor. They all had one thing in common however -- a feeling that "Web 2.0" wasn't the end of the story for the Web -- and that something new was brewing.

Personally speaking, the first time I ever heard the term  was actually in John Markoff's New York Times Article on the Intelligent Web, in which he mentioned several companies including my own. That article made the first connection between "Web 3.0" and The Semantic Web -- and since that time many people have come to think of these terms as synonymous.

My only contribution to the whole Web 3.0 debate has been to try to define the term as something that makes more sense, namely, just a decade characterized by a range of technologies that are coming to the fore, which will include The Semantic Web for sure, but will not be limited to it. I've probably put more effort into trying to clarify this term than most, and for that I apologize!

If you are interested in the history, I would encourage you to read the Wikipedia page on the subject for a more detailed account.

WEB 2.0 WAS A RENAISSANCE

I agree with Tim that the Web 2.0 era was a renaissance -- and that there were certain trends and patterns that I think Tim recognized first, and that he has explained better, than just about anyone else. Tim helped the world to see what Web 2.0 was really about -- collective intelligence.

In short Tim deserves a lot of credit for defining Web 2.0 and plugging for that meme incessantly for years. Had he not done that, the industry might not have come back the way it did. I am very thankful for those efforts -- after all they probably got my company funded.

WEB 2.0 = INDUSTRY RENAISSANCE + MARKETING HYPE

Tim is annoyed because he thinks that Web 3.0 is "marketing hype." But let's face it, Web 2.0 is essentially just a buzzword that is nothing other than a marketing term that was designed to promote a conference. Tim even admits this in his article:

Web 2.0 started out as the name of a conference!  And that name had a very specific purpose:  to signify that the web was roaring back after the dot com bust! The 2.0 bit wasn't about the technology, but about the resurgence of interest in the web. When we came up with the idea back in 2003, a lot of programmers were out of work, and there was a general lack of interest in web applications. But we saw a resurgence coming, and designed a conference to tell the story of what was going to be different this time.

Tim is tacitly agreeing with my view on how these terms should be used in his passage above. Web 2.0 is a period of time that began "back in 2003" when there was a resurgence. People who speak of Web 3.0 are also referring to a period of time -- THE NEXT period of time AFTER Web 2.0, in which some kind of discernible new pattern is starting to emerge.

WEB 2.0 WAS NOT MAINLY ABOUT BACK-END INNOVATION

Tim does, I think make a good point about Google being a backend play and being a great Web 2.0 company. But then he goes on to say that "Every major web 2.0 play is a back-end story." I would have to disagree with that statement in just about every case but Google. In fact, every major Web 2.0 play has been built on the LAMP stack, or on Google, or the equivalent, for the most part, not on some fancy new backend technology.

What other Web 2.0 companies can you think of, besides Google, that are truly backend-innovations? Technorati maybe? I'm trying to think of some but the fact is, most Web 2.0 era companies were about getting apps built fast, cheap and simple, by re-using open-source backend components (which were mostly just non-commercial remakes of technologies which had previously existed commercially).

When we look at Web 2.0, the major contributions have been focused towards user-experience. Del.icio.us: A better way to share bookmarks. Flickr: A better way to share photos. YouTube: A better way to share videos. Blogs: A better way to publish user-generated content. Wikis: A better way to share documents. Myspace: A better way to make hideously ugly homepages. LinkedIn: A better way to do social networking. Facebook: a better way to keep in touch with friends. Etc.

The backends of all of these kinds of apps (other than perhaps Facebook, lately) have been nothing to write home about. It's what they did on the front-end that really sang. And not just the front-end in terms of user-interface, but also in terms of design patterns and approach.

WEB 2.0 = THE SOCIAL WEB

Web 2.0 has been about "the social Web" in my opinion.That's the big distinction. Google is really a social network measurement algorithm based on similar ideas that had been brewing in the field of bibliometrics for a decade prior, at least.

Most of the big Web 2.0 successes have also been social in one form or another. They have been about harnessing social networks, user-generated content, folksonomies, the wisdom of crowds, and collective intelligence. That's the real contribution of this era to the Web. Web 2.0 is the Social Web.

WEB 3.0 IS WHAT'S AFTER THE SOCIAL WEB

If Web 2.0 has largely been about new social applications of existing back-end technologies, the question is what comes next? Clearly there is still a lot of room for improving on the ideas of Web 2.0. But that's not NEW.

When we think about what would actually be new, it would have to be a characteristic shift that would enable a lot of innovation, new capabilities, new kinds of applications, new design patterns, new technologies. The Semantic Web certainly fits the bill. But it's not the only technology that will matter.

Tim seems to think that Web 3.0 should be a completely new take on the Web. He says:

So for starters, I'd say that for "Web 3.0" to be meaningful we'll need to see a serious discontinuity from the previous generation of technology. That might be another bust and resurgence, or more likely, it will be something qualitatively different. I like Stowe Boyd's musings on the subject:

Personally, I feel the vague lineaments of something beyond Web 2.0, and they involve some fairly radical steps. Imagine a Web without browsers. Imagine breaking completely away from the document metaphor, or a true blurring of application and information. That's what Web 3.0 will be, but I bet we will call it something else.

I'm with Stowe. There's definitely something new brewing, but I bet we will call it something other than Web 3.0. And it's increasingly likely that it will be far broader and more pervasive than the web, as mobile technology, sensors, speech recognition, and many other new technologies make computing far more ambient than it is today.

Ironically, Stowe suggests that Web 3.0 will be a "blurring of application and information" -- that is exactly what the Semantic Web is in fact. Tim then goes on to agree, stating that he thinks there is going to be something far more pervasive than the web, etc. Again that's another point in support of The Semantic Web -- which in fact is not just about the Web, but all information and all applications. It's a better way to handle the DATA that they use.

The Semantic Web "blurs applications and information" because it starts to move the semantics out of applications and into the information itself. Applications can therefore be smarter while also being thinner -- more of what used to be application logic moves into the data itself; the data becomes "smarter."

THE SEMANTIC WEB IS THE DATA WEB

The Semantic Web is not about AI or anything fancy like that, it is really just about data. Another and perhaps better name for it would be "The Data Web."

RDF enables something as potentially important as HTML. Just as HTML enabled a universally reusable Web of content, RDF enables the Data Web, a universally reusable Web of data. The Web browser is a universal client for content, but not really for data. Web browsers can render any content written in HTML in a standard way. That was a big leap back in the early 1990's. Previously each type of content required a different application to view it. The browser unified them all -- this separation of rendering from data made life easier for programmers, and for end-users. A single tool could render any data because the data carried metadata (HTML) that described how to render it.

But currently although browsers can render the formatting and layout of data, they don't know anything about the  meaning of the data, unless they are explicitly programmed to do so. The same is true for all applications today -- they have to be explicitly programmed in advance to interpret each kind of data they need to use.

The Semantic Web provides a solution for this problem that is analogous to what HTML did for content -- RDF and OWL provide a standard way to describe the meaning of any data structure, such that any application that speaks these languages can correctly interpret the meaning without having to have been explicitly programmed to do so in advance.

In other words, the Semantic Web offers the promise of a "universal client for data." That would be a big improvement over how applications are written and how data is managed and stored today. It's a significant back-end level upgrade, and it requires not only that data is represented differently, but new tools for managing it (new kinds of databases, new API's, new forms of search, etc.).

There's also an added benefit to the Semantic Web -- one which is usually OVER-emphasized -- and that is reasoning. The rich semantics of the RDF and OWL languages enable metadata that not only describes the meaning of data, but also the logical relationships between data and various concepts.

This richer metadata can be used to support machine reasoning, such as simple inferencing, across data on the Web. That's powerful and will enable a whole new generation of smarter applications and services -- the so-called "Intelligent Web" -- but it's not the main point! I think that is rather far off in the future still. Today, just making the "Data Web" would be a huge innovation. Transforming the Web from a distributed file-server to a distributed database is a huge enough step on its own.

THE REAL POINT OF THE SEMANTIC WEB = OPEN DATA

The fact is, while I have great respect for Tim as a thinker, I don't think he truly "gets" the Semantic Web yet. In fact, he consistently misses the real point of where these technologies add value, and instead gets stuck on edge-cases (like artificial intelligence) that all of us who are really working on these technologies actually don't think about at all. We don't care about reasoning or artificial intelligence, we care about OPEN DATA.

From what I can see, Tim thinks the Semantic Web is some kind of artificial intelligence system. If that is the case, he's completely missing the point. Yes, of course it enables better, smarter applications. But it's fundamentally NOT about AI and it never was. It's about OPEN DATA. The Semantic Web should be renamed to simply The Data Web.

Watch how Tim Berners-Lee talks about it these days as a "universal data bus", for example in this video. That would be a much more accurate description of where the real thrust of these technologies is headed.

The real benefit of RDF and OWL is that they disrupt the idea of what a database is -- making it something that is much more open, more richly described, more decentralized, more extensible, more maintainable, more portable, more precisely definable, and more useful.

If you really look at RDF, OWL, and in particular GRDDL and SPARQL, it becomes crystal clear that this is a set of technologies about freeing data from platform and application lock-in. That is really what the Semantic Web is for.

The benefit of Open Data is that it enables databases and the data they contain to be designed, shared, and mashed-up in a totally bottom-up, user-driven, Web 2.0 manner. This is in fact collective-intelligence applied to data.

I'm really looking forward to the day when Tim O'Reilly sees that the true value of the RDF and OWL, not to mention SPARQL and GRDDL (aka "The Semantic Web") is Open Data. I think when he finally "gets it" he will actually be quite excited about it. When he sees how these technologies enable a bottom-up, distributed, user-generated open Web of Data, I think he will have an epiphany.

KEY POINTS OF DIFFERENTIATION

There are few important distinctions that Tim is starting to agree with however:

1. Web 3.0 is not equivalent the Semantic Web. Whatever Web 3.0 is, it is not only going to be limited to the Semantic Web. Many other technologies are emerging that are also going to matter. Collectively they will bring about a shift that is fundamentally disruptive and qualitatively different from the Web 2.0 content and applications we know today.
2. The Semantic Web is completely orthogonal to the issue collective intelligence. The Semantic Web is no more or less about collective intelligence than Web 2.0. In fact, it has the potential to provide a better medium for collective intelligence. Services like Metaweb's Freebase, and the new service that Radar Networks will be launching soon, both help to illustrate this point.
3. The Semantic Web is precisely the kind shift that can define a new era of the Web.  Transforming the Web from a fileserver model to a database model is a big, fundamental shift. The kind of shift that qualifies for a new term like "Web 3.0" if that is necessary.
4. This is only the very beginning. Let's not Over-Hype The Semantic Web. The Semantic Web has the potential to really upgrade the back-end of the Web and all the applications and content on it. But it has not done that YET. It is going to be an incremental process -- just like Web 2.0 was. Web 3.0 (2010 - 2020) has not really started yet. We are just in 2007 and there is much to be done to get there. We should be careful not to over-hype what we are doing.  The real value of the Semantic Web will take many years to truly emerge. The "proof is in the pudding" -- but the pudding is going to take some time before it is fully ready for mass-consumption.

In closing I want to also point out that while I am enthusiastic about The Semantic Web, I am not a purist -- I actually believe in using what works best, rather than being stuck on some ideology. To that extent, Radar Networks is making use of the Semantic Web, where appropriate, but we also use other techniques and technologies. We're pragmatists at heart.

Web 3.0 -- The Best Official Definition Imaginable

Jason just blogged his take on an official definition of "Web 3.0" -- in his case he defines it as better content, built using Web 2.0 technologies. There have been numerous responses already, but since I am one of the primary co-authors of the Wikipedia page on the term Web 3.0, I thought I should throw my hat in the ring here.

Web 3.0, in my opinion is best defined as the third-decade of the Web (2009 - 2019), during which time several key technologies will become widely used. Chief among them will be RDF and the technologies of the emerging Semantic Web. While Web 3.0 is not synonymous with the Semantic Web (there will be several other important technology shifts in that period), it will be largely characterized by semantics in general.

Web 3.0 is an era in which we will upgrade the back-end of the Web, after a decade of focus on the front-end (Web 2.0 has mainly been about AJAX, tagging, and other front-end user-experience innovations.) Web 3.0 is already starting to emerge in startups such as my own Radar Networks (our product is Twine) but will really become mainstream around 2009.

Why is defining Web 3.0 as a decade of time better than just about any other possible definition of the term? Well for one thing, it's a definition that can't easily be co-opted by any company or individual around some technology or product. It's also a completely unambiguous definition -- it refers to a particular time period and everything that happens in Web technology and business during that period. This would end the debate about what the term means and move it to something more useful to discuss: What technologies and trends will actually become important in the coming decade of the Web?

It's time to once again pull out my well-known graph of Web 3.0 to illustrate what I mean...

(Click the thumbnail for a larger, reusable version)

I've written fairly extensively on the subjects of defining Web 3.0 and the Semantic Web. Here are some links to get you started if you want to dig deeper:

The Semantic Web: From Hypertext to Hyperdata
The Meaning and Future of the Semantic Web
How the WebOS Evolves
Web 3.0 Roundup
Gartner is Wrong About Web 3.0
Beyond Keyword (And Natural Language) Search
Enriching the Connections of the Web: Making the Web Smarter
Next Step for the Web
Doing for Data What HTML Did for Documents

My Burma Meme Spreads to 17,000 Web Pages in just one week!

I've been tracking the progress of my Burma protest meme. In just under one week it has spread to almost 17,000 web pages and it continues to grow. (For the latest number, click here). It's great to see the blogosphere pick this up, and I'm glad to be able to do something to help raise awareness of this important human rights issue.

This meme is also an example of an interesting new way to spread content on the Web -- whether for a protest or an ad or any other kind of announcement. It's kind of like a chain letter, but via weblogs. There are many different ways to structure these memes with varying levels of virality and benefit to participants. For some earlier work I've done on meme propagation on the Web see my GoMeme experiments from a few years ago. In those experiments I created a series of memes that spread widely through the blogosphere, based on different viral messages, surveys, and benefits to participants. Other people then tracked the statistics of the memes as they spread. It turned out to be a very interesting study of superdistribution of content along social networks.

Gartner is Wrong about Web 3.0

I have a lot of respect for the folks at Gartner, but their recent report in which they support the term "Web 2.0" yet claim that the term "Web 3.0" is just a marketing ploy, is a bit misguided.

In fact, quite the opposite is true.

The term Web 2.0 is in fact just a marketing ploy. It has only come to have something resembling a definition over time. Because it is in fact so ill-defined, I've suggested in the past that we just use it to refer to a decade: the second decade of the Web (2000 - 2010). After all there is no actual technology that is called "Web 2.0" -- at best there are a whole slew of things which this term seems to label, and many of them are design patterns, not technologies. For example "tagging" is not a technology, it is a design pattern. A tag is a keyword, a string of text -- there is not really any new technology there. AJAX is also not a technology in its own right, but rather a combination of technologies and design patterns, most of which existed individually before the onset of what is called Web 2.0.

In contrast, the term Web 3.0 actually does refer to a set of new technologies, and changes they will usher in during the third decade of the Web (2010 - 2020). Chief among these is the Semantic Web. The Semantic Web is actually not one technology, but many. Some of them such as RDF and OWL have been under development for years, even during the Web 2.0 era, and others such as SPARQL and GRDDL are recent emerging standards. But that is just the beginning. As the Semantic Web develops there will be several new technology pieces added to the puzzle for reasoning, developing and sharing open rule definitions, handling issues around trust, agents, machine learning, ontology development and integration, semantic data storage, retrieval and search, and many other subjects.

Essentially, the Semantic Web enables the gradual transformation of the Web into a database. This is a profound structural change that will touch every layer of Web technology eventually. It will transform database technology, CMS, CRM, enterprise middleware, systems integration, development tools, search engines, groupware, supply-chain integration, and all the other topics that Gartner covers.

The Semantic Web will manifest in several ways. In many cases it will improve applications and services we already use. So for example, we will see semantic social networks, semantic search, semantic groupware, semantic CMS, semantic CRM, semantic email, and many other semantic versions of apps we use today. For a specific example, take social networking. We are seeing much talk about "opening  up the social graph" so that social networks are more connected and portable. Ultimately to do this right, the social graph should be represented using Semantic Web standards, so that it truly is not only open but also easily extensible and mashable with other data. 

Web 3.0 is not ONLY the Semantic Web however. Other emerging technologies may play a big role as well. Gartner seems to think Virtual Reality will be one of them. Perhaps, but to be fair, VR is actually a Web 1.0 phenomenon. It's been around for a long time, and it hasn't really changed that much. In fact the folks at the MIT Media Lab were working on things that are still far ahead of Second Life, even back in the early 1990's.

So what other technologies can we expect in Web 3.0 that are actually new? I expect that we will have a big rise in "cloud computing" such as open peer-to-peer grid storage and computing capabilities on the Web -- giving any application essentially as much storage and computational power as needed for free or a very low cost. In the mobile arena we will see higher bandwidth, more storage and more powerful processors in mobile devices, as well as powerful built-in speech recognition, GPS and motion sensors enabling new uses to emerge. I think we will also see an increase in the power of personalization tools and personal assistant tools that try to help users manage the complexity of their digital lives. In the search arena, we will see search engines get smarter -- among other things they will start to not only answer questions, but they will accept commands such as "find me a cheap flight to NYC" and they will learn and improve as they are used. We will also see big improvements in integration and data and account portability between different Web applications. We will also see a fundamental change in the database world as databases move away from the relational model and object model, towards the associative model of data (graph databases and triplestores).

In short, Web 3.0 is about hard-core new technologies and is going to have a much greater impact on enterprise IT managers and IT systems than Web 2.0. But ironically, it may not be until Web 4.0 (2020 - 2030) that Gartner comes to this conclusion!

The Semantic Web, Collective Intelligence and Hyperdata

I'm posting this in response to a recent post by Tim O'Reilly which focused on disambiguating what the Semantic Web is and is not, as well as the subject of Collective Intelligence. I generally agree with Tim's post, but I do have some points I would add by way of clarification. In particular, in my opinion,  the Semantic Web is all about collective intelligence, on several levels. I would also suggest that the term "hyperdata" is a possibly useful way to express what the Semantic Web is really all about.

What Makes Something a Semantic Web Application?

I agree with Tim that the term "Semantic Web" refers to the use of a particular set of emerging W3C open standards. These standards include RDF, OWL, SPARQL, and GRDDL. A key requirement for an application to have "Semantic Web inside" so to speak, is that it makes use of or is compatible with, at the very least, basic RDF. Another alternative definition is that for an application to be "Semantic Web" it must make at least some use of an ontology, using a W3C standard for doing so.

Semantic Versus Semantic Web

Many applications and services claim to be "semantic" in one manner or another, but that does not mean they are "Semantic Web." Semantic applications include any applications that can make sense of meaning, particularly in language such as unstructured text, or structured data in some cases. By this definition, all search engines today are somewhat "semantic" but few would qualify as "Semantic Web" apps.

The Difference Between "Data On the Web" and a "Web of Data"

The Semantic Web is principally about working with  data in a new and hopefully better way, and making that data available on the Web if desired in an open fashion such that other applications can understand and reuse it more easily. We call this idea "The Data Web" -- the notion is that we are transforming the Web from a distributed file server into something that is more like a distributed database.

Instead of the basic objects being web pages, they are actually pieces of data (triples) and records formed from them (sets, trees, graphs or objects comprised of triples). There can be any number of triples within a Web page, and there can also be triples on the Web that do not exist within Web pages at all -- they can come directly from databases for example.

One might respond to this by noting that there is already a lot of data on the Web, in XML and other formats -- how is the Semantic Web different from that? What is the difference between "Data on the Web" and the idea of "The Data Web?"

The best answer to this question that I have heard was something that Dean Allemang said at a recent Semantic Web SIG in Palo Alto. Dean said, "Sure there is data on the Web, but it's not actually a web of data."  The difference is that in the Semantic Web paradigm, the data can be linked to other data in other places, it's a web of data, not just data on the Web.

I call this concept of interconnected data, "Hyperdata." It does for data what hypertext did for text. I'm probably not the originator of this term, but I think it is a very useful term and analogy for explaining the value of the Semantic Web.

Another way to think of it is that the current Web is a big graph of interconnected nodes, where the nodes are usually HTML documents, but in the Semantic Web we are talking about a graph of interconnected data statements that can be as general or specific as you want. A data record is a set of data statements about the same subject, and they don't have to live in one place on the network -- they could be spread over many locations around the Web.

A statement to the effect of "Sue lives in Palo Alto" could exist on site A, refer to a URI for a statement defining Sue on site B, a URI for a statement that defines "lives in" on site C, and a URI for a statement defining "Palo Alto" on site D. That's a web of data. What's cool is that anyone can potentially add statements to this web of data, it can be completely emergent.

The Semantic Web is Built by and for Collective Intelligence

This is where I think Tim and others who think about the Semantic Web may be missing an essential point. The Semantic Web is in fact highly conducive to "collective intelligence." It doesn't require that machines add all the statements using fancy AI. In fact, in a next-generation folksonomy, when tags are created by human users, manually, they can easily be encoded as RDF statements. And by doing this you get lots of new capabilities, like being able to link tags to concepts that define their meaning, and to other related tags.

Humans can add tags that become semantic web content. They can do this manually or software can help them. Humans can also fill out forms that generate RDF behind the scenes, just as filling out a blog posting form generates HTML, XML, ATOM etc. Humans don't actually write all that code, software does it for them, yet blogging and wikis for example are considered to be collective intelligence tools.

So the concept of folksonomy and tagging is truly orthogonal to the Semantic Web. They are not mutually exclusive at all. In fact the Semantic Web -- or at least "Semantic Web Lite" (RDF + only basic use of OWL + basic SPARQL) is capable of modelling and publishing any data in the world in a more open way.

Any application that uses data could do everything it does using these technologies. Every single form of social, user-generated content and community could, and probably will, be implemented using RDF in one manner or another within the next decade or so. And in particular, RDF and OWL + SPARQL are ideal for social networking services -- the data model is a much better match for the structure of the data and the network of users and the kinds of queries that need to be done.

Folktologies

This notion that somehow the Semantic Web is not about folksonomy needs to be corrected. For example, take Metaweb's Freebase. Freebase is what I call a "folktology" -- it's an emergent, community generated ontology. Users collaborate to add to the ontology and the knowledge base that is populated within it. That's a wonderful example of collective intelligence, user generated content, and semantics (although technically to my knowledge they are not using RDF for this, their data model is from what I can see functionally equivalent and I would expect at least a SPARQL interface from them eventually).

But that's not all -- check out TagCommons and this Tag Ontology discussion, and also the SKOS ontology -- all of which are working on semantic ways of characterizing simple tags in order to enrich folksonomies and enable better collective intelligence.

There are at least two other places where the Semantic Web naturally leverages and supports collective intelligence. The first is the fact that people and software can generate triples (people could do it by hand, but generally they will do it by filling out Web forms or answering questions or dialog boxes etc.) and these triples can live all over the Web, yet interconnect or intersect (when they are about the same subjects or objects).

I can create data about a piece of data you created, for example to state that I agree with it, or that I know something else about it. You can create data about my data. Thus a data-set can be generated in a distributed way -- it's not unlike a wiki for example. It doesn't have to work this way, but at least it can if people do this.

The second point is that OWL, the ontology language, is designed to support an infinite number of ontologies -- there doesn't have to be just one big ontology to "rule them all." Anyone can make a simple or complex ontology and start to then make data statements that refer to it. Ontologies can link to or include other ontologies, or pieces of them, to create bigger distributed ontologies that cover more things.

This is kind of like not only mashing up the data, but also mashing up the schemas too. Both of these are examples of collective intelligence. In the case of ontologies, this is already happening, for example many ontologies already make use of other ontologies like the Dublin Core and Foaf.

The point here is that there is in fact a natural and very beneficial fit between the technologies of the Semantic Web and what Tim O'Reilly defines Web 2.0 to be about (essentially collective intelligence). In fact the designers of the underlying standards of the Semantic Web specifically had "collective intelligence" in mind when they came up with these ideas. They were specifically trying to rectify several problems in the closed, data-silo world of old fashioned databases. The big motivation was to make data more integrated, to enable applications to share data more easily, and to be able to build data with other data, and to build schemas with other schemas. It's all about enabling connections and network effects.

Now, whether people end up using these technologies to do interesting things that enable human-level collective intelligence (as opposed to just software level collective intelligence) is an open question. At least some companies such as my own Radar Networks and Metaweb, and Talis (thanks, Danny), are directly focused on this, and I think it is safe to say this will be a big emerging trend. RDF is a great fit for social and folksonomy-based applications.

Web 3.0 and the concept of "Hyperdata"

Where Tim defines Web 2.0 as being about collective intelligence generally, I would define Web 3.0 as being about "connective intelligence." It's about connecting data, concepts, applications and ultimately people. The real essence of what makes the Web great is that it enables a global hypertext medium in which collective intelligence can emerge. In the case of Web 3.0, which begins with the Data Web and will evolve into the full-blown Semantic Web over a decade or more, the key is that it enables a global hyperdata medium (not just hypertext).

As I mentioned above, hyperdata is to data what hypertext is to text. Hyperdata is a great word -- it is so simple and yet makes a big point. It's about data that links to other data. It does for data what hypertext does for text. That's what RDF and the Semantic Web are really all about. Reasoning is NOT the main point (but is a nice future side-effect...). The main point is about growing a web of data.

Just as the Web enabled a huge outpouring of collective intelligence via an open global hypertext medium, the Semantic Web is going to enable a similarly huge outpouring of collective knowledge and cognition via a global hyperdata medium. It's the Web, only better.

A Bottle That Purifies Enough Water for a Year

This is a really great invention -- a hand held water bottle that can purify a year's worth of water. It removes not only parasites and bacteria, but also viruses. It was just announced recently at a defense industry tradeshow and was a big hit among military commanders who need a better way to get water to their troops. Beyond that it could be a lifesaver in disaster areas and in developing countries where finding clean water is a daily struggle.

Open Source Projects for Extracting Data and Metadata from Files & the Web

I've been looking around for open-source libraries (preferably in Java, but not required) for extracting data and metadata from common file formats and Web formats. One project that looks very promising is Aperture. Do you know of any others that are ready or almost ready for prime-time use? Please let me know in the comments! Thanks.

Microsoft Astoria Not Supporting RDF

Microsoft's Astoria project has decided to make RDF a lower priority and is not supporting it for now. So much for Microsoft participating in the Semantic Web.

The Astoria project page describes the project thusly:

The goal of Microsoft Codename Astoria is to enable applications to expose data as a data service that can be consumed by web clients within a corporate network and across the internet. The data service is reachable over HTTP, and URIs are used to identify the various pieces of information available through the service. Interactions with the data service happens in terms of HTTP verbs such as GET, POST, PUT and DELETE, and the data exchanged in those interactions is represented in simple formats such as XML and JSON.

That sounds like a perfect description of a Web 3.0 service-- and a perfect use-case for RDF and SPARQL!  But the team at Astoria doesn't seem to feel anyone is actually using RDF enough to warrant supporting that open standard. Well, perhaps among Microsoft's customers that is true enough. But on the other hand, this is a missed opportunity to exercise forward-thinking leadership that will pay dividends to Microsoft in the future when millions of developers and applications will be using RDF as their common language. When is that future? Perhaps as soon as 2 - 3 years. Certainly not more than 5 years.

Welcome to the age of openness: The more open your platform and data, the more developers will want to use it. RDF is one of the ways to make data more open, and supporting it in a data-web platform makes good sense. But it's hard to teach a an old god new tricks!

Rogue Nodes Turn Tor Anonymizer Into Private Listening Post

A security researcher has figured out a novel way to compromise the security of messages traveling in the Tor anonymizer network. Messages in the Tor network are encrypted as they travel from node to node to their final destination. But the last node has to decrypt the messages before it can deliver them to their final destination on the Internet. Many Tor users mistakenly believe their message remains encrypted through the entire Tor network, when in fact this is not the case: the last node must decrypt them. The researcher simply ran a few of these nodes and was able to read all unencrypted last-node traffic that came through them. This included sensitive communications of many government embassies around the world. The researcher believes that intelligence agencies around the world are already taking advantage of this weakness to eavesdrop on Tor traffic. Interestingly, when he pointed this security hole out to some of the embassies that were sending non-secure message they didn't respond or even appear to understand the problem. Read more here.

DBpedia.org is Among the Coolest Semantic Web Datasets I've Seen

I've been poking around in the DBpedia, and I'm amazed at the progress. It is definitely one of the coolest (launched) example of the Semantic Web I've seen. It's going to be a truly useful resource to everyone. If you haven't heard of it yet, check it out!

Burning Salt Water -- A New Form of Fuel

Inventor, John Kanzius, has figured out a way to burn salt water. This could provide a clean, naturally available alternative fuel source. Salt water is one of the most abundant natural resources on our planet. Here's a video.

Virtual Out of Body Experiences

A very cool experiment in virtual reality has shown it is possible to trick the mind into identifying with a virtual body:

Through these goggles, the volunteers could see a camera view of their own back - a three-dimensional "virtual own body" that appeared to be standing in front of them.

When the researchers stroked the back of the volunteer with a pen, the volunteer could see their virtual back being stroked either simultaneously or with a time lag.

The volunteers reported that the sensation seemed to be caused by the pen on their virtual back, rather than their real back, making them feel as if the virtual body was their own rather than a hologram.

Volunteers

Even when the camera was switched to film the back of a mannequin being stroked rather than their own back, the volunteers still reported feeling as if the virtual mannequin body was their own.

And when the researchers switched off the goggles, guided the volunteers back a few paces, and then asked them to walk back to where they had been standing, the volunteers overshot the target, returning nearer to the position of their "virtual self".

This has implications for next-generation video games and virtual reality. It also has interesting implications for consciousness studies in general.

Plans for a Lunar Ark to Save Humanity

Researchers at the International Space University (ISU), of which I am an alumnus, are proposing an interesting initiative to build an ark on the moon to preserve human civilization and biodiversity, and the Internet, in the event of a catastrophe on earth, such as a comet impact, nuclear war, etc. This project is similar to what I proposed in my Genesis Project posting in 2003.

Humans are just beginning to send trinkets of technology and culture into space. NASA's recently launched Phoenix Mars Lander, for example, carries a mini-disc inscribed with stories, art, and music about Mars.

The Phoenix lander is a "precursor mission" in a decades-long project to transplant the essentials of humanity onto the moon and eventually Mars. (See a photo gallery about the Phoenix mission.)

The International Space University team is now on a more ambitious mission: to start building a "lunar biological and historical archive," initially through robotic landings on the moon.

Laying the foundation for "rebuilding the terrestrial Internet, plus an Earth-moon extension of it, should be a priority," Burke said.

Knowledge Networking

I've been thinking for several years about Knowledge Networking. It's not a term I invented, it's been floating around as a meme for at least a decade or two. But recently it has started to resurface in my own work.

So what is a knowledge network? I define a knowledge network as a form of collective intelligence in which a network of people (two or more people connected by social-communication relationships) creates, organizes, and uses a collective body of knowledge. The key here is that a knowledge network is not merely a site where a group of people work on a body of information together (such as the wikipedia), it's also a social network -- there is an explicit representation of a social relationship within it. So it's more like a social network than for example a discussion forum or a wiki.

I would go so far as to say that knowledge networks are the third-generation of social software. (Note this is based in-part on ideas that emerged in conversations I have had with Peter Rip, so this also his idea):

• First-generation social apps were about communication (eg. messaging such as Email, discussion boards, chat rooms, and IM)
• Second-generation social apps were about people and content (eg. Social networks, social media sharing, user-generated content)
• Third-generation social apps are about relationships and knowledge  (eg. Wikis, referral networks, question and answer systems, social recommendation systems, vertical knowledge and expertise portals, social mashup apps, and coming soon, what we're building at Radar Networks)

Just some thoughts on a Saturday morning...

Networked Genome -- New Finding Shatters Current Thinking

A new finding has discovered that the human genome may be highly networked. That is, genes do not operate in isolation, but rather they are networked together in a far more complex ecosystem than previously thought. It may be impossible to separate one gene from another in fact. This throws into question not only our understanding of genetics and the human genome, but also the whole genomics industry, which relies heavily on the idea that genes and drugs based on them can be patented:

The principle that gave rise to the biotech industry promised benefits that were equally compelling. Known as the Central Dogma of molecular biology, it stated that each gene in living organisms, from humans to bacteria, carries the information needed to construct one protein.

The scientists who invented recombinant DNA in 1973 built their innovation on this mechanistic, "one gene, one protein" principle.

Because donor genes could be associated with specific functions, with discrete properties and clear boundaries, scientists then believed that a gene from any organism could fit neatly and predictably into a larger design - one that products and companies could be built around, and that could be protected by intellectual-property laws.

This presumption, now disputed, is what one molecular biologist calls "the industrial gene."

"The industrial gene is one that can be defined, owned, tracked, proven acceptably safe, proven to have uniform effect, sold and recalled," said Jack Heinemann, a professor of molecular biology in the School of Biological Sciences at the University of Canterbury in New Zealand and director of its Center for Integrated Research in Biosafety.

In the United States, the Patent and Trademark Office allows genes to be patented on the basis of this uniform effect or function. In fact, it defines a gene in these terms, as an ordered sequence of DNA "that encodes a specific functional product."

In 2005, a study showed that more than 4,000 human genes had already been patented in the United States alone. And this is but a small fraction of the total number of patented plant, animal and microbial genes.

In the context of the consortium's findings, this definition now raises some fundamental questions about the defensibility of those patents.

If genes are only one component of how a genome functions, for example, will infringement claims be subject to dispute when another crucial component of the network is claimed by someone else?

Might owners of gene patents also find themselves liable for unintended collateral damage caused by the network effects of the genes they own?

And, just as important, will these not-yet-understood components of gene function tarnish the appeal of the market for biotech investors, who prefer their intellectual property claims to be unambiguous and indisputable?

While no one has yet challenged the legal basis for gene patents, the biotech industry itself has long since acknowledged the science behind the question.

"The genome is enormously complex, and the only thing we can say about it with certainty is how much more we have left to learn," wrote Barbara Caulfield, executive vice president and general counsel at the biotech pioneer Affymetrix, in a 2002 article on Law.com called "Why We Hate Gene Patents."

"We're learning that many diseases are caused not by the action of single genes, but by the interplay among multiple genes," Caulfield said. She noted that just before she wrote her article, "scientists announced that they had decoded the genetic structures of one of the most virulent forms of malaria and that it may involve interactions among as many as 500 genes."

Even more important than patent laws are safety issues raised by the consortium's findings. Evidence of a networked genome shatters the scientific basis for virtually every official risk assessment of today's commercial biotech products, from genetically engineered crops to pharmaceuticals.

Read the rest here