This Blog has Moved to NovaSpivack.com

Proof of Chinese Agent Provacateurs Dressing As Tibetan Monks?

You have to see this image.

Twine and Linked Data on the Semantic Web

Tim Berners-Lee just posted his thoughts about the importance of Linked Data on the Semantic Web. Linked data support is built-into Twine. All the data in Twine is accessible as open-standard RDF and OWL today and will be accessible to other applications via several API's including SPARQL. You can learn more about Twine's support for Linked Data and see some examples here.

Tim says:

In all this Semantic Web news, though, the proof of the pudding is in the eating. The benefit of the Semantic Web is that data may be re-used in ways unexpected by the original publisher. That is the value added. So when a Semantic Web start-up either feeds data to others who reuse it in interesting ways, or itself uses data produced by others, then we start to see the value of each bit increased through the network effect.

 

So if you are a VC funder or a journalist and some project is being sold to you as a Semantic Web project, ask how it gets extra re-use of data, by people who would not normally have access to it, or in ways for which it was not originally designed. Does it use standards? Is it available in RDF? Is there a SPARQL server?

Twine provides RDF and supports SPARQL (although while we are in beta we have not opened our SPARQL API yet, but we will...). At the same time Twine also protects privacy by only providing its data according to permissions. Apps can only get Twine data they permission to see such as their own data or their owner's or users's data, data that has been shared with them, or public data in Twine.

Twine is also designed to consume external Linked Data via it's APIs. Twine will be able to consume external RDF and OWL ontologies, as a means to enable other applications and users to extend its functionality and add new data to it.

My Visit to DERI -- World's Premier Semantic Web Research Institute

Earlier this month I had the opportunity to visit, and speak at, the Digital Enterprise Research Institute (DERI), located in Galway, Ireland. My hosts were Stefan Decker, the director of the lab, and John Breslin who is heading the SIOC project.

DERI has become the world's premier research institute for the Semantic Web. Everyone working in the field should know about them, and if you can, you should visit the lab to see what's happening there.

Part of the National University of Ireland, Galway. With over 100 researchers focused solely on the Semantic Web, and very significant financial backing, DERI has, to my knowledge, the highest concentration of Semantic Web expertise on the planet today. Needless to say, I was very impressed with what I saw there. Here is a brief synopsis of some of the projects that I was introduced to:

• Semantic Web Search Engine (SWSE) and YARS, a massively scalable triplestore.  These projects are concerned with crawling and indexing the information on the Semantic Web so that end-users can find it. They have done good work on consolidating data and also on building a highly scalable triplestore architecture.
• Sindice -- An API and search infrastructure for the Semantic Web. This project is focused on providing a rapid indexing API that apps can use to get their semantic content indexed, and that can also be used by apps to do semantic searches and retrieve semantic content from the rest of the Semantic Web. Sindice provides Web-scale semantic search capabilities to any semantic application or service.
• SIOC -- Semantically Interlinked Online Communities. This is an ontology for linking and sharing data across online communities in an open manner, that is getting a lot of traction. SIOC is on its way to becoming a standard and may play a big role in enabling portability and interoperability of social Web data.
• JeromeDL is developing technology for semantically enabled digital libraries. I was impressed with the powerful faceted navigation and search capabilities they demonstrated.
  
• notitio.us. is a project for personal knowledge management of bookmarks and unstructured data.
• SCOT, OpenTagging and Int.ere.st.  These projects are focused on making tags more interoperable, and for generating social networks and communities from tags. They provide a richer tag ontology and framework for representing, connecting and sharing tags across applications.
• Semantic Web Services.  One of the big opportunities for the Semantic Web that is often overlooked by the media is Web services. Semantics can be used to describe Web services so they can find one another and connect, and even to compose and orchestrate transactions and other solutions across networks of Web services, using rules and reasoning capabilities. Think of this as dynamic semantic middleware, with reasoning built-in.
• eLite. I was introduced to the eLite project, a large e-learning initiative that is applying the Semantic Web.
• Nepomuk.  Nepomuk is a large effort supported by many big industry players. They are making a social semantic desktop and a set of developer tools and libraries for semantic applications that are being shipped in the Linux KDE distribution. This is a big step for the Semantic Web!
• Semantic Reality. Last but not least, and perhaps one of the most eye-opening demos I saw at DERI, is the Semantic Reality project. They are using semantics to integrate sensors with the real world. They are creating an infrastructure that can scale to handle trillions of sensors eventually. Among other things I saw, you can ask things like "where are my keys?" and the system will search a network of sensors and show you a live image of your keys on the desk where you left them, and even give you a map showing the exact location. The service can also email you or phone you when things happen in the real world that you care about -- for example, if someone opens the door to your office, or a file cabinet, or your car, etc. Very groundbreaking research that could seed an entire new industry.

In summary, my visit to DERI was really eye-opening and impressive. I recommend that major organizations that want to really see the potential of the Semantic Web, and get involved on a research and development level, should consider a relationship with DERI -- they are clearly the leader in the space.

Disturbing Eyewitness Testimony from Lhasa

I got this picture from a friend who received it from a friend who is on the ground in Lhasa right now. Please redistribute, especially to the news media.

Things are much worse for the Tibetans than has been reported. Can we help to avoid another Tainanmen?

First Week of Twine Beta Phase II Report

This week we began letting the second wave of beta users into the Twine invite-only beta. It's been a very busy and exciting time for the Twine team. I'll be providing more detailed stats on an ongoing basis in a few weeks once we have more data to analyze. For now, I will just provide some qualitative observations.

Twine is still in the early beta process, but already we are seeing a rapid increase in adoption and scale. We have only let in a few hundred more users to get the process started, but we will be letting more and more in every week as we go forward.

It has been really exciting to watch Twine grow. I find that I am increasingly glued to my Interest Feed watching the fascinating information that is flowing through from all the new members. There have been many new twines created around a wide and growing range of interests and large amount of content added. The recommendations are also quite interesting -- I have already discovered a wide range of new people, twines and content that I didn't know about.

As of this writing, I now have 157 social connections in Twine. My social network in Twine has doubled in size in a week and is rapidly approaching the size of my Facebook network. That's pretty impressive considering this happened in a week (it took about half a year for my Facebook network to grow to that size).

We also had our first outside Twine client app, called "Entwine," written spontaneously by a beta user -- it browses through the RDF data from various items in Twine. That was very cool and unexpected! It really got the team jazzed to see this happen.

Twine is now full of active discussions around interests, questions, ideas, suggestions, current events, technologies and products. I have been pleasantly surprised to see so much interaction among users develop so quickly. As we had hypothesized, discussions are turning out to be a very key feature.

We have received a lot of great feedback from beta users within Twine, as well as many suggestions for how to improve Twine, streamline the user experience, and integrate Twine with other applications and services. This is exactly what we had hoped for from our beta. The team is hard at work analyzing this and prioritizing our next development sprints in light of what we are learning from our users (we do minor releases every week and major ones every 3 weeks).

Most of the press reviews and user stories point to Twine being very exciting, useful and full of potential, which has been great to hear after so much work --- they also universally agree that we still have room to improve the user experience and we need to work on making Twine easier to learn and use. That's not unexpected -- we opened the beta well before the app is finished in order to understand user priorities better. We are really focusing on usability and bug fixes for the next several sprints.  All this feedback has been incredibly valuable to the team. Keep it coming!

Another interesting observation. The quality of the users in Twine is distinctly impressive. It's a very smart community of leading-edge thinkers, builders, and technology adopters. Kind of like having your own TED Conference, 24/7 around the world. We will be inviting in a wider range of users in later phases, once the app is further along. In the meantime it is really great to see so many of my colleagues in Twine, and to be making so many new contacts and friends here. For this initial phase this is exactly the audience we need -- people who will really roll up their sleeves and help us make Twine into a great application.

Twine is also rapidly aggregating most of the leading minds in the worldwide Semantic Web development and research community into a social and collaborative interest network. It is great to have this global community of people interested in building and using the Semantic Web come together in Twine, an application that is built using Semantic Web technologies on the Radar Networks Semantic Web Applications platform. I look forward to beginning to share Twine with this worldwide community, and to collaborate with others to extend it and integrate it with other semantic apps and data sets. This is definitely our goal.

It's been a great week. I haven't slept much. I'm having too much fun in Twine!

Twine Perspective on Yahoo Semantic Web Search Announcement

The Beginning of the Mainstream Semantic Web?

It is being reported that Yahoo will be indexing a wide array of structured metadata, including Semantic Web metadata. This will make Yahoo's search index potentially better than Google's, although it will also open their index up to sophisticated attempts to "game the system" as well that will need to be solved. But in any event, this will undoubtedly prod Google to begin indexing and making sense of structured metadata as well (actually, Google is already indexing FOAF, a Semantic Web metadata format).

I believe Yahoo's announcement marks the beginning of the mainstream Semantic Web. It should quickly catalyze an arms race by search engines, advertisers, and content providers to make the best use of semantic metadata on the Web. This will benefit the entire semantic sector and all players in it.

As they say, "a rising tide lifts all boats."

Where Twine Fits Into This Ecosystem

From the perspective of a company working on a large Semantic Web driven portal venture (Twine), and full platform for semantic applications (and search), this is good news. We'll be happy to open up Twine's content to Yahoo's index (when we go into General Availability in the summer timeframe, or maybe even sooner...). In addition, as more content providers add metadata to their content, it will make Twine's job of helping users collect, organize, share and discover interesting content, that much easier.

Where does Twine fit into the emerging Semantic Web ecosystem? Twine provides presence and content on the Semantic Web. It enables individuals and groups to homestead on the Semantic Web and get immediate value, without having to learn RDF.

Currently we are not going after the "be the search engine of the Semantic Web" opportunity -- we are focused on the "help users manage their information and connect with others who share their interests" and the "build thriving communities of interest" opportunities.

Our feeling is that incumbent search engines are probably best positioned to win the search engine of the entire Semantic Web war, when they decide to (as Yahoo just did, and Google most likely will soon decide to do as well...).

Twine is generating high-quality Semantic Web metadata about people, groups, topics of interest, and resources on the Web (Web pages, images, videos, books, products, documents, etc.). The metadata we are creating results from a combination of automated processing and user-contributions from our community.

The metadata Twine generates is then provided back to the users and community as open RDF that can be accessed and reused elsewhere. So we are effectively making a semantic graph of RDF about content around the Web, and related people, groups and their interests. Ultimately we become a semantic annotation layer above the Web. I can imagine that this is a dataset that Yahoo and Google and many others are going to want to be able to search.

The content in Twine is rapidly growing into a large semantic graph of information around people, groups and interests on the Web. We and our users are producing a large volume of high-quality original content and semantic metadata about existing Web content, that will undoubtedly make the Yahoo index much richer (and will drive traffic back to Twine and the sites we link back to from our graph).

The Semantic Web Eliminates Traditional Silos By Opening Up and Linking the Data

Twine is a hosted online service, but is not actually a "silo" in the traditional sense because all of our data is represented in open-standards-based RDF, and we are already providing access to that data on an experimental basis, and will provide even more via upcoming API's in the future.

This means that the data Twine is creating and gathering, is open, linked data, that can be reused in other applications and services. Ultimately this makes Twine a part of a growing distributed ecosystem. Semantic Web metadata in RDF and OWL is even better than microformats because it carries its own meaning about how to use it. Software that speaks RDF and OWL can instantly reuse it without any additional programming. To learn more about Twine's open RDF availability, see the Twine Tour: Semantic Web section.

I believe that the open-standards of the Semantic Web eliminate silos. Effectively all services that participate in using these standards and make their data open are becoming part of one big distributed worldwide database, rather than old fashioned silos. That's the benefit of open linked data services powered by RDF, OWL, SPARQL, and GRDDL.

How Will End-Users Participate in the Semantic Web?

If Yahoo and possibly Google make search better by indexing all sorts of metadata, there is then an even larger opportunity to help non-technical end-users create and use that metadata. This is where services like Twine fits in. End-users need ways to author, organize, share, reuse, and discover Semantic Web content.

We don't believe ordinary Webmasters or end-users are going to write microformats or RDF by hand. Even hard-core Semantic Web researchers don't do that. Ultimately end-users need user-friendly services that do this for them automatically, or at least make it easier to do. Twine helps these users to participate in the Semantic Web, without requiring them to have a degree in computer science. Twine provides an (increasingly) user-friendly hosted place where users can collect, organize, share and discover other interesting content around their interests, using the Semantic Web transparently "under the hood."

Concluding Thoughts

In short, Twine is where ordinary non-technical individuals and groups can join the Semantic Web, get a presence there, and start using it in useful ways, today. If Yahoo and Google become the search engines of the Semantic Web, that will make Twine even more necessary as the place where end-users can participate in this emerging ecosystem. We believe our community, and the rich the semantic graph we are growing will become increasingly valuable as the major search engines begin to index the Semantic Web.

But this is just the beginning of our story. Twine is designed to become a platform that others can build on and integrate with as well. There is more to our strategy than we have currently opened up about. In time we will be telling the rest of our story. We have some fun surprises in store in the future...

Reminder: Twine is a Beta -- A note about what Beta means

I want to remind everyone, TWINE IS A BETA. It is only a beta. Beta means not finished, under development, work in progress, construction site, imperfect, open to feedback, undergoing testing, getting better everyday, in need of more work, etc. and many other things that are not synonymous with "finished" or "ready for consumer launch." We know this. We never claimed otherwise. We opened Twine early to get feedback and let the community play around and give us feedback to guide our future work.

Some of the recent coverage of our project has seemingly misunderstood the meaning of the term "beta" or forgotten it, or simply expected a beta to be more of a finished application. Perhaps this is because many companies never come out of beta or use beta to mean "1.0, only cooler." In our case, beta really means Beta. We knew there were bugs and unfinished features, but we decided to open up anyway in order to get user feedback to guide our further work.

But even though Twine is a beta, it is already quite useful, and there is a large and thriving community in there sharing knowledge about interests including the Semantic Web, Web 3.0, Web 2.0, venture capital, politics, art, fashion, travel, cultures, religion, books, and many other interests.

In fact, the number of connections I have in Twine is rapidly approaching, and will probably soon surpass, the number of connections I have in Facebook. And in terms of use, we are finding that our users are visiting Twine many times a day and actively adding information, searching, and participating in discussions and debates there.

The hype around the Semantic Web (and even Twine) is in my opinion justified, but it will take time for that opinion to be obvious to everyone. In the meantime, I do think it has gotten a bit out of control. There is too much wild speculation and a general feeling that somehow the Semantic Web (or services like Twine) will solve every problem on the Internet. That won't be the case. However the Semantic Web and services like Twine that are built with it will  improve the content of the Web and enable applications to become smarter with less work.

To some degree the hype around the Semantic Web has set unrealistic expectations and it's not surprising that there is now some backlash. Some folks who came into Twine may have had impossible expectations -- perhaps thinking Twine would be some kind of a three-dimensional interface to all information, or a kind of Hal 9000 intelligent assistant. I'm sorry to disappoint them. Twine is much more pragmatic and focused on things like organizing, sharing and discovering information around interests. It is also just a first step in a long development path in which much more will be added in the future. And let's not forget... Twine is in Beta. It's not finished yet.

I think the backlash is good actually -- it will reset expectations to realistic levels. Hopefully then folks can focus on what the Semantic Web (and Twine) do today, rather than what they imagine they might do in 20 years, or what they don't do yet.

In the case of Twine, it is not a panacea, but it is certainly well on its way to becoming a leading semantically-driven online service with some interesting opportunities in the marketplace. There is certainly a lot more in the application than can be discovered in 7 minutes of using it and I can understand how that might be frustrating to reviewers who have little time and high expectations of a finished consumer app. That is something we are working on and when we eventually move out of beta, it is something we will be able to say we have solved it.

Meanwhile, Twine is a beta and while there is already a LOT there, we can, must, and will be doing much, much more to address usability and finish features that are still under development and imperfect.

Response to Read Write Web article about Twine

UPDATE: I posted some further notes on the fact that Twine is in beta, and what "beta" actually means and why we are in beta here. 

Marshall Kirkpatrick wrote a critical review of Twine today that identified several known issues the team is working on. These are points well-taken -- we certainly understand that Twine is still a work in progress and there are many areas where we can improve usability. After all, Twine is still in private invite-only beta and is not a finished application yet. There is much that is still under development and we are learning from our users everyday.

However, we have also been getting quite a lot of very positive feedback from our beta testers as well. Twine is already quite useful and works surprisingly well on a wide selection of Web content today, as our growing beta user base can attest to.

So on balance, while Marshall points out several issues we are aware of and are working on, there is much we are proud of in what we have been able to accomplish so far.

But I want to address some of the specific points Marshall made. Marshall pointed out the following issues:

• Sometimes Twine is unable to auto-summarize the content of some pages it sees, when they are added to the service. 

That's true -- it's sometimes hard for Twine to identify the "content part" of a page when the page has complex structure (including tables, Flash, Ajax, frames, multiple DIV areas, etc.). In the meantime, Twine does actually do a good job on things like Typepad Blogs, the Wikipedia, Youtube, Flickr, Amazon books, Wordpress, and most sites that have relatively standard page structure and/or metadata. That said however, we are working on making Twine smarter so that it can do a better job, even when there is uncertainty about the content and structure of a page. As Marshall points out this is a hard problem because there is so much non-standard content on the Web, but it's not an insurmountable one. Twine will steadily improve over time on this front.

• Twine doesn't recognize article authors as related people.

Actually, if Twine can see the author's name, it will recognize them as a related person. But the author's name is not always visible on the article. It would be easier to manage this if there was better metadata on pages, but until that happens, the natural language approach is the main option, and it is not always perfect.

• Marshall mentions that he thinks Twine could be better organized.

Marshall mentions that he had a hard time getting oriented and finding his way through the application because there is so much there. One of the challenges we have is simply educating users about how to Twine and what it is capable of. In addition there are many improvements we know we can make to the user-interface and information design to make it easier to figure out.

Marshall also asked for RSS feeds and visualizations.

RSS output is already supported to a limited extent and we will have more support for it next month. We are also planning to add RSS input as well in coming months.

Regarding visualizations, we've done a lot of work on visualizations in the past. Our feeling is that they usually don't add much value, other than being eye-candy. However, we will be opening up our API's eventually to allow others to make all the visualizations they want. If someone makes a really useful one, perhaps we'll include it back into Twine.

Finally, I would also like to correct one thing that Marshall mentioned: We are not in fact going into general release next month -- we are just starting to let more people in from our waiting list to continue to help beta test Twine. There will still be a members-only policy in effect for several more months. The full public opening (when Twine will be opened to non-member guests, and search engines, etc.) will be in the summer timeframe. Even then, Twine will still be in beta. There is a good year of additional work to do on Twine before it will be fully "baked," to use Marshall's term. Between now and that time we will be working to improve (and finish) the app, in partnership with our beta community.

In closing, as I have said many times, Twine is still an early Beta and we have to keep expectations in line with reality. Twine is already far and beyond what any other semantic app I know of is capable of, but that still isn't good enough. We have to push further and focus more on usability. We are opening it up early in order to get feedback and more help testing and guiding the direction of the app from users.

Hopefully as we work on Twine further, and we move out of Beta, Twine will eventually meet Marshall's high expectations. Meanwhile, his comments are helpful in that they do give us feedback about what aspects of Twine we need to focus on more as we head towards a more consumer-friendly application.

Do You Want Early Access to the Twine Beta?

Special offer to readers of my blog...

There are now well over 30,000 users in the queue to get into the Twine beta. We're going to start letting people in from the waiting list in waves and it should take about a month or two to let everyone in.

But what good is a waiting list if there's no way to cut to the front, right? Fortunately, there is a way to skip ahead to the front of the line...

Write a blog post about Twine on your blog and why you want early access, and send me the link to nova (at) radarnetworks (dot) com. along with your first name, last name, and email address. If I like your post, I'll get you an early access VIP pass to front of the line.

See you in Twine!

Insightful Article About Twine

Carla Thompson, an analyst for Guidewire Group, has written what I think is a very insightful article about her experience participating in the early-access wave of the Twine beta.

We are now starting to let the press in and next week we will begin to let waves of people in from our over 30,000 user wait list. We will be letting people into the beta in waves every week going forward.

As Carla notes, Twine is a work in progress and we are mainly focused on learning from our users now. We have lots more to do, but we're very excited about the direction Twine is headed in, and it's really great to see Twine getting so much active use.

How about Web 3G?

I'm here at the BlogTalk conference in Cork, Ireland with a range of bloggers and technologists discussing the emerging social Web. Including myself, Ian Davis and Paul Miller from Talis, there are also a bunch of other Semantic Web folks including Dan Brickley, and a group from DERI Galway.

Over dinner a few of us were discussing the terms "Semantic Web" versus "Web 3.0" and we all felt a better term was needed. After some thinking, Ian Davis suggested "Web 3G." I like this term better than Web 3.0 because it loses the "version number" aspect that so many objected to. It has a familiar ring to it as well, reminding me of the 3G wireless phone initiative. It also suggests Tim Berners-Lee's "Giant Global Graph" or GGG -- a synonym for the Semantic Web. Ian stayed up late and put together a nice blog post about the term, echoing many of my own sentiments about how this term should apply to a decade (the third decade of the Web), rather than to a particular technology.