“Panace is an Augmented Reality application that uses real time face/image recognition technology, and provides for Contacts Management and HyperLocal communities inside an Enterprise.”

You can find the presentation about Panacea below, or find it on Slideshare here.

The core idea behind the product is that, inside a closed community, it should be really easy to exchange contact and schedule information. Just like how inside an enterprise there is a Global Address Book and also shared calendaring. Panacea extends that by associating a facial identity with each person in the closed community. I mention closed community a number of times, for even I cannot fathom the privacy implications such an application will have in the public domain. We used Face.com‘s face recognition technology to aid in Face Recognition and used OpenCV’s face detection algorithm by accessing their compiled libraries via JNI calls on the android platform.

Scenario 1: You meet a colleague in your organization or at your local community meet. You need exchange contact information. Now, instead of exchanging visiting cards, you could just switch on our app, point the camera at the person, wait till the algorithm recognizes the person, and then add the person’s contact information via a single click. Better explained by a video demo of the solution.

Like any pattern matching algorithm, face.com’s algorithm is also a trained algorithm. Better the sample set or the reference set, better the matching. To make the algorithm work consistently, we need to train the service by sending at least 7 to 8 different photos of the person and associate a unique identifier for that set. When a candidate input is presented, the image clicked via the camera on the phone is sent to face.com and we get back that unique identifier with a matched probability score. Now, this unique identifier could be anything from your corporate id or your twitter handle. The important thing is that there is a backend system which you can query using that unique identifier and retrieve contact details associated with that id.

There is a problem though – I cannot ask every person in the community to upload 10 mugshots to our system to be included in the training set. I would need a more scalable solution. We solved this problem by providing a training mode in the app itself.

Scenario 2: If you realized that one of your colleagues or friends isn’t registered on the service, you can switch on the app and switch to training mode. Start the training by pointing the camera to the person and associate the unique identifier for the person. You can then fill in the contact details yourself or fetch it from the centralized contacts store. Follow the progress bar in the training mode and wait for the training to complete. Once done, any user of the app can now add your recently added contact using the mechanism described in scenario 1. Adding a new user to the application & service is crowdsourced. Again, better explained with a Video demo.

Now that we had sort of envisioned this technology and its working, fleshed out some initial prototypes, we brainstormed about how this technology can be used to solve other problems. I was also in the market to buy a smart phone myself. I did some research and found out that smart phones in India, with Assisted GPS support, started at around Rs 12,000. But high resolution camera phones were selling as low as 6. Also, I was fascinated by the whole location service apps like FourSquare and Gowalla. It turned out that only 13% of all smart phones sold in the world were location aware. This means there was a big market for location based services for non GPS enabled smart phones.  This is how we tried to crack this problem.

Scenario 3: You have a smart phone with an internet connection but no GPS. You want to use a location based service to get some deals on items, check into the place, or just give some feedback or tips for other travelers.  Our application defines what we call as a PlaceMark, a unique image of some sort which is linked to a location in our service. To perform any of these functions, open our app, point to the place mark and wait for the location features to load up. You will now get a feed of past conversations, tips, tricks shared by people at that place. The owner of the location can also target people by providing them deals for checking in and giving a social recommendation of sorts. You can even use these as mechanisms to receive feedback, complaints or engage in a Q&A at a large conference. The possibilities are endless. The example video shows a distributed discussion happening on the wikileaks issue. You can join this discussion by using our app and start conversing. The app even sends notifications to your phone when new messages come in.

This was mostly what we came up with. There are plenty more things that can be done with this application, and this, or a similar technology, can be used to solve many other problems.  We also want to take this product to the market and see what responses it receives. Feel free to leave comments.

There are claims that Open Source will one day completely displace closed software. Same goes with free online advertising, sometimes also referred to as ad exchanges, where a person places an ad on his/her site and in turn gets his/her ad placed on somebody else’s site. There are claims that free online advertising will one day displace Google’s monopolistic hold on online advertising. Frankly I think most of these claims are exaggerated and baseless to a certain extent.

Am I the only one who has a problem with transactions that do not obey the traditional laws of Economics? The market, or as Adam Smith would say – the foundations of human society,  runs on the buy-sell model. When something is free, it means other non quantifiable commodities get traded, like goodwill and trust.  In such circumstances fairness is the biggest casualty. But is there really a model which can ensure fairness in these situations ?

Yes, there are plenty of trust based networks at play in our real life, but they all reach a saturation point. A typical game theoretic scenario, the person to defect first gets a very high payoff ; In a network, everyone else suffers. People will cheat, one way or another. Like I mentioned before, fairness becomes a higly debatable issue. Though, the categorical imperative bestows humans with decision making powers, it doesn’t state the thresholding for these decisions. The only final outcome of these networks is that one person is going to hit it big and make the others suffer.

Naturally the follow up question will be:  how does Open source survive? I explain this using results from game theory. In any homogeneous population, ie either a defecting or a co-operating , a small group of people,  co-operating with each other while defecting against the general consensus , always has the ability to hold their own and, in some cases, even win against the entire population. Examples of this are plenty in real life,  for ex: corruption or the mafia. So, as long as that small bunch of people stay loyal to each other, such networks will continue. Open Source, is slightly a more complex beast than I described it to be, but it broadly falls into the category I mentioned.

Incentive mechanisms like reputation points, virtual currency etc usually try and emulate a real world materialistic economy, but they are limited in their reach and appeal. Eventually, people loose the initiative to compete for these non materialistic resources. Besides, in a medium such as the internet, where identity itself is questionable, how will people trust each other. Has anyone ever wondered if those mails, which inform you of the billions you have won, were ever real? I guess not.

In conclusion,  I ask, is there some mechanism design or a strategy that can ensure fairness (provided its defined accurately) whilst ensuring the continuing growth of such trust based networks ?

I was supposed to be involved in a discussion about cloud computing at Cloudcamp Bangalore, but due to other commitments, I could not attend the event. I had a small writeup about the limitations and challenges in Application clouds. Here is the full text of it.

Cloud Computing is a way of providing dynamically scalable and available resources such as computation, storage etc as a service to users who can use it to deploy their applications and data. Cloud Computing can handle data in both the public and the private domain. But this seemingly harmless way of thinking about building applications has its own set of issues.I am primarily referring to application cloud providers, the kind where you deploy your applications. Not storage and service clouds. Google AppEngine would be a good example for the cloud that I am describing. I note some of them here :

From the Users perspective:

1. New unstructured and non standard paradigm of programming: Each cloud has its own supported programming language and syntax requirements for programming, though most of these clouds expose the typical hashtable based cache and datastore interfaces. There is an urgent need for standardization of interfaces and methods of programming them. One of the reasons why shared hosting environments work great is because , as a programmer, I know that I can move my PHP/PERL code to another server and it will work without too much of a fuss. Moving from one of the dozen odd cloud providers to another requires considerable developmental efforts, not to forget time (for businesses, this could spell doom).  A look back at history shows languages like SQL, C etc being standardized to stop exactly this sort of undesirable proliferation.
2. Restrictions on the programming model : For cloud based applications to be highly available, they must be easy to dynamically mirror on multiple machines. Once these applications are mirrored, they can be served on demand by load balancing servers which makes them highly available and the user doesn’t face delays in being serviced. This is an old trick used by busy websites from the early days of web publishing but these solutions were custom built for websites. So, extending this concept to cloud based platforms, servicing thousands of applications, mandates the platform providers to automate this task of replication and mirroring. This job is easier said than done. This process can be made seamless when the program stores as little state information as possible. By state, I mean transactional variables, static variables, variables in the context of the entire application etc. These things are almost a given in traditional programming environments but are very hard to come by in cloud based environments. The unnatural way of dealing with this situation is using the datastore or the cache to store state of an application. There are a lot of restrictions like lack of privileges to install third party libraries, no access to file system to write files etc ( which forces you to use the datastore and pay for it)
3. A good local debugging experience: A good local development environment, debugging experience is a must for programming on the cloud. Most cloud providers do not provide good local development environments. There is also a lack of good IDE’s that can help with programming and debugging programs written for the cloud. The providers that do provide a local debug experience, do not simulate real cloud like conditions. Both from my personal experience and from conversations with other developers, I have come to realize that most people face problems when moving code from their local development servers to the actual cloud. This is only due to inconsistencies in the behavior of the local dev env compared to the cloud.
4. Appropriate metrics and documentation of programming best practices : On a cloud, since a user pays for almost every CPU cycle, appropriate metrics on usage of processing time and memory must be presented to the users. Typically a profile of the application with function names and their corresponding time taken, memory used, processing cycles used will definitely help the developer tune his/her code to optimize on usage of processing power. The best solution for this is for cloud providers to abstract common code patterns into optimal libraries so that the users can be assured that they are running the most optimal code for a certain operation. An example of this is Apache PIG, which gives a scripting like interface to Apache Hadoop’s HDFS for data analysis. Also, Most cloud providers do not provide enough statistics and also profiling capabilities.

From the providers perspective:

Here I look at challenges that cloud providers have to face:

1. Ensuring availability of the cloud: This proves to be crucial as Clouds host critical business applications, for whom, downtime would mean monetary losses. Effective monitoring and load balancing solutions are to be built. Most clouds employ virtualization technology to get the most out of any resource. In such cases, tools should be written to figure out a resource hog early and move the application to a more powerful grid or a machine, so that the other users get their share of the cloud without delays.
2. Ensuring Consistency: Both the data and code is replicated on the cloud and maintaining consistency of data is extremely crucial. This is the reason why most transactional updates are not allowed on the cloud. Example: sequence objects, which are almost a given in traditional databases are not provided, probably because maintaining state across machines for such statements is non trivial. Problems like distributed updates, locking, partitioning, sharding etc  arise when dealing with data. Such constructs are to be provided to the users as most of it is given in the non cloud deployment space.
   Most datastores provided by cloud vendors (except the ones that provide cloud based database services) do not support relational models. Which means all object relations have to be programmatically established. This could always lead to bad code, unnecessary joins, cascading problems and tons of other problems that developers faced before working with relational datastores.
3. Program verification : One of the biggest worries about deploying applications on the cloud is the correctness of the program in execution. Erroneous conditions, like infinite loops, can not only put the machine at the risk of being overloaded and unavailable, but also cost the user a significant amount of money. Tools like static analysis should be used to analyze code uploaded on the cloud and it should be checked for infinite loops, possible race conditions,  null references, unreachable code etc. The code uploaded should also be optimized or suggestions should be provided to the users about how they could optimize code to best utilize the available resources.

Conclusion : The cloud should become a complete nonrestrictive platform for applications. There should be no restrictions on the constructs, functionality and privileges on the cloud. Also, it should be dead simple to move everyday applications onto the cloud without too much of rework. This could mean writing migration utilities, import/export options and other artifacts that make the transition to a cloud much easier.  This will prove essential as most live applications, at least currently, do not run on a cloud and helping them migrate easily will mean more revenue and adoption.

I used friendfeed because I wanted to start a conversation based on the items I shared. But, again like all social media sites, my ability to get the conversation started depended directly on the number of people who were using friendfeed. So, I sat down to fix the problem myself and after one nights work, Moffe was born.

For the unitiated moffe stands for My Own FriendFeed Emulator. It gives the same features that friendfeed provides and also provides an easy way for people to leave comments on the items I share. Plus, the link that I shared, is federated directly into the page. The outcome is that now people can leave comments on items I share plus see the page all with one click. I have also incorporated canned comments for that restless user who doesn’t have time to write in comments. Plus, I get to monitor moffe on this cool dashboard.

In essence, its a microsharing service which lets me keep my content on my site and not rely on other services like friendfeed ,twitblogs etc. Thanks to Easy on the Slaw for giving me the twitter wrapper in PHP.

 If you are using a widescren display or a theatrical display with resolution sizes bigger than 1024 *768 then use More Y! . More Y! displays thirty results in three columns and looks legible on widescreen displays (no guarantees for normal displays though). Whats even more handy is that More Y! can be integrated into your favorite browser (Firefox 2+ or Internet Explorer 7+ ) by following the instruction provided on the More Y! page. Its a cinch.

 Have fun with More Yahoo! and do reply back with comments and feedback.

Tags: boss, search, More Y!, yahoo, itsmeritesh

Read this wonderful article and I realized, we are hooked to the internet!! Not a day goes by without us plugging in. The tools you work with clearly alter the way you do things and not necessarily in the right way. To give you an example, an electrician came to my place and couldn’t put a hole in a switch board because his drill bit was broken; an electrician from the pre-drill age would heat a sack needle to do the same. Did modern day tools spoil the electrician ? I would say, yes!

The same is the case with everyday programmers and web workers. I remember programming without search engines, an age where I peered through command lines and man pages to figure out why in gods name was the screen showing segmentation fault. Debugging was a very involved process, no modern day debuggers or code analyzers, just the program and console outputs. Though it sounds really painstaking ( and trust me, it was), it did make me a better programmer. I knew where not to screw up and also where things could fail. I see people start programming using IDE’s like Eclipse and Visual Studio and I really wonder whether they will ever have all the learnings that I had. Search engines and effective ones at that have paralyzed out ability to think, read and find information. Its affected us in adverse ways, so not comprehend able.

To become humans again, not impaled by search technology, I second the concept of a web sabbath. A day or two in a week, where we do not use modern day amenities like the mobile phone, Television , Internet or search. Such a day will help us to think, read, scour for knowledge the way we would have done ten years ago and hopefully that extra effort will make us think beyond technology and its limitations. I would like to hear from people who agree with me.

Tags: google, web, sabbath, nosearch, websearch, technology

Incorrect Semantic Interoperability describes mismatch in the format and representation of data belonging to two parallel applications which prohibit them from interacting with each other or prohibits possible migration. Take the example of ERPs like SAP and Oracle Apps, they essentially perform the same computations and solve the same problems, but do they interoperate? No. The data representations for both these applications are bespoke, which makes it unique to one product or a line of products. It is reasonable considering the fact that market domination is obtained by developing custom formats and providing custom decoders, but there is a bigger concern. For large players, data is collated and there is usually a Business Intelligence solution in place and with different products being deployed at different centres, the additional overhead of conversion of the data is imminent. Also two separate applications cannot talk to each other though they are probably linked up sequentially, you have to bring in additional middleware for format conversion and make a common bus.

Efforts like RSS, SOAP and others have been successful only to a certain extent that the format is correct but semantics are still loose. It has to come together sometime, if not now then later, when the data is huge. And the answer is yes, the Indian IT industry survives because of such semantic interoperability. Most of the workforce is maintaining these enterprise bridges (that holds together these different applications) if not building them. A majority of the work involved in the service sector has to do with writing compatibility plugins or writing migration scripts and patch ups. If it was for SI , we wouldn’t have jobs.

There are a set of people that like the innovation happening on the web front and are adopting 2.0 technologies like there is no tomorrow. Everything from office automation to project management is now managed online on productivity service providers.  Concepts like wiki, blogs, forums etc are fast appearing as mainstream applications in organizations. Surely as technology evolves and takes new shape, we will see a dramatic shift in adoption of these new tools .

In contrast , there are the other people who have been around for a long time and have seen a lot of productivity applications. To these people, technology is nothing more than a fast changing fad and prefer to stick to their old time favorites. Take people who have seen the main frame era, such folk just don’t appreciate concepts like distributed computing, virtual servers etc. Quotes like ” our mainframes never needed mirroring”, are common. People who still live reminiscing innovation of their times like spreadsheets and ERP’s.

It may be hard to believe but these form the majority of the so called power users of organizations and these legacy softwares( pun intended) , are maintained and supported just for their usage. Its distrubing to know that enterprise software lags open source software by at least 3 years , in terms of innovation. This lag can clearly be accounted to the legacy lovers who insist on using their accustomed softwares. Where does product development go in such a case. Office 2007 is seeing very slow adoption due to a change in the usability. Will this set of users be responsible for the sluggishness of product development? who will convince these users to adopt newer software? more importantly how? What will these users demand 20 years from now?

Its a strange question, but yes its an emerging market.

Google is going to put a cookie in your browser that will record information everytime you read an ad served by Google. The cookie helps Google predict what kinds of ads the person clicks on and will probably help serve better and more relevant ads. Up until now the ads are served based on the content of the site and also the publisher’s preference on the ads. The  cookie also doubles up as a proof of click ( think click fraud) and more material for publishers to analyse the hits to their ads and mine for patterns. Though the purpose is for ads, Im happy its a step ahead in personalization. The cookie acts like a distributed store of personal information that can be mined by ad servers and then acted upon.

The possibilities of this technology could be endless. Based on my previous post, where I wrote about a distributed implementation of a personalization database which all applications can use to customize data better, google’s cookie is one step towards such a solution. If every browser had cookies of personalized information then applications ( aka good applications) can mine these cookies and get useful information about the persons Net habits and preferences. But a cookie has size restriction and cannot probably be used to store the abundance of personalization information. You can use a add on activex plugin or a firefox addon like google gears with underlying SQLite database and do wonders with it. Of course this would mean about two years of thinking of standards and more bureaucratic gibberish, but once done, it will be a truly distributed system and the web, partially personalized.

You could be served the right category of blogs on wordpress, get the right videos on the opening page of you tube, see the relevant news on CNN , see the most relevant feeds on your RSS reader and much more. Personalization ahoy !!

Think of the same thing in software. A software that is programmed based on generic constraints and the software dynamically learns from its environment through input devices like sensors and then adapts to the changed environment. The classical shortest path problem can be taken as an example. If a crawler running through the shortest path is lifted from its path and put somewhere else, will it be ale to comprehend the change and then quickly adapt or will it be lost. What if the graph changes and produces a lot of cycles or what if the problem statement changes during the course of the program.

I know that a very few people are relating to what they are reading, but the belief that a machine can learn and adapt is what scientists are trying to prove everyday. Machine learning is taboo after movies like iRobot and Terminator, but trust me, we are far far away from something like that. If we can solve a subset of problems of adaptability through an expert system, that will be an achievement in itself.

But how do you go about designing software with such requirements. I would go one step ahead and call it no requirements or changing requirements with no defined thresholds. No current methodologies like  OO or aspects can cater to such a requirement. Probably a new scheme of designing learning software has to be developed. Taking tips from AI and the Turing thesis, a perfect turing machine is what is required. Its a hard call but we will get there one day.

There are innumerable number of apps that are being created on the facebook platform. Existing applications like wordpress are taking heed and porting their apps into the platform. Some of the initial apps have been taken over and VC’s are announcing seed fund for apps built on the facebook platform. Its like everyone wants a part of the 26 Million uniques visiting the site and want to cash in on the phenomenon. So what did facebook do right ?

As I had written earlier in my barcamp post, it makes more sense for applications to be launched as a part of a larger community rather than brave the web alone. Social networks have proved time and again to be good source for informing people or attracting traffic to your site, essentially great place for marketing. Facebooks plans of monetizing the network had to make the site open to public , which they did by opening up to public signups. The next step was to innovate inside the network. Knowing very well that cramming features into facebook will not result in anything but chaos, facebook needed a serious shake up in terms of the features offered to the users. Rather than putting in the effort themselves, they put efforts into making a platform that allowed for people to build apps on top of the facebook community. This ensured a plentitude of apps and also increased pageviews because of these apps. Its a killer strategy.

The popularity of the platform speaks for itself with tons of apps being built on facebook and the increase in activity on the network. Does this mean we will see more social sites opening up to inorganic growth ? There’s no doubt sites like myspace and digg bring in large number of uniques to their site, but will they provide a way for external sites to tap into the community, if it guarantees them more pageviews and more uniques. This has an implication of spinning off into a business of its own in which you sell a community to a service provider. I call it community as a service.  Or will they risk deterioration in quality and increase of spam due opening up of their services and continue to stay organic. Its pretty obvious that the wealth of resources the platform got facebook was plenty but it was facebooks inherent security and privacy features that made sure that there was minimal exploitation.

Do you think your app can provide Community as a service ?

The fact that we don’t understand what value others get from social web apps is part of the paradigm of social software. The key is that each person has their own social lives, their own social circle, and thus their own social values. What is important to their social life will almost certainly be unimportant to us because we have our own to worry about.

Think of it this way. Each person has their own social network. Chances are that social network overlaps very little with yours. If, say, that person wanted recommendations for watching a movie, they might turn to their social network, which is made up of their family, friends, and colleagues. They would ask these people, the people they know and trust, what movies they recommend.

Now, would you turn to the same social network for movie recommendations? Of course not. You trust the people you know…your social network, and so any of the chatter from their social network has no value for you. It’s meaningless chatter. Just like most people’s Tweets on Twitter.  In a larger sense, this opaque value problem affects most social software. Twitter, MySpace, Facebook, most social software is built around providing personalized, socially-focused conversation. It’s person-centered and as a result is difficult for anybody but that person to really appreciate: the value becomes opaque in this way. Designing social software is going to be very difficult. The designers won’t be able to put themselves into a position of someone who wants to keep up to date with their social network, which is something that all of these sites are doing.

What the excerpt from the original blog clearly states is that a person using a social software has his/her own reasons to do so. The fact that the others don’t know what value the other person sees in a social network is the entire basis behind social software. I may log into myspace to chat with my peer group, but I dont know what my friend X does in her network, so what do I do ? I snoop around their profiles and walls to get a glimpse of their world coz her value from the network is opaque to me .

The goal of social software is evolve from this paradigm to a more productive one. Efforts are on to make social software enter the productivity space and that too with good success. Based on all of these observations, how do you design for a social concept. How do we actually add value to the users ? Its not easy to please everyone, but at least if the majority is kept happy, you can be sure of winning product.

How about usability ? the way people use the internet has changed dramatically. Tons of data being uploaded everyday. Almost all of information of a person can be found on the internet. Thats why any criminal is always looked up in myspace or other social networks. Also how people interact with internet applications is also a factor to consider. Today we have IM, telephony, music players, media libraries, wikis, photo sharing services, blogs, and many other ways to interact with the internet. Who knows? probably in three years time you can transfer money from one account to another by just sending an IM through your favorite client.

Its not just we who perceive the internet. The way the internet perceives users has also changed. Its not just X searching on google, its X who lives here, likes this, hates that, has searched for Y before and is more probable to buy Z. The future could hold machines that can identify with you, relate to you and even deliver specially tailored content to you. The web has also become extremely customizable with every site offering you the concept of skins and templates.
What bout user behavior and usage? ten years back double clicking on the browser wasn’t even a concept. Autofilling textboxes and combo boxes were a fantasy. and Drag Drop, well it wasn’t available for some the best desktop apps, forget the web. Today people use them like they have been around for ages.
what about user evolution ? I have gone on from becoming a mail checker -> sleeze surfer -> information hunter -> programmer -> webmaster -> blogger and I have missed out probably tens of roles. How do we account for user evolution and experience.
Its pretty fuzzy for me. How do you define evolution on the internet ?

Text makes the web. Those characters that are 1s and 0s in their stripped down version form the basis of what we call information. Right from websites to search, feeds to news, wikis to chat they are all text. Its unquestionable that people consume a lot of text on an everyday basis and a simple look at your bandwidth bills will show you just how much. Why am I ranting on about these things ? Well, it so happens that the textual information that we consume actually help in realizing a very big dream, the dream of personalization.

Personalization, the word is one of the hardest to define and yet very simple to perceive. Its the need of the hour, no matter where you go, there are people craving personalization. As developers, our view of personalization is very limited. We tend to be more involved in the visual aspect of things like the color, the theme or at best the layout. These are akin to having balloons and glow bugs on your workplace , nothing more than visual appeal. What we must strive is the personalization of information and not its presentation.

Thats really hard : no it isnt. Take a look at my RSS reader and you will be able to judge what kind of information I am interested in. Why restrict it to only blogs. Take a look at my mails, my socionet profile, my friends data , my search results. All these have some details of the data that I consume everyday.

Why do we need that : Its a time saver, in terms of finding the information that I need. My search results can be more inclined towards the information that I am interested in. My social network can show me more like minded people rather than an arbitrary selection. Why even the ads can be targeted more towards things that I buy. I find the biggest ROI in terms of advertising. Let me explain with an example: Supposing i bought an Ipod , I download the itunes software, read abotu the instruction manual, next search for some popular tricks, themes, games etc. Now supposing I were to go to ebay and click a button “Show me things to buy” , based on my previous surfing trends its not hard to predict that I would want but accessories and other iPod related stuff. Now isn’t that what personalization should be about.

How do we do it: Its not practical to expect a Google or a Yahoo to do all the personalization, of course Google is taking serious strides towards personalization in its truest sense. The system has to be decentralized, it should be ip based, even if multiple users use a system, there is more likely a chance that their geographical location has a pivotal role to play in their surfing habits( take a hostel for example, more often that not you find people with similar necessities on the web) I think an OpenId like implementation of a personalization filter would definitely help a lot. You have distributed servers that keep relaying information about a certain persons information consumption habits and then when you ascertain certain patterns, you writeback to the server the pattern. Upon querying or any other action, you get a collated version of your pattern and then make decisions based on your content vs the required content. Aint it a cinch ?

Truly 2.0 has been about You but 3.0 will be about me….. everything me.

Penetration of user generated data into spaces like technology and entertainment have been exponential. Every person who wants information on code and solutions looks up forums and blogs but there are certain niches which are still firmly controlled by the creators like astro physics and rocket science ( pun) . Now when we have such a divide how can we make sure that universal search is actually delivering to its full potential. The primary criteria I guess is to put the search query in its proper classified subset. By this I mean, if I search for a topic that has more material in a blog than a wiki or a full blown site, then how will the algorithm know its relevance. Ranking clearly fails in these conditions and its pretty evident.

The second biggest problem is of course, the real estate. What I may be looking for may not turn up in the first search result and with most of my observation people don’t like shifting pages to more results, they decide to change the query instead. In that minimal screen how will you fit in the right contents based on the rank, relevance, semantics and many other jargons that even I,  haven’t heard of. One thing clear though is you need to mine.

Mine data from the users, their previous search results, probably from the one place where they have disclosed something about their life, their interests, hobbies etc – Social Networks.  Even the real estate management should change , there should be a new scheme of viewing results for a query, probably a tag based scheme that can subdivide the query into many groups and subsets of relevance and then refine it on top of the tags. It requires a change, in peoples usage, which is a dreaded de-motivator , but tags started the same way and its doing pretty alright in sites like flickr and del.icio.us.

The concept and implementation of web search is still unexplored and untapped. There are a wealth of things that can be done and I have mentioned a couple above.  Semantic web could solve some major problems, but its still a long way away and more time means more space for innovation.

claim 1:  An apparatus for determining an interestingness rank for at least one media object, comprising: logic for accepting at least one metadatum concerning the at least one media object from at least one user; and logic for ranking the at least one media object based at least in part on the quantity of user-entered metadata concerning the at least one media object.

The apparatus of claim 1, wherein the user-entered metadata comprises a member of the group consisting of: tags, comments and annotations.

Well there are new web metrics being rolled out almost everyday and its only time, at least some of them, become mainstream. One of the strong contenders I feel is the interestingness metric. Based on the number of bookmarks, clicks , click throughs, views, tags, annotations, visits, visit length and many many more factors interestingness can be defined, although this may not be Flickr’s definition. Any piece of information like a music piece , a podcast, a blogpost, a video clip,  a picture  etc can be associated with interestingness – It simply gives the most popular item amongst a list of inventory. These items can also possess geographical information, time line and other information that can be used to extract essential patterns from these items.

The usage, could be a phenomenon on its own. Define a properly ranked algorithm and you could have your top selling items on your e commerce site. You could target geographically distributed audience with content of their choice and also give interesting timeline applications that let you see trends over time. Say for example you run a shopping portal, you could give a time line graph highlighting your most sold items over a period of time. This could also prove a big step in the analytics area where ROI and conversion from visitors to buyers play a significant role in everyday business and concepts like these could really boost your sales. Its still unsolved from a very broad sense, meaning there isn’t a standard set of practices to encompass such a wide variety of data and there isnt such capable systems that could mine patterns out of these collations. But its definitely interesting

click on the image to enlarge……

 This mind map was created using MindMeister , a wonderful tool still in beta which lets you create mind maps and also collaborate in real time with your friends. If you are interested in making mind maps and are on the lookout for a tool, then this is the one for you. Its got a real neat interface and is very responsive unlike the other tools that i have used and its got a great export option. Thumbs up from me.

 In the Middle Ages, we were told what we knew by the Church; after the printing press and the Reformation, by state censors and the licensers of publishers; with the rise of liberalism in the 19th and 20th centuries, by publishers themselves, and later by broadcast media – in any case, by a small, elite group of professionals.

In the midst of all of these, at least there is realization that not all information is credible and who controls or contributes that information is as essential as the information itself. I wouldn’t what category to put that into, but we have an unsolved problem on our hands – Information relevance and credibility. Though the promise of a semantic web does tend to blur out the relevance or the contextual aspect of the problem ,  the credibility part of it is lost. Take this blog post for example, the quoted text is actually from Larry Sanger’s article which was referenced in Nicholas Carr’s article which in turn found its way here. Now supposing I were to look at that text alone without knowing the background, how would I know who was responsible for that piece of information.

Knowledge and Information Management techniques are not well equipped to handle such situations. Though images have watermarks and EXIF information, text has a very loosely coupled digital signature. Though these systems could probably solve the origination problem, the problem of information purity and fact/fiction is still left unabated. A conservative solution of allowing experts to monitor and control data ( which Larry Sangers next project citizendium plans to implement) just abstracts the problem to a higher level, besides , how many experts can you find in specialized areas who would voluntarily monitor the data available in their domain. More work and research needs to be done to ascertain accuracy of data and more importantly facts.  I cannot relate to machines solving the problem though a combinations of semantically linked documents with a lot of mining could give partial results. What do you think ? Have you come across any solutions to this sort of a problem ?