I was supposed to be involved in a discussion about cloud computing at Cloudcamp Bangalore, but due to other commitments, I could not attend the event. I had a small writeup about the limitations and challenges in Application clouds. Here is the full text of it.

Cloud Computing is a way of providing dynamically scalable and available resources such as computation, storage etc as a service to users who can use it to deploy their applications and data. Cloud Computing can handle data in both the public and the private domain. But this seemingly harmless way of thinking about building applications has its own set of issues.I am primarily referring to application cloud providers, the kind where you deploy your applications. Not storage and service clouds. Google AppEngine would be a good example for the cloud that I am describing. I note some of them here :

From the Users perspective:

1. New unstructured and non standard paradigm of programming: Each cloud has its own supported programming language and syntax requirements for programming, though most of these clouds expose the typical hashtable based cache and datastore interfaces. There is an urgent need for standardization of interfaces and methods of programming them. One of the reasons why shared hosting environments work great is because , as a programmer, I know that I can move my PHP/PERL code to another server and it will work without too much of a fuss. Moving from one of the dozen odd cloud providers to another requires considerable developmental efforts, not to forget time (for businesses, this could spell doom).  A look back at history shows languages like SQL, C etc being standardized to stop exactly this sort of undesirable proliferation.
2. Restrictions on the programming model : For cloud based applications to be highly available, they must be easy to dynamically mirror on multiple machines. Once these applications are mirrored, they can be served on demand by load balancing servers which makes them highly available and the user doesn’t face delays in being serviced. This is an old trick used by busy websites from the early days of web publishing but these solutions were custom built for websites. So, extending this concept to cloud based platforms, servicing thousands of applications, mandates the platform providers to automate this task of replication and mirroring. This job is easier said than done. This process can be made seamless when the program stores as little state information as possible. By state, I mean transactional variables, static variables, variables in the context of the entire application etc. These things are almost a given in traditional programming environments but are very hard to come by in cloud based environments. The unnatural way of dealing with this situation is using the datastore or the cache to store state of an application. There are a lot of restrictions like lack of privileges to install third party libraries, no access to file system to write files etc ( which forces you to use the datastore and pay for it)
3. A good local debugging experience: A good local development environment, debugging experience is a must for programming on the cloud. Most cloud providers do not provide good local development environments. There is also a lack of good IDE’s that can help with programming and debugging programs written for the cloud. The providers that do provide a local debug experience, do not simulate real cloud like conditions. Both from my personal experience and from conversations with other developers, I have come to realize that most people face problems when moving code from their local development servers to the actual cloud. This is only due to inconsistencies in the behavior of the local dev env compared to the cloud.
4. Appropriate metrics and documentation of programming best practices : On a cloud, since a user pays for almost every CPU cycle, appropriate metrics on usage of processing time and memory must be presented to the users. Typically a profile of the application with function names and their corresponding time taken, memory used, processing cycles used will definitely help the developer tune his/her code to optimize on usage of processing power. The best solution for this is for cloud providers to abstract common code patterns into optimal libraries so that the users can be assured that they are running the most optimal code for a certain operation. An example of this is Apache PIG, which gives a scripting like interface to Apache Hadoop’s HDFS for data analysis. Also, Most cloud providers do not provide enough statistics and also profiling capabilities.

From the providers perspective:

Here I look at challenges that cloud providers have to face:

1. Ensuring availability of the cloud: This proves to be crucial as Clouds host critical business applications, for whom, downtime would mean monetary losses. Effective monitoring and load balancing solutions are to be built. Most clouds employ virtualization technology to get the most out of any resource. In such cases, tools should be written to figure out a resource hog early and move the application to a more powerful grid or a machine, so that the other users get their share of the cloud without delays.
2. Ensuring Consistency: Both the data and code is replicated on the cloud and maintaining consistency of data is extremely crucial. This is the reason why most transactional updates are not allowed on the cloud. Example: sequence objects, which are almost a given in traditional databases are not provided, probably because maintaining state across machines for such statements is non trivial. Problems like distributed updates, locking, partitioning, sharding etc  arise when dealing with data. Such constructs are to be provided to the users as most of it is given in the non cloud deployment space.
   Most datastores provided by cloud vendors (except the ones that provide cloud based database services) do not support relational models. Which means all object relations have to be programmatically established. This could always lead to bad code, unnecessary joins, cascading problems and tons of other problems that developers faced before working with relational datastores.
3. Program verification : One of the biggest worries about deploying applications on the cloud is the correctness of the program in execution. Erroneous conditions, like infinite loops, can not only put the machine at the risk of being overloaded and unavailable, but also cost the user a significant amount of money. Tools like static analysis should be used to analyze code uploaded on the cloud and it should be checked for infinite loops, possible race conditions,  null references, unreachable code etc. The code uploaded should also be optimized or suggestions should be provided to the users about how they could optimize code to best utilize the available resources.

Conclusion : The cloud should become a complete nonrestrictive platform for applications. There should be no restrictions on the constructs, functionality and privileges on the cloud. Also, it should be dead simple to move everyday applications onto the cloud without too much of rework. This could mean writing migration utilities, import/export options and other artifacts that make the transition to a cloud much easier.  This will prove essential as most live applications, at least currently, do not run on a cloud and helping them migrate easily will mean more revenue and adoption.

 The answer is of course non trivial and I will only make a fool of myself by trying to predict the answer, but there are certain inferences I have made based on my observation which I shall pen down. I divide the broad category of pc users into prospects and adless users. Prospects are users who are new or ignorant to concept of online advertising, people like my mother who doesn’t know people endorse links for money. This category could also include people who are open to targeted advertising and see a value add with it. These are people who wholeheartedly click on interesting links. The other category I call adless users, users who have been around the internet long enough to understand irrelevant ads and can spot and ignore ads in a page. Take the example of the ad below.

The adless users wont even realize they scrolled though the content and didnt even notice the ad, or read it. 

More time a user spends online, the probability of him/her realizing the web is filled with irrelevant ads and over time becoming adless users. As a result, almost all users tend to move towards becoming adless users. This is dangerous for marketers, ad companies, publishers etc as there is a whole eco system depending solely depending on money made out of ads. As new users discover the web, their prospect phase is what publishers can hope to cash in on, but eventually the shift will happen. What happens then ?

Search engines are arguably the best places for advertising and probably the best place for demonstrating the phenomenon I call intrusive or endorsed content. Take the example below. 

 Ads will stop being sidekicks and move into the foreground, I have shown the shift pictorically.  Payperpost got the next concept right, people wont read ads, but social media yes, so pay people to write about your product/service etc. More results on search engines will be  endoresed and most of them already are, how do you know a review you are reading of some product isnt already endorsed. Now here is the strangeloop bit, you could say you will search for bad reviews instead of good like this. It wont take long for the advertisers to see this trend as well and then pay for people to write moderately bad reviews inturn endorsing the product. You know that they know that you are looking for bad reviews !! 

 A surprising result on top caught my eye. A visit to the site will tell you immediately that the site isn’t half as good as the second or the third result, but still its on top. SEO has come a long way and to cheat search engines into making a page popular isn’t that hard. You can hire professionals to do that job. That in a wierd sense is a form of endorsing, a professional SEO group can start bidding for making pages more popular and start their own cartel for endorsed content. 

 The other strange phenomenon I see that people recognize big search brands, Google in particular, but don’t necessarily relate to the results( you can’t possible relate to the results). You could have the Google homepage serving ads from ask.com and nobody would know the difference if the results looked like Google returned them. Thats probably the reason there still are companies trying to capitalize on the search market. Take a look at the results page below.

In this case the difference between a result from the index and an endorsement is a mere patch of color. How difficult do you think it is to remove that demarcation during difficult times.  Ethical boundries as meagre as color differences can be crossed very easily and corporations have showed time and again it can be done. 

Thanks to the falling prices of bandwidth and also social media, video is the next big delivery mechanism and it was quite understandable that Google paid a billion and a half to capitalize on youtube’s huge market share and put intrusive ads on videos( you dont have a choice there, no adblock plus !! ). Same goes with pictures and audio. Radio, papers and the television have been doing it for years. 

The world thought that we moved away from pop up advertising but we have just made the situation far worse. Ads will become more and more intrusive and there could come a time when content and advertisement are indistinguishable. More on this later.

 - Scaffolding : the rails feature makes it dead simple and mandatory to make the MVC architecture. Just run the rake and rails command to create all the MVC functionality. Add some jazz to the front end and you can make a standard database application in under 10 minutes.

- .NET web services : during my struggle with web services in Java, I never realized that most of the programming I was doing with Axis could actually be automated and thats what Visual studio did. define a web service and say generate functions for it and you wont believe the functionality that it implements. VS then goes to wrap the service with a class and all its calls and exceptions. Imagine my surprise when I saw a 1000 line program generated by just a click of a button.

- Chaining : imagine doing  banner.generate().show().navigate().hide().unload()  – ruby, jQuery and to a little extent python does it.

- reflections :  going through an entire package structure (jars) or through assemblies (dlls) to figure out what class you want to instantiate and then creating an object of the same during run time. Tons of design patterns are becoming obsolete thanks to reflections.

- AOP : using aop is a fad from a long time but a current technique made me think twice about the power of aop. Imagine a tool that uses deep reflections and figures out where you access the database and then automatically adds aspects that are relevant to the module. The programmer doesn’t even need to know what aspects are essential to the module he/she is developing. The tool automatically adds the functionality. Ex : you are writing into the database – tool figures database entries should be 1024 bit encrypted and adds the required encryption functionality. Now isnt that cool ?

- Program the cloud : write programs without knowing the hardware behind it ; just know that your program will scale like no other using the underlying cluster of hardware that makes the cloud.

 These are the ones that I can remember while writing this post. Do add more programming innovations that you come across in the comments.

Tags: code, programming, cloud, reflections, scaffolding

Data has far outgrown the domains of just text. Today we talk of mutlimedia data, urls, semantic data and many more application specific formats. Information on the Web is in JSON, REST , XML , Microformats etc. With this vareity in data formats and representations comes the inherent need for flexibility in storage and querying of such information. Almost all database users know of the conceptual modelling required for the design of any database, the key principle being that more tighter the model, more efficient the database. The integrity of the database is only as good as the integrity of the data. But you cannot talk of data integrity with the kind of formats available today.

Clearly markup data dominates the web . Though databases have developed features to better support , store and validate markup data , the initial design of databases was never to store the wide variety of loosely organized data. Querying of such markup data is fruitless and so is the attempt to index, sort , aggregate this data. To develop a custom database capable of all the above mentioned operations could be a solution, but the given the non standardized nature of this data and its probability of change, you would have a tough time scouring the web to search for changes. Plus these databases will not be semantically inter operable.

Developers are taking notice of a new scheme of storing data, I call it the bucket store. The design is roughly the same as that of a hash table, where data blocks are stored in buckets and hashes are used to index or refer to these buckets. A little improvisation in terms of adding upper layers like domains, groups and so on to complement the schema, table in a database is done to make the data easily classifiable. The advantage with this scheme is heterogeneity in data formats and the absence of constraints.

Several products are offering such services at dirt cheap prices. Take Amazon’s S3 or the recently launched Simpledb or CouchDb which offers a host it yourself version of this storage. Amazon S3 has businesses running on top of it; of the many I can recall Slideshare running on S3. With the advent of more mashups and heterogeneous data being churned out by the web more of such non DBMS related storage options will be employed. Given that this paradigm does implement all the enterprise important features like security, access control , backups, transactions etc and mature modeling methodologies that can rival the ER are proposed , I don’t see any problem in this becoming the most viable and cost effective option for data storage.

There are a set of people that like the innovation happening on the web front and are adopting 2.0 technologies like there is no tomorrow. Everything from office automation to project management is now managed online on productivity service providers.  Concepts like wiki, blogs, forums etc are fast appearing as mainstream applications in organizations. Surely as technology evolves and takes new shape, we will see a dramatic shift in adoption of these new tools .

In contrast , there are the other people who have been around for a long time and have seen a lot of productivity applications. To these people, technology is nothing more than a fast changing fad and prefer to stick to their old time favorites. Take people who have seen the main frame era, such folk just don’t appreciate concepts like distributed computing, virtual servers etc. Quotes like ” our mainframes never needed mirroring”, are common. People who still live reminiscing innovation of their times like spreadsheets and ERP’s.

It may be hard to believe but these form the majority of the so called power users of organizations and these legacy softwares( pun intended) , are maintained and supported just for their usage. Its distrubing to know that enterprise software lags open source software by at least 3 years , in terms of innovation. This lag can clearly be accounted to the legacy lovers who insist on using their accustomed softwares. Where does product development go in such a case. Office 2007 is seeing very slow adoption due to a change in the usability. Will this set of users be responsible for the sluggishness of product development? who will convince these users to adopt newer software? more importantly how? What will these users demand 20 years from now?

Its a strange question, but yes its an emerging market.

Google is going to put a cookie in your browser that will record information everytime you read an ad served by Google. The cookie helps Google predict what kinds of ads the person clicks on and will probably help serve better and more relevant ads. Up until now the ads are served based on the content of the site and also the publisher’s preference on the ads. The  cookie also doubles up as a proof of click ( think click fraud) and more material for publishers to analyse the hits to their ads and mine for patterns. Though the purpose is for ads, Im happy its a step ahead in personalization. The cookie acts like a distributed store of personal information that can be mined by ad servers and then acted upon.

The possibilities of this technology could be endless. Based on my previous post, where I wrote about a distributed implementation of a personalization database which all applications can use to customize data better, google’s cookie is one step towards such a solution. If every browser had cookies of personalized information then applications ( aka good applications) can mine these cookies and get useful information about the persons Net habits and preferences. But a cookie has size restriction and cannot probably be used to store the abundance of personalization information. You can use a add on activex plugin or a firefox addon like google gears with underlying SQLite database and do wonders with it. Of course this would mean about two years of thinking of standards and more bureaucratic gibberish, but once done, it will be a truly distributed system and the web, partially personalized.

You could be served the right category of blogs on wordpress, get the right videos on the opening page of you tube, see the relevant news on CNN , see the most relevant feeds on your RSS reader and much more. Personalization ahoy !!

What if you want to hire someone to build an online community? Somebody to create and maintain a virtual world in which all the players in an industry feel like they need to be part of it? It would help if that person understood technology, at least well enough to know what it could do. They would need to be able to write. But they also have to be able to seduce stragglers into joining the group in the first place, so they have to be able to understand a marketplace, do outbound selling and non-electronic communications.

Seth Godin writes about the Online Community Organizer as the job of the future.

I agree with Seth on this point. Apps are becoming more and more social. The social product is only as good as the hype it creates. Take Kevin Rose’s Pownce or Guy Kawasaki’s truemors , these sites generated enough hype prior their launch because of their high profile entrepreneurs. Almost every person wanted to take a peek at these products when launched. Aren’t these people organizers in their own ways ? Marketing a product has been closely associated with getting Techcrunched or Read/Written, its word of mouth marketing at its best and these people are the ones that are drawing audiences and giving them reason to stay.

Even Joshua of Social Design vouches by it. After a certain point, it will be people who will influence the audience of a social app. I sure as hell would join a social network where I could find the Real Bill Gates, if he can be there, so can I, thats the idea. An Online community organizer will have to be one with an impeccable reputation and a neat resume, one that can turn heads. He/she must be able to talk people into joining a product and using it and even deliver the promise of a fruitful app. Its not enough having high profile people in your Organizers list, you need to a have a good product. Marketing will only take you half the distance, the product has to endure the other half of the audience’s prejudice. Its a tricky predicament to let people from all walks of life into a product that was never meant for them; your organizer must have been a patron of the genre or must be at least influential enough to draw audiences of the right kind.

Its indeed a job for the future.

I had written about Amazon’s Mechanical Turk and how it used the power of collaboration combined with automated project management to get arduous work done from people. Taking and extending on the same paradigm are newer applications that try and achieve some good from these collaborative applications. Its like the Seti project which uses your computational resource when idle, these applications use the power of human intelligence to contribute to a greater cause.

Take GalaxyZoo for example.  A project from Oxford, the site aims at using the power of collective human intelligence to aid in pattern recognition. Users are given images of spirals and ellipses to recognise and defragment. The images in essence are images from deep space and the project aims at mapping a virtual universe using GalaxyZoo.

Similarly you have clickWorkers from NASA, a project that aims at identifying craters on Mars. Another useful outcome from these breed of apps is the ability to help in digitizing scanned text or old books. Distributed Proofreaders and ReCaptcha both provide simple to read captchas that form parts of old books that were garbled up by OCR’s while being digitized. Users help in recognizing these garbled , machine unreadable  text in the form of captcha’s and the user input is then given back to the system.

Another interesting collaborative app is the Google Image Labeler. This attempts to solve a problem of tagging images , a usually boring task is made interesting by allowing users to pair up and brainstorm over words that they can come up with for images. Points are given as an added bonus and to keep the game interesting.

There are many other projects that are currently incubating in research labs which are toying with concepts that use this form of collective human intelligence. The challenge will be to streamline these apps into everyday applications and make them more inherent and not be looked at as an overhead. Take for example digg, they ask for captcha auths on submitting a new story. Rather than projects like recaptcha, why not have these captcha’s fit into sites like digg .  That way, the the number of captcha’s served up and translated will be more and will seem more useful to do.

Google Gears is designed ingeniously. Gears is an activex plugin on IE and an XPI on firefox(installables) . Gears then works in your browser for any applications designed to use the gears technology. The foremost application that uses gears is Google Reader, which can store and retrieve almost 2000 articles. The transition between online and offline web is supposed to be seamless, as in one taking over when the connectivity is out and the other when its back. In reader, you have to explicitly make the shift from online to offline, something like the work offline option in IE.

The core essentially consists of three modules :

• Local Server Module : Creates a local server like emulator in which you can cache urls javascript, image etc. The ResourceStore module is used to capture resources that have to be accesed using a URL. ie supposing you were visiting my homepage, the resources like my resume or my portfolio , that need to be accessed via URL’s can be cached. The other variant is the ManagedResourceStore which automatically updates the cached information. In each case the URL’s and the cache in general has to be specified in a manifest file. The LocalServer intercepts HTTP/HTTPS requests and serves them from the cache if URL is cached in a ResourceStore or ManagedResourceStore (which is enabled) and the cookie of that information is present. This happens irrespective of the network connection.
• Local Database : This module is taken as an enhancement to SQLite , the micro database in C that can be configured and run in under 250Kb of memory. The gears javascript library provides method to create a new local database and also allows from complete SQLite SQL operations.
• Worker Pool : Almost every web developer has come across the long running script -> continue or break dialog. This usually happens due to an infinite loop in javascript or sometimes a very intensive calculation. The downside to this is that UI becomes non responsive and no action can happen on the front end of the app. The worker pool helps the cause by running these operations in the background. People familiar with thick app programming can relate this to threading, but at the javascript level. But unlike threads the workers do not share state information. The workers can communicate with each other by passing message strings or use JSON.

This advancement not only makes offline web possible but also allows for a new breed of applications to be born. Applications that need a server only to sync and not to compute or perform functionality. Think of a MyBlogLog offline for example, you visit a blog where you have a MyBlogLog reader that you have already encountered before the system tells you about your past encounters and on what pages. A distributed social network of sorts!!

Ten years back you couldn’t imagine IM in enterprise software;today most of the enterprise vendors supply their own chat service for support and maintenance. The recent acquisition of Tribe.net by Cisco and the more recent IBM including a social network in the recently revamped Lotus suite are just indicators of how big this phenomenon is. The general know how is somehow concocted to a statement “regular productivity apps like email will be ditched“, is not all there is to enterprise 2.0.

Enterprise 2.0 will see penetration of tools like blogs, RSS, wikis , Instant messaging and collaboration, which can be defined as the aorta of web2.0 enter enterprise space. The benefits are tremendous in terms of productivity. A social network as a base for all applications makes the software more friendly , collaborative and builds better communication capabilities.  Instant messaging can replace those single sentence emails, wikis can replace knowledge bases. Community portals can represent newletters and bulletin boards. Discussion forums can help solve and analyse problems better. Mindmaps makes brainstorming more productive. RSS and its enterprise filtering can make information management and overload a little less cumbersome to deal with ( see newsgator and Knownow ).

Take Basecamp for example, it encompasses a true project management tool with 2.0 capabilities. Updates through RSS. Blog like memos, easy to use resource and task management interface all this with a very social feel to it. If you look at Zoho and their productivity 2.0 suite, its got everything that any business needs. Online Office automation, wiki tools, mail and chat , their newly released meetings and project all are excellent examples of how 2.0 can be introduced into everyday enterprise applications and not compromise on quality.

The race is on and more and more enterprises will compete to cover as many 2.0 technologies as possible. Not all of them will prove useful but they will be an asset anyway , if not that at least for the sake of competition, the 2.0 radar will be covered. If you are an  enterprise customer and is jittery about trying 2.0 , don’t fret, it may be too late when you do. You wouldn’t something as unimportant as new technology let your competitor get the better of you .

Click on the image to enlarge.

The biggies themselves are toying with these start pages as they see it as a personalized version of their already popular portals like my.yahoo or the Infamous Google search page. The idea is that there are already millions flocking to these pages, why not make it more viral and add to it personalization and provide a dashboard so that people just don’t have to go to another website. Though it has worked well for homegrown services, I don’t really know how other services that bank for users , whose revenues depend on users visiting their sites and generating pageviews, will reach to such an offering.

Its just evolution I suppose that the concept is dying a premature death, primarily due to the growing popularity of desktop widgets, thanks to the efforts of mac, google desktop and of course , Windows Vista. But still discovery of these widgets will remain online and technologies like Apollo and Silverlight, are making this process of discovery and done more easier. Using Apollo technology, YourMinis , another flash based startpage service will allow for drag and drop of their web widgets onto the desktop. Now how about that, you like a widget on your start page and so you decide to give it some attention by dragging it to your desktop. The concept is more fact than fiction with a lot more services working to achieve the similar desktop-web experience.

I guess my job of ranting on start pages is done, here are some popular start pages to start of with. You can even read about some start page reviews here.

Some start pages to get you started

The amount of contributors is very less. Thanks to the 2.0 era that made that figure possible otherwise that figure could look a bleak 1 or 2 pc. Also there are lot of non users who could become users in the next 5 years or so , adding that many more users to your service, thats potential. The amount of collectors also have to improve, its not enough if you have producers unless you have consumers. I had expected to see a value much more than the producers but it isn’t so.

As you can see, there has been a clear shift in way people are perceived in this age. Earlier it was the producer consumer model but now you have a lot more roles . The concept of an online identity has changed dramatically over the past few years and on this note, I have to quote Nicholas Carr’s article :

In a column in today’s Guardian, I examine how the invention of virtual drugs last week may open up lucrative new opportunities for pharmaceutical firms able to create therapies for the psychological ailments that beset avatars.

Here’s a snippet: “Up to now, avatars have led fairly narrow lives. Their main pursuits have been limited to fighting ogres and dragons and having simulated sex using artificial genitalia. Virtual reality has been like a pornographic version of Middle Earth. Now, avatars have a third and more modern alternative: abusing substances. Fighting, screwing, and getting wasted: Virtual life is becoming more like real life every day.”

Its funny where we have landed up in this confusing era of information, but the idea seems to be, now that we are here, lets make the most of it.

Ooops!! I also forgot to mention that the entire scheme is depicted as a ladder, meaning people eventually will climb up with age.

Thats where the biggies are trying to cash in whilst some startups are trying to consolidate. MS has their Axapta and BI suite and Google have their Office suite. MS sells their ERP Axapta and BI to SME’s at unbelievably low prices compared to the other tools around , infact, the products themselves have been designed for this target audience. Google offers their apps for a very low cost, where as MS gives their entire BI suite with database, ETL and Charting for the price of a little known database. MS also has a range of small business softwares which they sell to SME who don’t want to left behind in terms of technology like their small business servers and other software.

In India, there is a huge boom predicted in the retail sector. ERPs, CRMs, Warehouse management systems, portals , Business Intelligence reporting and other similar developmental work will see a upsurge in the next couple of years. People just starting up, an essential piece of advice, DO NOT THINK OF YOUR EXIT STRATEGY. Doing so will only deteriorate the quality of your product, and Im saying this because I see a lot of individuals talking about a concept , which isnt even built yet and they talk about their exits. Exits are for extremely successful or totally unsuccessul products, more so the latter. If you survive in the industry in a competitive place like India, its likely that you will be moderately successful and that you will have your worth adversaries. In such a situation you will have to strive to keep your product feature rich and niche and not many may want to buy your product if your competitor decides to sell it for a quarter of a million less. To talk about your exit strategy even before building your product isn’t really a good idea and flaunting it around definitely isn’t.
The SME space is buzzing and it will be worthy seeing a domination in that segment. Right now its open for play, so start pitching in those great ideas. If you are looking for a place to present your ideas, I suggest Proto.in happening in the first week of June, rush in your entries.