“Panace is an Augmented Reality application that uses real time face/image recognition technology, and provides for Contacts Management and HyperLocal communities inside an Enterprise.”

You can find the presentation about Panacea below, or find it on Slideshare here.

The core idea behind the product is that, inside a closed community, it should be really easy to exchange contact and schedule information. Just like how inside an enterprise there is a Global Address Book and also shared calendaring. Panacea extends that by associating a facial identity with each person in the closed community. I mention closed community a number of times, for even I cannot fathom the privacy implications such an application will have in the public domain. We used Face.com’s face recognition technology to aid in Face Recognition and used OpenCV’s face detection algorithm by accessing their compiled libraries via JNI calls on the android platform.

Scenario 1: You meet a colleague in your organization or at your local community meet. You need exchange contact information. Now, instead of exchanging visiting cards, you could just switch on our app, point the camera at the person, wait till the algorithm recognizes the person, and then add the person’s contact information via a single click. Better explained by a video demo of the solution.

Like any pattern matching algorithm, face.com’s algorithm is also a trained algorithm. Better the sample set or the reference set, better the matching. To make the algorithm work consistently, we need to train the service by sending at least 7 to 8 different photos of the person and associate a unique identifier for that set. When a candidate input is presented, the image clicked via the camera on the phone is sent to face.com and we get back that unique identifier with a matched probability score. Now, this unique identifier could be anything from your corporate id or your twitter handle. The important thing is that there is a backend system which you can query using that unique identifier and retrieve contact details associated with that id.

There is a problem though – I cannot ask every person in the community to upload 10 mugshots to our system to be included in the training set. I would need a more scalable solution. We solved this problem by providing a training mode in the app itself.

Scenario 2: If you realized that one of your colleagues or friends isn’t registered on the service, you can switch on the app and switch to training mode. Start the training by pointing the camera to the person and associate the unique identifier for the person. You can then fill in the contact details yourself or fetch it from the centralized contacts store. Follow the progress bar in the training mode and wait for the training to complete. Once done, any user of the app can now add your recently added contact using the mechanism described in scenario 1. Adding a new user to the application & service is crowdsourced. Again, better explained with a Video demo.

Now that we had sort of envisioned this technology and its working, fleshed out some initial prototypes, we brainstormed about how this technology can be used to solve other problems. I was also in the market to buy a smart phone myself. I did some research and found out that smart phones in India, with Assisted GPS support, started at around Rs 12,000. But high resolution camera phones were selling as low as 6. Also, I was fascinated by the whole location service apps like FourSquare and Gowalla. It turned out that only 13% of all smart phones sold in the world were location aware. This means there was a big market for location based services for non GPS enabled smart phones.  This is how we tried to crack this problem.

Scenario 3: You have a smart phone with an internet connection but no GPS. You want to use a location based service to get some deals on items, check into the place, or just give some feedback or tips for other travelers.  Our application defines what we call as a PlaceMark, a unique image of some sort which is linked to a location in our service. To perform any of these functions, open our app, point to the place mark and wait for the location features to load up. You will now get a feed of past conversations, tips, tricks shared by people at that place. The owner of the location can also target people by providing them deals for checking in and giving a social recommendation of sorts. You can even use these as mechanisms to receive feedback, complaints or engage in a Q&A at a large conference. The possibilities are endless. The example video shows a distributed discussion happening on the wikileaks issue. You can join this discussion by using our app and start conversing. The app even sends notifications to your phone when new messages come in.

This was mostly what we came up with. There are plenty more things that can be done with this application, and this, or a similar technology, can be used to solve many other problems.  We also want to take this product to the market and see what responses it receives. Feel free to leave comments.

One of my side projects has a very active code base. We use Git as our Source Code Management system and Hudson for Continuous Integration. Since most of the application backend is pretty stable, the bulk of the changes on our trunk are mostly changes in logic, new features , enhancements and bug fixes. We also run a dogfooding instance where we push the latest changes and test our changes. On our Linux server, we use Tomcat as our app container, MySql and Cassandra for datastores, Solr for indexing and all builds use Gradle. Till now, pushing changes was a manual task. Here is how I automated this task. @prateeksha , who is a unix command line ninja, was kind enough to guide me through the entire process.

Pushing changes involved , updating the code base to the latest master branch, doing the database migrations and restarting the App Container. Though I still have to get my head around database migrations, here is where automation stands now.

- The continuous deployment script can either be triggered by a git Post-Receive hook or by using a Cron Job.

-  If triggered by a Cron job, we need to poll git for changes. I found this neat little script that figures out whether there have been changes to the source repository and prints out the status of the repository.The script is shown below:

#!/bin/sh
        remote=$1
        o=.git/remote_cache.$remote
        n=$o.new$$
        git ls-remote $remote >$n
        if [ -f $o ]
        then
                if diff $o $n >/dev/null
                then
                        echo "No Changes"
                else
                        mv $n $o
                        echo "Updated"
                fi
        else
                mv $n $o
                echo "New remote remembered..."
        fi

If the return value from this script is “Updated” then changes from Git are pulled, Gradle build scripts are run, and the app container(Tomcat) is restarted. Using Hudson’s wonderful API’s, you can check whether the latest build of the entire project is successful or not. If it is successful, we can either take Hudson’s build artifact, or build the project again and deploy it.  Also, a small little utility extracts the log information, date and time of the push and presents it on the App itself. At any point in time, if you want to know what version of the App is actually deployed, you can see a version page which displays deployment time and commit log. Screen shot below.

Click to Enlarge - Version info of the App

This has been my first experiment with Continuous deployment, I still need to understand and solve the migrations issue. Next? More experiments with Chef and Puppet coming up.

1
2
3
4
5
6
7
8

javascript:
   if(navigator.userAgent.indexOf('Safari')%20>=%200)
    {
         Q=getSelection();
   }else{
    Q=document.selection?document.selection.createRange().text:document.getSelection();
  }
   location.href='http://wordpress.php?username=blah&text='+encodeURIComponent(Q)+'&popupurl='+encodeURIComponent(location.href);

You will see that this is calling a RESTful implementation of some service with query parameters and the whole code is stored and rendered from inside the bookmarklet. Lets leave the security aspects of this implementation aside, that will require another post in itself. Such a bookmark is a nightmare waiting to happen. Imagine the trouble you would face if you changed the implementation tomorrow. You will have to write to ensure backward compatibility or take the user to a page where he/she can upgrade to the newest version of the bookmarklet. This is a bad way of delivering software.

Ideally, the bookmarklet in accordance with a shortcut/bookmark must mainly be a pointer to the actual web resource. Even for trivial bookmarklets, make sure you follow this design principle because you might want to do something different or more advanced later on. I am sure you have come across situations where you realized the bookmarklet you wrote didn’t work on a browser like Opera or Safari. It may be too late to correct it later on if people have already installed it on their browsers.

Lets now take a bookmarklet that loads the code to be executed from a script on the web. This is now the preferred means of deploying bookmarklets as it puts the control back into the hands of the web developer. This example is from the Friendfeed bookmarklet that lets you share things on FriendFeed.

1
2
3
4
5
6
7

 javascript:void((function()
  {
    var%20e=document.createElement('script');
    e.setAttribute('type','text/javascript');
    e.setAttribute('src','http://someserver.com/bookmarkletcode.js');
   document.body.appendChild(e)
  })())

All this bookmarklet does is to append a new script tag in your html page and assigns the source to the bookmarklet source. The difference now is that there is no code on the client side that is tied to an implementation. You can change the code in the bookmarkletcode.js as and when you feel necessary and be assured that the people using your bookmarklet will always use the latest version.

I was supposed to be involved in a discussion about cloud computing at Cloudcamp Bangalore, but due to other commitments, I could not attend the event. I had a small writeup about the limitations and challenges in Application clouds. Here is the full text of it.

Cloud Computing is a way of providing dynamically scalable and available resources such as computation, storage etc as a service to users who can use it to deploy their applications and data. Cloud Computing can handle data in both the public and the private domain. But this seemingly harmless way of thinking about building applications has its own set of issues.I am primarily referring to application cloud providers, the kind where you deploy your applications. Not storage and service clouds. Google AppEngine would be a good example for the cloud that I am describing. I note some of them here :

From the Users perspective:

1. New unstructured and non standard paradigm of programming: Each cloud has its own supported programming language and syntax requirements for programming, though most of these clouds expose the typical hashtable based cache and datastore interfaces. There is an urgent need for standardization of interfaces and methods of programming them. One of the reasons why shared hosting environments work great is because , as a programmer, I know that I can move my PHP/PERL code to another server and it will work without too much of a fuss. Moving from one of the dozen odd cloud providers to another requires considerable developmental efforts, not to forget time (for businesses, this could spell doom).  A look back at history shows languages like SQL, C etc being standardized to stop exactly this sort of undesirable proliferation.
2. Restrictions on the programming model : For cloud based applications to be highly available, they must be easy to dynamically mirror on multiple machines. Once these applications are mirrored, they can be served on demand by load balancing servers which makes them highly available and the user doesn’t face delays in being serviced. This is an old trick used by busy websites from the early days of web publishing but these solutions were custom built for websites. So, extending this concept to cloud based platforms, servicing thousands of applications, mandates the platform providers to automate this task of replication and mirroring. This job is easier said than done. This process can be made seamless when the program stores as little state information as possible. By state, I mean transactional variables, static variables, variables in the context of the entire application etc. These things are almost a given in traditional programming environments but are very hard to come by in cloud based environments. The unnatural way of dealing with this situation is using the datastore or the cache to store state of an application. There are a lot of restrictions like lack of privileges to install third party libraries, no access to file system to write files etc ( which forces you to use the datastore and pay for it)
3. A good local debugging experience: A good local development environment, debugging experience is a must for programming on the cloud. Most cloud providers do not provide good local development environments. There is also a lack of good IDE’s that can help with programming and debugging programs written for the cloud. The providers that do provide a local debug experience, do not simulate real cloud like conditions. Both from my personal experience and from conversations with other developers, I have come to realize that most people face problems when moving code from their local development servers to the actual cloud. This is only due to inconsistencies in the behavior of the local dev env compared to the cloud.
4. Appropriate metrics and documentation of programming best practices : On a cloud, since a user pays for almost every CPU cycle, appropriate metrics on usage of processing time and memory must be presented to the users. Typically a profile of the application with function names and their corresponding time taken, memory used, processing cycles used will definitely help the developer tune his/her code to optimize on usage of processing power. The best solution for this is for cloud providers to abstract common code patterns into optimal libraries so that the users can be assured that they are running the most optimal code for a certain operation. An example of this is Apache PIG, which gives a scripting like interface to Apache Hadoop’s HDFS for data analysis. Also, Most cloud providers do not provide enough statistics and also profiling capabilities.

From the providers perspective:

Here I look at challenges that cloud providers have to face:

1. Ensuring availability of the cloud: This proves to be crucial as Clouds host critical business applications, for whom, downtime would mean monetary losses. Effective monitoring and load balancing solutions are to be built. Most clouds employ virtualization technology to get the most out of any resource. In such cases, tools should be written to figure out a resource hog early and move the application to a more powerful grid or a machine, so that the other users get their share of the cloud without delays.
2. Ensuring Consistency: Both the data and code is replicated on the cloud and maintaining consistency of data is extremely crucial. This is the reason why most transactional updates are not allowed on the cloud. Example: sequence objects, which are almost a given in traditional databases are not provided, probably because maintaining state across machines for such statements is non trivial. Problems like distributed updates, locking, partitioning, sharding etc  arise when dealing with data. Such constructs are to be provided to the users as most of it is given in the non cloud deployment space.
   Most datastores provided by cloud vendors (except the ones that provide cloud based database services) do not support relational models. Which means all object relations have to be programmatically established. This could always lead to bad code, unnecessary joins, cascading problems and tons of other problems that developers faced before working with relational datastores.
3. Program verification : One of the biggest worries about deploying applications on the cloud is the correctness of the program in execution. Erroneous conditions, like infinite loops, can not only put the machine at the risk of being overloaded and unavailable, but also cost the user a significant amount of money. Tools like static analysis should be used to analyze code uploaded on the cloud and it should be checked for infinite loops, possible race conditions,  null references, unreachable code etc. The code uploaded should also be optimized or suggestions should be provided to the users about how they could optimize code to best utilize the available resources.

Conclusion : The cloud should become a complete nonrestrictive platform for applications. There should be no restrictions on the constructs, functionality and privileges on the cloud. Also, it should be dead simple to move everyday applications onto the cloud without too much of rework. This could mean writing migration utilities, import/export options and other artifacts that make the transition to a cloud much easier.  This will prove essential as most live applications, at least currently, do not run on a cloud and helping them migrate easily will mean more revenue and adoption.

 Lets take the example of facebook. The apps killed the facebook profile page. There were days when a single page would take 15 minutes to load thanks to the 50 odd apps that people had added to their profiles. Facebook’s new design rocks coz they have removed the clutter. Facebook’s usp, the newfeed, is what is shown as the profile , which is nice considering you get to see what the person you are seeing is upto recently. The apps have moved to a completely new page which means, no clutter Also the profile information, which people hardly change is gone into another page. Another cool addition I saw is the “What are you doing now ?” is highlighted and big. Given the run twitter is having with its fans, Im sure facebook realizes that micotexting about yourself is the rage right now and integrates that at the head of your homepage. Pretty ingenious if you think about it. Not all redesigns work , for ex , I hated the analytics redesign for the first few days and so was the redesign of wordpress but regular usage and some groking made me very comfortable with the interface. When you are redesigning interfaces especially for sites like flickr, facebook then it becomes essential that you get a value add and Facebook certainly nailed this one.

 Now lets go back to del.icio.us. Firstly the dots are off and you can access it as delicious.com, which saves me the trouble of explaining people the position of the dots in the url. I like the old del.icio.us and its UI. It was very basic, no frill, plain text based UI. The reason was probably the speed which it would work. But one thing that annoyed me was the clutter.The URL, Description, tags, other taggers, edit  and remove links etc were fit into a really small space; the new UI had to break this clutter.  The other problem was the tag cloud on the right. The older version would buch up tags and it would be difficult to spot the tags if you had more than say 200 which is common for most delicious uses. The new UI does a wonderful job of reducing the clutter. All the above mentioned items of the URL are still present in the same space and look darling. The tag cloud has also become more navigable and the social aspect of the site is utilized superbly if you see your networks page.

 I am now looking at some of the properties that I have built that can possibly be redesigned for better usage.

Tags: interfaces, redesign, facebook, delicious

 In fact, for any third party application, there must be some scalability requirements enforced and only when they meet such requirements should they be allowed to publish. An alternative would be for the mother app to build its own cloud and make sure that the 3rd party application uses that cloud so that its highly available ( Case in point Google’s App Engine – Use for widgets). Or, there must be some load test designed when the 3rd party app registers with the mother app ( ex : Automated load tests on sites like TopCoder ). An even more elegant way of figuring out scalability problems would be to statically analyze the submitted code to check for bottlenecks and other possible leaky parts of the code and then reach a decision whether to allow for the code to run or not. Until such rigorous checks and constraints are not enforced, third party apps will keep ruining our experience.

Tags: cloud, facebook appliation, third party applications, static analysis

Data has far outgrown the domains of just text. Today we talk of mutlimedia data, urls, semantic data and many more application specific formats. Information on the Web is in JSON, REST , XML , Microformats etc. With this vareity in data formats and representations comes the inherent need for flexibility in storage and querying of such information. Almost all database users know of the conceptual modelling required for the design of any database, the key principle being that more tighter the model, more efficient the database. The integrity of the database is only as good as the integrity of the data. But you cannot talk of data integrity with the kind of formats available today.

Clearly markup data dominates the web . Though databases have developed features to better support , store and validate markup data , the initial design of databases was never to store the wide variety of loosely organized data. Querying of such markup data is fruitless and so is the attempt to index, sort , aggregate this data. To develop a custom database capable of all the above mentioned operations could be a solution, but the given the non standardized nature of this data and its probability of change, you would have a tough time scouring the web to search for changes. Plus these databases will not be semantically inter operable.

Developers are taking notice of a new scheme of storing data, I call it the bucket store. The design is roughly the same as that of a hash table, where data blocks are stored in buckets and hashes are used to index or refer to these buckets. A little improvisation in terms of adding upper layers like domains, groups and so on to complement the schema, table in a database is done to make the data easily classifiable. The advantage with this scheme is heterogeneity in data formats and the absence of constraints.

Several products are offering such services at dirt cheap prices. Take Amazon’s S3 or the recently launched Simpledb or CouchDb which offers a host it yourself version of this storage. Amazon S3 has businesses running on top of it; of the many I can recall Slideshare running on S3. With the advent of more mashups and heterogeneous data being churned out by the web more of such non DBMS related storage options will be employed. Given that this paradigm does implement all the enterprise important features like security, access control , backups, transactions etc and mature modeling methodologies that can rival the ER are proposed , I don’t see any problem in this becoming the most viable and cost effective option for data storage.

Comet is essentially an design principle which aims at achieving true push technology using http. Comet is not a technology in itself,  cometd is an implementation effort to make such a paradigm possible, but comet itself isnt a technology. For real time systems and other applications like stock tickers you don’t need an entire page refresh, its enough if you can get the updated values in your respective places. Ajax seemed to be the answer for sometime with browsers asynchronously requesting for changes and then updating the same in DOM. The problem with  Ajax was the polling, since the job was asynchronous, there was a interative pattern of asynchronous requests and replies. Even though there wasn’t any change in the server values, the client would request repeatedly to check if the values changed, which led to the server being overloaded. The solution was to push the updates to a client without the client requesting for it. Sorta like the good old days of socket programming.

There are many solutions to this problem, each one with its own set of advantages and disadvantages but I shall explain a single implementation that I worked on. Its called the Long Looping method.

 Long Looping Method :

This method involves making a request to the server which in itself will reply only when there was a change. Your browser will keep showing you the loading icon coz the server hasn’t replied as yet. The server end can achieve the desired result by either suspending a request to a thread that will wait till notified or a putting the request through an infinite loop. Once there is a change of state on the server the loop ends and the reply sent. The client receives the reply and immediately sends another request for further changes and so on.

In this way you can see that, there is always a low latency http connection between the client browser and the server. The delay in state changes to the server is very low and will definitely be ten orders better than the regular ajax looping method.

I had written about Amazon’s Mechanical Turk and how it used the power of collaboration combined with automated project management to get arduous work done from people. Taking and extending on the same paradigm are newer applications that try and achieve some good from these collaborative applications. Its like the Seti project which uses your computational resource when idle, these applications use the power of human intelligence to contribute to a greater cause.

Take GalaxyZoo for example.  A project from Oxford, the site aims at using the power of collective human intelligence to aid in pattern recognition. Users are given images of spirals and ellipses to recognise and defragment. The images in essence are images from deep space and the project aims at mapping a virtual universe using GalaxyZoo.

Similarly you have clickWorkers from NASA, a project that aims at identifying craters on Mars. Another useful outcome from these breed of apps is the ability to help in digitizing scanned text or old books. Distributed Proofreaders and ReCaptcha both provide simple to read captchas that form parts of old books that were garbled up by OCR’s while being digitized. Users help in recognizing these garbled , machine unreadable  text in the form of captcha’s and the user input is then given back to the system.

Another interesting collaborative app is the Google Image Labeler. This attempts to solve a problem of tagging images , a usually boring task is made interesting by allowing users to pair up and brainstorm over words that they can come up with for images. Points are given as an added bonus and to keep the game interesting.

There are many other projects that are currently incubating in research labs which are toying with concepts that use this form of collective human intelligence. The challenge will be to streamline these apps into everyday applications and make them more inherent and not be looked at as an overhead. Take for example digg, they ask for captcha auths on submitting a new story. Rather than projects like recaptcha, why not have these captcha’s fit into sites like digg .  That way, the the number of captcha’s served up and translated will be more and will seem more useful to do.

Google Gears is designed ingeniously. Gears is an activex plugin on IE and an XPI on firefox(installables) . Gears then works in your browser for any applications designed to use the gears technology. The foremost application that uses gears is Google Reader, which can store and retrieve almost 2000 articles. The transition between online and offline web is supposed to be seamless, as in one taking over when the connectivity is out and the other when its back. In reader, you have to explicitly make the shift from online to offline, something like the work offline option in IE.

The core essentially consists of three modules :

• Local Server Module : Creates a local server like emulator in which you can cache urls javascript, image etc. The ResourceStore module is used to capture resources that have to be accesed using a URL. ie supposing you were visiting my homepage, the resources like my resume or my portfolio , that need to be accessed via URL’s can be cached. The other variant is the ManagedResourceStore which automatically updates the cached information. In each case the URL’s and the cache in general has to be specified in a manifest file. The LocalServer intercepts HTTP/HTTPS requests and serves them from the cache if URL is cached in a ResourceStore or ManagedResourceStore (which is enabled) and the cookie of that information is present. This happens irrespective of the network connection.
• Local Database : This module is taken as an enhancement to SQLite , the micro database in C that can be configured and run in under 250Kb of memory. The gears javascript library provides method to create a new local database and also allows from complete SQLite SQL operations.
• Worker Pool : Almost every web developer has come across the long running script -> continue or break dialog. This usually happens due to an infinite loop in javascript or sometimes a very intensive calculation. The downside to this is that UI becomes non responsive and no action can happen on the front end of the app. The worker pool helps the cause by running these operations in the background. People familiar with thick app programming can relate this to threading, but at the javascript level. But unlike threads the workers do not share state information. The workers can communicate with each other by passing message strings or use JSON.

This advancement not only makes offline web possible but also allows for a new breed of applications to be born. Applications that need a server only to sync and not to compute or perform functionality. Think of a MyBlogLog offline for example, you visit a blog where you have a MyBlogLog reader that you have already encountered before the system tells you about your past encounters and on what pages. A distributed social network of sorts!!

The fact that we don’t understand what value others get from social web apps is part of the paradigm of social software. The key is that each person has their own social lives, their own social circle, and thus their own social values. What is important to their social life will almost certainly be unimportant to us because we have our own to worry about.

Think of it this way. Each person has their own social network. Chances are that social network overlaps very little with yours. If, say, that person wanted recommendations for watching a movie, they might turn to their social network, which is made up of their family, friends, and colleagues. They would ask these people, the people they know and trust, what movies they recommend.

Now, would you turn to the same social network for movie recommendations? Of course not. You trust the people you know…your social network, and so any of the chatter from their social network has no value for you. It’s meaningless chatter. Just like most people’s Tweets on Twitter.  In a larger sense, this opaque value problem affects most social software. Twitter, MySpace, Facebook, most social software is built around providing personalized, socially-focused conversation. It’s person-centered and as a result is difficult for anybody but that person to really appreciate: the value becomes opaque in this way. Designing social software is going to be very difficult. The designers won’t be able to put themselves into a position of someone who wants to keep up to date with their social network, which is something that all of these sites are doing.

What the excerpt from the original blog clearly states is that a person using a social software has his/her own reasons to do so. The fact that the others don’t know what value the other person sees in a social network is the entire basis behind social software. I may log into myspace to chat with my peer group, but I dont know what my friend X does in her network, so what do I do ? I snoop around their profiles and walls to get a glimpse of their world coz her value from the network is opaque to me .

The goal of social software is evolve from this paradigm to a more productive one. Efforts are on to make social software enter the productivity space and that too with good success. Based on all of these observations, how do you design for a social concept. How do we actually add value to the users ? Its not easy to please everyone, but at least if the majority is kept happy, you can be sure of winning product.

That was a lesson, no harm done, no money lost. But there are some applications that have to be delivered to be robust , scalable and always available. I found this article which highlights some rules for high performance website. I suggest you visit this site and read about these rules, its worth it .

• Rule 1 – Make Fewer HTTP Requests
• Rule 2 – Use a Content Delivery Network
• Rule 3 – Add an Expires Header
• Rule 4 – Gzip Components
• Rule 5 – Put CSS at the Top
• Rule 6 – Move Scripts to the Bottom
• Rule 7 – Avoid CSS Expressions
• Rule 8 – Make JavaScript and CSS External
• Rule 9 – Reduce DNS Lookups
• Rule 10 – Minify JavaScript
• Rule 11 – Avoid Redirects
• Rule 12 – Remove Duplicate Scripts
• Rule 13 – Turn Off ETags
• Rule 14 – Make AJAX Cacheable and Small

I did some research on S3 sometime back when I intended to use it for some project, and found it really compelling to use. There were lot of problems reported by users and some outages also which hampered lot of sites who depended completely on S3. There is also the problem that your traditional taxonomic structure that you were working on will now be replaced by a Hashtable like datastore that you essentially query. I found it a great way to store shared media, photos and other non text based information and rightly so. S3 fits in as a replacement to a database in the more traditional sense, but it has its own limitations, but its power nevertheless.
So what is this IAAS( see title) , according to me its the death of the network engineer.Gone are those days when you would rent our servers and rope in DBA’s and Network engg who would manage the hardware and availability part of your application. You want unlimited computational scalability? try mediaTemple’s Grid server, want unlimited datastore? try S3, want scalable content delivery? try Akamai Edge. Infrastructure management is at an all time low in the 2.0 era, especially since technology is more easily available , manageable and scalable. Infrastructure is also turning into another SOA where pre built components define how you develop and deploy your applications and you don’t have to worry about downtimes, reliability, scalability and other metrics that are considered important for any web service.
Whenever such disparate elements contribute to a unified cause the next step is  consolidation. But will it ever happen ? these service providers bank on the fact that you need these systems to run your application, otherwise you wouldn’t need a database or a blade server. Have services like S3 given companies like oracle a run for their money? Will there be a similar offering from the other majors?
Picture Oracle providing database hosting on their enterprise level infrastructure for a minimal cost. Updates and upgrades done automatically, maintenance not a pain anymore. Its a space worth debating on and worth watching out for.

  2.  Use your JS wisely
         I have seen a lot of apps currently using a wealth of js libraries, and since your js needs url to access, the XMLHttpRequest object with its url is wide open for the entire world to see. You dont even need to look at the source , with a sniffer like firebug, you can see all the async requests made, with the url and the response. Scripting attacks are the most prevalent of the security attacks and people can do wonders with malicious scripts. The trick here would be to confuse the hackers. Obfuscate the scripts wherever you can, there are good JS obfuscating libraries available. Be advised , obfuscation is usually irreversible , so use it wisely. Use POST instead of GET for async requests and make sure its the right person requesting the data. You could probably generate a sessionid of sorts on the server end and pass it to the browsers as a key to use for any further requests. And please, don’t rely on cookies, they can only go this far.

  3. Follow the MVC methodology
      it usually helps in abstracting the finer points of the application. Separating out the data, logic and accessibility will usually give you more control on what goes where and a lot of server side validation.

  4. SQL Injection
   I understood the complexity behind these attacks only recently. A seemingly simple line of DB code could prove disastrous for applications. Always use good wrappers around your db code. Many good SQL Helper libraries help you against such attacks and the overhead is adding another server side library, which is not so big considering it can save your application.

 There is also the new generation of security threats thanks to Ajax and is open ways. But there are safer alternatives being developed like JSONRequest which abstracts the JSON based calls  and provides a safe way of making sync calls.

  I shall write more about web based security when I get time.

I remember studying Advanced computer architecture and parallel computers in college. Just the sheer size and complexity of the book gave me nightmares. But lately, I have become interested in the field of Scalable programming, application clustering and parallelization. One of the reasons, I lost out qualifying for the Google Code Jam was because my algorithm wouldn’t scale, the logic was pretty correct though. And now, after almost two years of my tryst with building a web app and watching it crash before our  very eyes has given me the understanding of why its so important to build good scalable applications.

Now scalability can happen at many layers, the code, the design / architecture and also hardware, the last one being the easiest – more power. But the first and more importantly the second make up for all the pain. Developing scalable code has been discussed a million times with lots of tricks on efficient memory usage,controlling memory leaks, serialization and the critical section problem but those scale only to a certain point.

At barcamp, the Yahoo team presented Hadoop. A distributed file system plus a parallelization framework that can be built for very computation intensive applications. The framework provides api’s that you can use to parallelize your tasks, store in a distributed way and more. There has been an upsurge in people developing processor and memory intensive applications, mainly due to the growth of concepts like crawling, semantic web etc. The 2.0 moguls who are trying to build new age crawlers, and indexers with a niche like social media, photos or videos will be the ones actually interested in these frameworks.

So how easy is it? Take a big bunch of machines with a 2GHZ processors ( considered outdated) ; must not cost you more than 6-8k a piece, add to each box 2 * 40 Gb hardisks, That you can pick up for almost 1k a piece. Install a stable flavor of Linux , put them on a network and your done!! So a 7 processor server grid with almost 1/2 Tb of storage will cost you in the order of Rs 60,000 , the price of a good laptop. Also for those who have startups, the hardware is already there, you just need the right software. You can use your existing resources to run a massive computation engine, typically in the nights when there is no work being done.

There is respite for folks who use windows, MS has made it extremely easy to achieve grid computing with their Compute Cluster Product. Whats more, its MS so it will be easy to install, with a lot of documentation and visual studio plugins for execution, profiling and debugging. Comes at a price though, but the results that it can achieve are tremendous.

Its almost like we have come full circle, we go on from these core concepts in the pre web era and now we are back to where we started with these. For more resources on cluster computing go here

  http://www.clustermonkey.net/

http://www.beowulf.org/community/index.html

1 . anticipate what the users expect and provide those features.
2. Autonomy : giving the users a sense of mastery. and providing status messages.
3. don’t rely only on colors , many are color blind
4. consistent commands and shortcuts. don’t change usage patterns and standard shortcuts

In house consistency and maintaining the template across your application
common icons
use inconsistency when necessary, like when you need more marketing attention. change the login space to somewhere else
5. worry about users productivity , not yours, give easily manageable patterns.
6. clear help messages. use the keywords first and emphasize
7. explorable interfaces with colorful icons and images.
give users stable perpetual cues for the sense of home
make actions reversible, example gmails messages.
8. Fitt’s law : the time to acquire a target is a function of the distance to and the size of the target.
corners of the screen are the most easily accessed.
large objects, images for important actions.
9. push latency to the background and reduce latency
visual feedback – for background activities, typically for ajax writebacks and DOM changes.
Indicate the potential length of the wait, if you can .
8. metaphors.
9 . imporve readability y using contrasting fonts and colors.
8. track the state of users behavior.
10. give visual navigation wherever possible.

this was a live blog. my first live blog.

Whilst coding some of my infant steps in Ajax, I looked around for hacks to the back and the refresh button functionality but was unable to find any . Thankfully I did read some useful pointers about hacks on the navigation front from Brad Neuberg( thanks YUI blog for getting me back to this article). Even a old school article about how not to disrupt user behavior when developing Rich apps.

But there is hope in the horizon and a new addition to the recently released YUI v2.2.0 intends to do just that. The new version of YUI has browser history manager, a simple library that helps in developing rich apps, all ajaxified, yet maintaining the traditional behaviors like bookmarks and navigation. Its still experimental, but worth a look if you are trying to break that browser history barrier.

Almost all of my favorite websites, be it flickr, meebo or more recently gdocs, have been internationalized to more than one language. As internet defines a truly global network, it just makes sense that a popular service must support more languages for people in countries like Russia, Germany or France to use.

Though most of these internationalization happens through wikis and tools, I don’t think they really cling on to the colloquial essence of the language, which is generally more acceptable. The problem lies in the dual paradigm that in most languages in inherent, one being the speaking or more familiar part of the language and the other being the official part of the language. Take any Indian language for example, I know Tamil and am fluent in conversing, but I cannot understand a single word in any of the news channel. The language on news channel is more of the official type, with official words, one that you would find in government forms or letters. Its understood most of the languages have this dual paradigm and most website functionalities point towards the official language. Clearly people cannot connect to that language on the official level, try localizing your phone and you will find yourself struggling to change the setting back.

Isn’t it a waste of time internationalizing a site and not allowing relevant people to relate to it? I see this becoming a profession for many a people , as internationalization becomes a necessity for any site. The trick will be to develop it with a sound understanding of the local (s)language and provide more familiar words than to confuse a person with official gibberish. Even developing applications will require forethought about internationalization and certain design patterns have to be applied to display and internationalize a site with least possible effort. I am aware of such patterns and methodologies but haven’t really seen them being standardized or used appropriately. Its catching on and pretty fast !!

Educators and teachers in under developed countries suffer a poor a student to PC ratio and this technology will enable them to make any students work together on a single PC. All you have to do is buy USB slots and more mice, the technology will provide distinct colored mouse pointers ont eh screen for each mouse. Students can then work on a collborative enviroment, porbably on a quiz, a puzzle, a painting  or a game of some sort.

A SDK is also ready to let people harness this technology and is available for download here. If you feel your application is something that can work better in such an environment then go ahead. The scheduled release date is sometime in May.