Think of the same thing in software. A software that is programmed based on generic constraints and the software dynamically learns from its environment through input devices like sensors and then adapts to the changed environment. The classical shortest path problem can be taken as an example. If a crawler running through the shortest path is lifted from its path and put somewhere else, will it be ale to comprehend the change and then quickly adapt or will it be lost. What if the graph changes and produces a lot of cycles or what if the problem statement changes during the course of the program.

I know that a very few people are relating to what they are reading, but the belief that a machine can learn and adapt is what scientists are trying to prove everyday. Machine learning is taboo after movies like iRobot and Terminator, but trust me, we are far far away from something like that. If we can solve a subset of problems of adaptability through an expert system, that will be an achievement in itself.

But how do you go about designing software with such requirements. I would go one step ahead and call it no requirements or changing requirements with no defined thresholds. No current methodologies like  OO or aspects can cater to such a requirement. Probably a new scheme of designing learning software has to be developed. Taking tips from AI and the Turing thesis, a perfect turing machine is what is required. Its a hard call but we will get there one day.

Of some of the metrics that I found interesting apart from interestingness , entry points, exits , pageviews , timespent etc I liked the bounce rate very much. Yes bounce rate is the most amazing metric when it comes to the web.

  In a nutshell bounce rate measures the percentage of people who come to your website and leave “instantly”. – Avinash Kaushik

Sites with good content have very low bounce rates, that coz users stay on the site for longer than just their immediate need. This metric is extremely important to consider as it could lead to some startling discoveries. Bounce rates directly translate to your site’s usability and design. A complicated design often confuses a user and he or she tends to leave a site if the site seems too cluttered ( which is why I hate GoDaddy’s design ). A blank design with no navigational links is also bad. Once the user finishes reading what he/she came to read, you need to coax them to stay longer. Either by giving them sneak peaks of other related articles or any abstract writeup, tags, internal links and so on. For bloggers, please do read this article to make a list of things to lookout for when picking your blogs design.

I am no genius in the mechanics of running an organization, but I know technology and I do stay in touch with the blogosphere. This company has been one of my all time favorites, in fact I wouldn’t mind joining it even though I know they going through a rough patch. Its not just this company that is struggling, I can list at least half a dozen companies , not necessarily  web, struggling to move ahead of their current position.

You will undoubtedly blame the strategists of the company and crib about them not doing their job. But its not just strategists that must be worried , the management is also equally to blame. Any company that doesn’t include innovation and research in its portfolio is going down. The fad nowadays has been of making hay while sunshines and trying to do too many things at one time. The case isn’t any different for service companies in India. These publicly owned companies have to satisfy their shareholders, clients and employees and do all this keeping in mind the future and their vision. Its a tough call, seriously but not one that cannot be achieved. The dynamics of the IT industry change faster than fastest changing fads. Its essential to keep your head above water to see what lies ahead of you, not just keep swimming as fast as you can. The result can be  disastrous for companies that don’t consider adding Innovation to one of their to do lists.

Innovation is not easily achieved, there have to be strides made to achieve them. Google’s famous 20% personal time is one such great initiative that encourages an employee to innovate and also reap the benefits of doing so. Fostering and more importantly nurturing innovation is very important, you never know, you may have just passed the next million dollar idea. How do you innovate ? With all your company activities intact, give employees the freedom to express themselves. Have code jams and hack days, encourage fresh new ideas and initiatives. Setting up an innovation department in your company and helping talented employees understand their complete potential could be a good start. Have regular workshops on latest technologies and keep yourself abreast of the current trends.  Channelize ideas into a fruitful outcomes and promote such people. Innovation knows no bounds, so if you think you have come across an idea that is too far fetched, hold on it. You never know, 20 years from now you may be a forerunner in something. Document and categorize ideas, invest in a knowledge management system.

phew !!

The Creative Commons is a non-profit organization that encourages people to share their creativity and make it available for others to legally build upon and share.

I found this great Podcast from Indian Podguru Kiruba Shankar ( podcasts from the realm of technology to business and entrepreneurial experiences) about Creative Commons India initiative. Speaking to Lawrence Liang, the head of the CCI initiative, Kiruba dwells into the specifics of CC and other topics. Lend me your ear

Click here to listen to the Podcast.

Its not so bad in the real world. There are techniques being thought of to reduce the memory footprint of these mamoth js bootstraps. First step would definitely be to use the right library , if you use dojo make sure you use the right flavor of dojo like dojo for ajax or dojo for UI etc. There is no point in having functions bloating if they arent used at all. Next step would be use some brains and filter out unwanted functions in your bootstrap files. Its not that hard I guess, get your self Venkman or the recent firebug and you can be on and reducing in no time.

Now for the real deal:

1. use Dojo Shrink safe to reduce, compress and pack all your js files together. Most of the times your file reduces by almost a third using this. This utility doesnt obfuscate so you can call the same methods and not worry about changing function names.

2. Use Dean Edwards packer to further reduce the size by removing linebreaks and other unwanted characters. This usually reduces the file by another 20%

But make sure you always have a backup of the files. As most of these things are unreversible. These are just performance optimizers, they dont help your programming.

In the worst case, if you still a Kb or two short of your SLA then go ahead and obfuscate your code. Packer does a little bit but there are some really neat ones out there that can do a wonderful job. whats the advantage you ask ? a function OnWindowFocusAndDoubleClick() will get replaced by say g() , now thats improvement.

I found this utility which does reduction, try it out :  jsjuicer 

I did some research on S3 sometime back when I intended to use it for some project, and found it really compelling to use. There were lot of problems reported by users and some outages also which hampered lot of sites who depended completely on S3. There is also the problem that your traditional taxonomic structure that you were working on will now be replaced by a Hashtable like datastore that you essentially query. I found it a great way to store shared media, photos and other non text based information and rightly so. S3 fits in as a replacement to a database in the more traditional sense, but it has its own limitations, but its power nevertheless.
So what is this IAAS( see title) , according to me its the death of the network engineer.Gone are those days when you would rent our servers and rope in DBA’s and Network engg who would manage the hardware and availability part of your application. You want unlimited computational scalability? try mediaTemple’s Grid server, want unlimited datastore? try S3, want scalable content delivery? try Akamai Edge. Infrastructure management is at an all time low in the 2.0 era, especially since technology is more easily available , manageable and scalable. Infrastructure is also turning into another SOA where pre built components define how you develop and deploy your applications and you don’t have to worry about downtimes, reliability, scalability and other metrics that are considered important for any web service.
Whenever such disparate elements contribute to a unified cause the next step is  consolidation. But will it ever happen ? these service providers bank on the fact that you need these systems to run your application, otherwise you wouldn’t need a database or a blade server. Have services like S3 given companies like oracle a run for their money? Will there be a similar offering from the other majors?
Picture Oracle providing database hosting on their enterprise level infrastructure for a minimal cost. Updates and upgrades done automatically, maintenance not a pain anymore. Its a space worth debating on and worth watching out for.

  2.  Use your JS wisely
         I have seen a lot of apps currently using a wealth of js libraries, and since your js needs url to access, the XMLHttpRequest object with its url is wide open for the entire world to see. You dont even need to look at the source , with a sniffer like firebug, you can see all the async requests made, with the url and the response. Scripting attacks are the most prevalent of the security attacks and people can do wonders with malicious scripts. The trick here would be to confuse the hackers. Obfuscate the scripts wherever you can, there are good JS obfuscating libraries available. Be advised , obfuscation is usually irreversible , so use it wisely. Use POST instead of GET for async requests and make sure its the right person requesting the data. You could probably generate a sessionid of sorts on the server end and pass it to the browsers as a key to use for any further requests. And please, don’t rely on cookies, they can only go this far.

  3. Follow the MVC methodology
      it usually helps in abstracting the finer points of the application. Separating out the data, logic and accessibility will usually give you more control on what goes where and a lot of server side validation.

  4. SQL Injection
   I understood the complexity behind these attacks only recently. A seemingly simple line of DB code could prove disastrous for applications. Always use good wrappers around your db code. Many good SQL Helper libraries help you against such attacks and the overhead is adding another server side library, which is not so big considering it can save your application.

 There is also the new generation of security threats thanks to Ajax and is open ways. But there are safer alternatives being developed like JSONRequest which abstracts the JSON based calls  and provides a safe way of making sync calls.

  I shall write more about web based security when I get time.

So, whats all this buzz about? Its about making web applications work even when you are not connected to the net -think of it as pulling an outlook in web applications. This area has seen a lot of action recently with the launch of many products and technologies that help achieve the cause. Om Malik has called it the age of the hybrid apps.

The biggest problem, of course is the availability of ubiquitous broadband, without which even the smartest web application is as bright as Paris Hilton -Om

There have been a flurry of launches int his space starting with the Adobe’s hope for the future, Apollo, which was released in its early alpha stages to users, downloadable here. The other releases like Yahoo Widgets 4 which offered desktop widgets which were available offline, and Joyent’s slingshot which allows offline capabilities on rails applications. Some of the early products that gave offline capabilities include Zimbra mail and now released desktop, Zoho office suite and some other productivity apps that I cannot recall. We even have Dekoh a sub continental player in this space that allows for developers to build collaborative webtop based apps on their platform. The newly released Live mail service could definitely use an offline offering to maintain its monopoly in the market.

Firefox 3 is also adding a storage element  , sort of like a datastore for applications to access and store data to be sync’d later with the server. Apollo solves this problem by allowing filesystem access which is a rare commodity in web technologies.

One project I followed closely throughout its development was the Dojo offline toolkit that has been released now with some minor bugs. If you are a developer interested in building offline apps, then I suggest you use Dojo as there is none simpler. With the advent of more widget based applications, webtops and other concepts of the future,we will be seeing a lot of activity in this space.

1 . anticipate what the users expect and provide those features.
2. Autonomy : giving the users a sense of mastery. and providing status messages.
3. don’t rely only on colors , many are color blind
4. consistent commands and shortcuts. don’t change usage patterns and standard shortcuts

In house consistency and maintaining the template across your application
common icons
use inconsistency when necessary, like when you need more marketing attention. change the login space to somewhere else
5. worry about users productivity , not yours, give easily manageable patterns.
6. clear help messages. use the keywords first and emphasize
7. explorable interfaces with colorful icons and images.
give users stable perpetual cues for the sense of home
make actions reversible, example gmails messages.
8. Fitt’s law : the time to acquire a target is a function of the distance to and the size of the target.
corners of the screen are the most easily accessed.
large objects, images for important actions.
9. push latency to the background and reduce latency
visual feedback – for background activities, typically for ajax writebacks and DOM changes.
Indicate the potential length of the wait, if you can .
8. metaphors.
9 . imporve readability y using contrasting fonts and colors.
8. track the state of users behavior.
10. give visual navigation wherever possible.

this was a live blog. my first live blog.

I am spending this weekend at Barcamp Bangalore, an unconference, a gathering of people who share and discuss technology. I myself gave a talk on the evolution of social networks. You can find the the presentation at my home page or just click here.

The next couple of articles will basically center around things I learnt at barcamp3 . There are a lot of topics that are being discussed ( click here to view the list of topics ), Im unhappy i couldnt attend them all, but whatever i could attend ,I shall cover.

• The magnetic HDD is 50 years old (#). In 1956 IBM introduced 305 RAMAC (random access method of accounting and control), which is like the great-great-great grandfather of today’s disks. It was the size of a refrigerator, and stored a total of 4.4 megabytes on 50 doubled-sided, two-foot-diameter disks. The disk had a density of 2,000 bits of data per square inch and had a purchase price of $10,000,000 per Gbyte. (#)
• Today’s laptop drives are typically 2.5 inches and are a size of a deck of cards, and can store upto 160 gigabytes – or 131 billion bits per square inch. Price is less than $1 per gigabyte. (#)
• Consumers bought 739.7 million gigabytes of hard-drive storage space last year. That is 11 times what they bought in 2003. (NYT)
• In the U.S. alone, $600 million worth of external hard drives were sold in 2006, up 53% from 2005, The NPD Group, a market research firm, says. (NYT)
• External hard drive prices declined 28.4% from $197 in 2003 to $141 in 2006 and the amount of storage space on the drives doubled.(NYT)
• Per Gigabyte retail price of hard disk drive storage in 2003 was $2.04, but in 2006 it was 77 cents, according to The NPD Group.
• The recording density for data — aka capacity — has increased 60,000,000-fold in 50 years. (#)
• The amount of worldwide information is projected to grow from 161 exabytes in 2006 to 988 exabytes in 2010. An Exabyte is a million terabytes.(WWD)
• By 2010, the total amount of data will overwhelm the total amount of digital storage by a factor of nearly 2 to 1. 2007 is the year that our ability to stuff bits into the digital universe will outstrip our ability to store them.(WWD)
• Research shows that in large-scale IT installations, the annual disk replacement rates typically exceed 1%, with 2-4% common and up to 13% observed on some systems. (#)

courtesy: GigaOM

OpenId is an initiative to decentralize the authentication mechanism to any website. Its easier said than done, the single sign on has been a goal of almost all the big majors, the GoogleId, YahooId and LiveId( .Net Signon) have been essential to many web workers. The difference here is that, all authentication to a Google Service gets routed through the Google Auth server. Problem – its a centralised mechanism, which doesn’t scale and doesn’t work for other service providers.

In OpenId, users identify themselves using a URI or an XRI , these URI’s are provided by websites who register themselves as OpenId providers or i-Brokers. When a user visits a site that supports OpenId, he just has to provide his OpenId URI and the site requests the URI provider to authenticate the user. Its saves the trouble of multiple signons and accounts. Since this method is decentralised, people can use any or all ids to authenticate themselves.

A lot of websites have already embraced OpenId, some of them include LiveJournal, Zooomr, Wikitravel, ma.gnolia.com, claimid.com, and Jyte. Most recent being wikipedia. OpenId authentication is by default present in Firefox3 and even Microsoft is working on OpenId2.0 for windows vista.

 update : Digg founder Kevin Rose has announced at FOWA that digg will support OpenId . Microsoft and AOL have also announced the embracing of OpenId . 

It figures perfectly, people need to spend more time on the internet and mail, social netowrks and blogs just dont do it for some of them. Games would be a fun way to look at things, but thats no fun if there are no real winners or money involved. So ? almost everybody likes sports, my entire office is biting dust when India is playing cricket. Combine sports perdiction with fake(or real) money , add a dash of social networking to it, offer up to the minute stats on the game itself and you have a perfect idea for a site that will make millions. Sports fans will go gaga over such a concpet, add to this the unsurmountable ” I said so” ego to the equation and you will have gangs of friends pooling in to write about games and their early predictions. Whats better is they will come back later to review their victory and prove their worth. Add incentives like currency, more authority etc and people wil be flocking like there is no tomorrow.

Lot of betting based sites like gottabet, pickspal and pickspop have already turned up and cashing on people power. There is also Xuqa that allows people to play and earn more peanuts which is akin to this. I see this concept really growing wings the next year or so? what do you think?

You buy the mini, give it power and also a LAN connection and voila, you have a dedicated server which crawls the entire network in your enterprise and indexes information. A built in server powers a web based Google Enterprise Search site and you can google your documents like its in your own machine. The concept is definitely astounding as are the possibilities. No need of manually maintaining documents or the pains of version control or shared area space, a dedicated server does all this and more. IF you are an enterprise that wants to invest in a search solution for yoru network, look no beyond the Google Mini, its a cinch.

If you call yourself a 2.0 proponent, you just have to read this. Read the article here.

In light of some of the controversies over websites, companies are spending billions of dollars to defend their own against people accusing them of  crimes they didnt commit. Take for example the violation of copyright because of some background music played in a home video on youtube. Or the lawsuit against MySpace for letting predators in( what do you expect them to do, ask on signup, are you a predator ? ) . Such lawsuits are pointless and should be disregarded at the very instant. Unlike before, site creators are not the Information Publishers. If people are offended, they must seek the person who put up that information. YouTube doesnt need to apologise because some person posted defaming videos of the Mahatma. But country laws are tricky cards to play and you can be assured no lawyer will leave a case which promises him a buck or two ( lawyers please dont take offense ) .

To work towards standardising and rewriting governing laws on the web, Biggies like Yahoo , Google , Microsoft and Vodafone are starting a dialogue together to challenge the laws governing businesses running internationally. In association with Business for Social responsibilty  and advice from the Berkman Center for Internet & Society at Harvard Law School

quote : The process — which aims to produce a set of principles guiding company behavior when faced with laws, regulations and policies that interfere with the achievement of human rights — marks a new phase in efforts that these groups began in 2006.

At the same time, the Center for Democracy and Technology (CDT) was also convening technology leaders, investors and human rights advocates to discuss how to advance civil liberties on the Internet in the face of laws that run contrary to international standards for human rights.

I wish them the best of luck and hope to see a unified set of rules for all countries alike.

The first graphic i got from Mr Sadagopans blog.

Decline in demand for BI/ DW. Great increase in the SOA sector-  This field will keep blooming till everything becomes an SOA,  god only knows what will follow. Upgrades and migrations also take a piece, lot of emphasis on security and Application Integration. Networkign also saw a big plus. Services companies are going to have a field day rethinking strategies and advances.

The second graphic is from the Gods themselves, McKinskey. They plot out components that CIO’s will try incorporating this year.

Again we have similar trends ERP( aka SOA’s), IT Infrastructure and security et al took precedence over most of the other sectors. Data Intergration, BI and mobile solutions are the key forerunners.

The comet paradigm is of course higly experimental and not much is known about how much it can scale or how available is the system, but those attributes will come in time and we will definetely have a new paradigm to look forward to.

Check out this demo of a stock price application using push technology. Only the updated data is fetched and also charted, this demo uses the Dojo chatring and grid facility. The system looks almost real time.

There is also a pdf by lightstreamer technologies which is a push framework/ engine to be deployed on top of existing Http servers to achieve push paradigm. Click here to view the Web application paradigms Pdf

Content Delivery network is a platform for delivery of whatever content you wish to send or share. Say you are a company which does p2p work, you have the application, but you dont have the neccesary infrastructure to make it out in the mainstream. Even if you manage to rent a server and start off, scalability becomes a big issue, in comes CDN’s to save the day. For certain sum of money the CDN takes care of the  delivery, scalability, reliability needs of your application. All you have to do is use the underlying architecture and do some changes to the way your content is delivered.

Akamai does delivery of your streaming audio/video, downloads, FTP and ofcourse http. Akamai does this using their Edge Platform, apparently one of the largest distributed network in the with almost 20,000 servers in over 71 countries. The content is mirrored across these servers and when a user requests for content, based on certian algorithmic computation the content is delivered from the most nearest non clogged server. It even takes care of load distribution and clog times. Apart from all this, they even provide analytics for the delivered content.

The future is really bright for these applications and people really don’t want to worry about hardware nitty gritty’s and concentrate more on the application. Grid hosting, S3 and now Akamai are just stepping stones to achieve an extremely scalable, reliable and a fast application.