Next Leap in Mobile Interaction!

Author: Raj Tumuluri
CEO, Openstream
The keyboard and mouse as input devices have revolutionized the computer interaction for decades now. Perhaps, nothing could make me realize the profound effect these input methods have on the new generation, until I heard a friend's daughter ask matter-of-factly, "Why do we have five fingers, when the mouse has only 2-buttons?"

With more than three billion mobile phones in the world today, analysts predict that the number of mobile devices accessing the Internet will cross the one billion mark over the next four years.

With the wealth of information and services available from almost everywhere, Internet-connected mobile devices are reshaping the way we go about our personal and professional lives. According to John Gantz, Chief Research Officer at IDC, "With an explosion in applications for mobile devices underway, the next several years will witness another sea change in the way users interact with the Internet and further blur the lines between personal and professional."

During the early years of mobile Internet, the industry focused primarily on trying to fit the existing web sites on to a small mobile screen, through a process called transcoding (translation + encoding). This process essentially involved the translating of the existing HTML pages of Web-sites into the Wireless Markup Language (WML) pages, with some image re-sizing capability. But, the market quickly realized this did not really help adoption of mobile internet, due to lack of the convenient interface to interact with the Internet applications using the phone keypad. In those days, users had to press each key up to three times to select the desired alphabet to input.

Then came the Research In Motion’s (RIM) Blackberry as a mobile email appliance with a QWERTY keypad that quickly became popular with users as they could read and reply to the emails without having to carry bulky laptop computers for email access outside their homes and offices. While many loyal users of Blackberry swear by the convenience of a track-wheel (mouse) and keypad, the primary use of that interface could not help increase the use of mobile internet access beyond email.

Clearly, the keyboard and mouse outlived their purpose. After trying, foldable key-boards and accessories, the mobile phone industry has until recently settled on sliding keypads for user interaction.

The launch of Apple’s iPhone with its innovative touch interface and on-screen keypad, revolutionized the mobile Internet access, taking it beyond simple email access. With over 100,000 applications, it quickly became more than just a phone, but an information access device, a GPS and a mobile TV and a gaming console and a social networking tool.

Given the addictive nature of the convenience of information access, anytime, anywhere, several research efforts around the world have been focused on new and innovative interfaces that are suitable for interaction while on the move, including projection keyboards, speech recognition, multi-touch and projection screens.

A recent research project at MIT has popularized the power of combining the mobile phone camera with a mobile projector, allowing users to project the output of the mobile screen to any surface making it touch sensitive for user input.

Research has shown that combining the speech and gesture (touch and tap) reduces the amount of time we take to communicate our intent be it with other humans or with systems. It is less ambiguous and takes less time to say, "I want that (gesturing at the item you want)" than speaking item name and location when you are in a shop. The geek-speak for such "natural interaction" is multimodality.

Imagine the convenience of listening to your emails, text messages or personalized news stories, while driving to work or when stuck in the traffic, without having to type or take your eyes off the road. Or, sharing the picture with your family of that dress you wanted to buy for your daughter, with quick voice notes and annotations all from the convenience of your mobile internet device.

The power of voice search on the internet has been demonstrated by Google, Vlingo among others. While combining speech and gesture interfaces for interaction with mobile internet devices makes it very easy for the users to interact with applications and peripherals, it can be complex to implement such interfaces commercially, that only a handful of companies such as Microsoft, AT&T, Nuance, Openstream, IBM, Kirusa and some others provide such solutions.

The WorldWideWeb Consortium, the apex body that develops the web standards, has several research groups that are focused on mobile and voice web interaction. The W3C Multimodal Interaction ( MMI ) Working Group, focuses on the development of standards for combining speech, touch (Ink), text and emotion into user interface ( imagine if your phone could detect when you are tired or angry and adjust its presentation/intonation of the output accordingly!).

The W3C MMI Architecture provides application developers the ability to combine various modalities of interaction through an Interaction Manager (IM) using asynchronous events among various constituents of the application, separating the application logic from user interface.

For example, an application, can have Speech Interface (VoiceModality) developed in VoiceXML markup and gesture-annotations in InkML and the Visual application in HTML, all exchanging the user-input in EMMA ( extensible multimodal annotation) markup with the business-logic layer of the application. The markup-based development of interfaces, allows the developer community to fully leverage the benefits of portability and extensibility and inter-operability of the web application paradigm on multiple mobile platform and devices, as opposed to native development of such interfaces on each of those devices and platforms.

With mobile phonespacking more functionality, such as GPS, Camera, Accelerometers, Multimedia Players, RFID and other sensory capabilities, the applications on these mobile devices will get richer in features than the traditional web applications designed for desk-top computers.

The appeal and applicability of multimodal interfaces is not just for consumer/personal applications. The use and convenience of such multimodal applications in the insurance, healthcare, realty, media and entertainment industries can be readily seen, as several enterprises have already started incorporating the rich features of multimodality in to their field data collection applications using voice form-fill, image & signature capture, map-integration with visual annotations and spoken driving directions using mobile devices.

In the coming months, mobile applications will become increasingly multimodal allowing users the choice of modes of interaction based on their situational needs with intelligent detection of the ambient conditions.

Previous  article
Next article
Write your comment now

Email    Password: 
Don't have SiliconIndia account? Sign up    Forgot your password? Reset
Reader's comments(2)
1: From: Mrs. Mary David

This mail may be a surprise to you because you did not give me the permission to do so and neither do you know me but before I tell you about myself I want you to please forgive me for sending this mail without your permission. I am writing this letter in confidence believing that if it is the will of God for you to help me and my family, God almighty will bless and reward you abundantly. I need an honest and trust worthy person like you to entrust this huge transfer project unto.

My name is Mrs. Mary David, The Branch Manager of a Financial Institution. I am a Ghanaian married with 3 kids. I am writing to solicit your assistance in the transfer of US$7,500,000.00 Dollars. This fund is the excess of what my branch in which I am the manager made as profit last year (i.e. 2010 financial year). I have already submitted an annual report for that year to my head office in Accra-Ghana as I have watched with keen interest as they will never know of this excess. I have since, placed this amount of US$7,500,000.00 Dollars on an Escrow Coded account without a beneficiary (Anonymous) to avoid trace.

As an officer of the bank, I cannot be directly connected to this money thus I am impelled to request for your assistance to receive this money into your bank account on my behalf. I agree that 40% of this money will be for you as a foreign partner, in respect to the provision of a foreign account, and 60% would be for me. I do need to stress that there are practically no risk involved in this. It's going to be a bank-to-bank transfer. All I need from you is to stand as the original depositor of this fund so that the fund can be transferred to your account.

If you accept this offer, I will appreciate your timely response to me. This is why and only reason why I contacted you, I am willing to go into partnership investment with you owing to your wealth of experience, So please if you are interested to assist on this venture kindly contact me back for a brief discussion on how to proceed.

All correspondence must be via my private E-mail ( for obvious security reasons.

Best regards,
Mrs. Mary David.
Posted by: mary lovely david - Monday 26th, September 2011
2: Hi my dear,
My name is Mounace, i would like to establish a true relationship with you in one love. please send email to me at ( i will reply to you with my picture and tell you more about myself. thanks and remain blessed for me,
Your new friend Mounace
Posted by: mounace love love - Thursday 09th, June 2011
More articles
by Kaushal Mehta - Founder & CEO, Motif Inc..
The retail industry is witnessing an increased migration of customers from traditional brick and mortar retail to E-commerce (online retail)...more>>
by Samir Shah - CEO, Zephyr .
You probably do because you are on the phone with them! For all of you working in some technical management capacity here in Silicon Valley,...more>>
by Raj Karamchedu - Chief Operating Officer, Legend Silicon .
These days are a mixed bag for me. Of late I have been considering "doing something bigger and better," in my life, perhaps seriously though...more>>
by Madhavi Vuppalapati - CEO of Prithvi Information Solutions .
IT Services Rise of Tier II companies The Indian IT outsourcing industry is going through very exciting phase in its business life...more>>
by Bhaskar Bakthavatsalu- Country Manager, India and SAARC of Check Point Software Technologies.
Data loss occurs every day through corporate email. In fact, given the sheer number of emails an organization sends every day, data loss inc...more>>