Data explosion
The amount of information produced in the world is increasing manifold, by almost 30-40 percent every year. Trends imply technology evolution is leading to information explosion from kilobytes to petabytes. As the amount of available data grows, the challenges of managing the information become more difficult and this can lead to information overload or information fatigue.

Take for example the telecom service industry in India. During 2005-2006, on an average, 5.5 million cellular lines were added every month, taking the mobile subscriber base to 157 million from 91 million during 2004-2005. (Source: Annual survey conducted by Voice & Data).

In another case, a large wireless service provider has a subscriber base of nearly 5 million subscribers. Imagine, on an average, every subscriber makes 20 calls per day. Each call generates a call detail record (CDR), which is used for calculating the bill and subsequent collection process, while each CDR implies a revenue opportunity for the telecom service provider. Thus, around 100 million raw CDRs can be generated per day and they get transformed before reaching the billing system. All this data is further stored for 3 months online and subsequently archived. Moreover, the same data (refined and transformed) is used for various other telecom applications like billing, customer care, and business intelligence and this translates the total information to terabytes over a period.

This condition results from having a rapid rate of increase in the amount of information generated, even while days remain 24 hours long and our brains remain in roughly the same state of development as they were when cavemen communicated by scrawling messages on stone. IT managers, therefore, need to develop skills for managing this information overload.

Managing data smartly

Managing such huge amounts of data can be a tough challenge. It involves securing, managing, and archiving that data in such a way that concerned executives have access to it and use it for making more informed decisions. This also increases the management and storage acquisition costs. Apart from achieving operational goals from the data, new government regulations and guidelines are a key driving force in how and why data is being retained. As a result organizations are forced to retain and control all their confidential and public information in storage for very long periods of time.

An analysis of data will most likely reveal that 75 percent of data is never accessed later while barely 25 percent is accessed and maybe updated on a frequent basis. As the age of the data increases, its access frequency, if any, diminishes to almost zero. Thus, data can be described as being active, less active, historical, or ready to be archived. With so much data being held, that will be moved during its lifetime to several different physical locations. Additionally, the business user’s requirement of information on demand forces the information’s being made available online.

The Solution

In a traditional approach, the data is stored in large storage boxes and once the information is used, it is stored in secondary storage media like tapes. With flat file systems, this sounds simple. In a relational database management systems environment, it requires downtime for both archiving and retrieval. Users have to wait for information to be restored. So, managing the information is one of the key requirements in modern day IT enabled businesses.

Information Life Cycle Management (ILM), the new building block in the IT management strategy, is one of the solutions for managing information overload. ILM is a combination of data management capabilities of the database server with appropriate storage technology. Oracle Database enhances support for information lifecycle management, enabling organizations to ensure regulatory compliance, optimize storage, streamline business processes, and reduce cost.

India landscape

Organizations that manage very large volumes of data require self-managing databases. They need to carry out database upgradation with minimal disruption and have to deal with a large number of security and compliance issues. Financial services, communications, and government organizations in India stand to gain greatly from using a database with an ILM platform.

Industry analysts claim that the worldwide relational database management systems market is poised for continued growth through 2010 as data management and integration become strategically important in an increasing number of organizations.

The market will witness healthy growth, reflecting ongoing demand for relational database management systems, including those for online transaction processing, which have traditionally been the drivers of growth in the past. It will also experience increased demand from organizations opting for business intelligence and data warehousing technologies to manage their data effectively.

The author is Director – Database Sales Consulting, Oracle India