The oil and gas industry is experiencing multi-dimensional landscape changes, including economical, production and consumption areas. The impact of these changes starts with the upstream business also referred to as exploration and production (E&P) industry.

The E&P industry is one of the most complex operations among any of the industry verticals. This industry has been at the forefront of massive data collection historically, well before the term Big Data has been in use. The data collected is from instruments, sensors, computer software and hand-written notes for conducting effective and efficient operations.

On a daily basis this data collection can range from a few terabytes to a few petabytes, depending upon the phase of oil well life cycle and the complexity and modernity of the oil field. The overarching goal of the E&P industry is to produce maximum oil at the lowest cost possible with the highest degree of safety for all stakeholders. Another important aspect for E&P industry is that the above goal must be achieved within the ever-increasing costly restrictions imposed by the global, regional and local regulatory authorities. Thus, the E&P industry has to remain innovative and competitive at all times.

Forward looking E&P companies must adopt Big Data, the second industrial revolution to remain competitive and innovative. The industry has a strong foundation for the seven pillars of Big Data namely, Volume, Velocity, Variety, Veracity, Virtual, Value and Variability.

The meta-data associated with tools used in exploration and drilling could be in few megabytes while the metrics it collects daily could range from megabytes to gigabytes to petabytes, covering a wide range of volume through the lifecycle.