Big Data, Big Challenge

Bill Gattle, President, Harris Space and Intelligence Systems. Credit: Harris Space and Intelligence Systems

Bill Gattle, President, Harris Space and Intelligence Systems. Credit: Harris Space and Intelligence Systems

We’re awash in big data. Never before has it been easier for users to quickly collect imagery, from the big picture grabs offered on Google Earth to the detailed images offered on platforms such as MapMart (full disclosure, this is a Harris platform). For business users who rely on this data, it is both a blessing and a curse — how do you make all of it meaningful? For data providers, how do you deliver value while balancing widespread availability of this data?

There’s been a dynamic flip over time in the space industry from an emphasis on sensors to much more of an emphasis on the information and how it can be used. Sensors available today have revolutionized data collection. For example, Geiger-mode LiDAR, once only a Department of Defense technology, is transforming mapping. Linear-mode LiDAR collection takes a long time and can be cripplingly expensive. In fact, over the last 20 years, the U.S. Geological Survey has mapped only 28 percent of the continental United States. Today, sensors can scan the entire United States in less than a tenth of the time at a tiny fraction of the cost.

The data generated by Geigermode is so accurate, buildings and even power lines show up clearly in any imagery the sensor generates. It can generate over 70 points of data per square meter, which means objects measuring about 30 centimeters across will show up clearly.

The sensor is only one part of the equation — turning all that data into usable information requires analysis (that’s where the curse comes in). Perhaps the most important technology that has emerged alongside powerful sensors is automated processing. New computers and new software tools allow for automatic analysis of the vast quantities of data sensors are capable of generating.

Traditionally, sensor data has been processed inefficiently. However, technology now allows for near-real time on-board analysis. Meanwhile, there has been a quantum leap in ground-based processing capability, which means data can be analyzed automatically. New application software such as machine learning and deep knowledge allows customers to solve unique and challenging problems. Additionally, improved data storage capabilities mean that data can be stocked away in vast libraries of information. That stored data can be delivered to customers at a fraction of what it would cost normally — and allows for collection costs to be shared, thereby further reducing everyone’s expenses.

The challenge with big data doesn’t only pertain to the geospatial industry. The entire field of remote Earth observation, of which geospatial is a critical piece, is struggling with the big data balance.

The National Weather Service is responsible for preparing the nation for severe weather events. GOES-R, NOAA’s newest weather satellite launching in October, will provide more frequent, higher-resolution data for users throughout North, Central and South America and handle more data than what is produced by today’s weather satellites. The ground system is expected to process more than 16 terabytes of data per day. To give you an idea of the data tsunami coming for NOAA, in the same amount of time the current ground system for polar missions processes 10 percent of the Western Hemisphere, the ground station will have processed the entire Western Hemisphere 20 times.

Not every user needs end-to-end services or even the level of detail that many government agencies require — “good enough” data will do and that’s where the industry has seen the rise of the service providers. We are starting to hit the critical mass of sensors; companies that have relied on sensors alone are going to struggle. Automated processing and analysis is where the industry is headed. Are you ready for the challenge?

Bill Gattle is President of Harris Space and Intelligence Systems