Google to handle telescope data

Nearly a decade ago, Anthony Tyson and some colleagues at Bell Labs were mulling how dark matter and dark energy might be studied, a notion that later led to the idea of designing a huge telescope. Meanwhile, another group at Bell wondered how to manage gargantuan amounts of data. Over the years, members of the two groups worked together on a telescope concept even as many left Bell for other pastures. Today, Tyson, now a physics professor at the University of California, Davis, is director of the Large Synoptic Survey Telescope, and some of his former Bell colleagues are at Google Inc, which recently announced it is joining the project.

The 8.4-meter telescope, slated to see first light in Chile in 2014, will survey the entire visible sky every three nights with its 3-billion-pixel digital camera. A new image will emerge every 15 seconds—adding up to a staggering 30 to 50 terabytes a night. That’s where Google comes in. The company, known principally for creating and operating one of the world’s most popular Internet search engines, expects to apply its data management expertise to processing, organizing, and storing the reams of information the LSST will produce. “There’s no way individuals can look at this data and discover things,” Tyson said.

Don Sweeney, LSST project manager and a physicist at Lawrence Livermore National Laboratory, pointed out that often, privately owned telescopes have proprietary data that are not publicly available. “These [LSST] data will be available to everybody—an astronomer at Harvard or a high-school student in Iowa,” Sweeney said. Google’s involvement also means that all the information generated will be available to the public in a far more organized and ordered fashion than in the past.

The project’s price tag is $380 million in 2006 dollars, and the money is expected to come from private foundations and US government agencies, including NSF and the Department of Energy. As of January, 20 organizations, including universities and other research institutions, had joined LSST. Google is the only private-sector partner.

Google spokeswoman Amanda Angelotti wouldn’t provide specific information on exactly how Google will process and store the LSST data, saying technological details haven’t been worked out yet. But company engineers have already begun wrestling with some data-management issues. Tyson said Google is also working to develop the most efficient way of processing LSST images. “We actually have to detect things that change in real time,” he said.

Rob Pike, a former Bell scientist who is now Google’s lead engineer on the LSST, said one of the project’s challenges will be to continue to manage the constant influx of data in the face of inevitable technological breakdowns.

“[The LSST] will build on Google’s expertise in providing reliable, large-scale distributed computing platforms. The flow of data is large and unrelenting, so computational downtime can cause logistical problems for data that pile up waiting to be processed,” Pike said. “Reliability is important for throughput, and with the quantity of data involved, many machines must process it in parallel. Pieces of the system are sure to fail at some point—computers break, and the more you have, the more likely one will break—so the key is to keep the system running even when some of its machines are down.”