DPP | Software | Library | Surveys | NSA | Education | LSST | NVO
The NOAO Data Products Program
The NOAO Data Products Program (DPP) is a new program aimed at identifying scientifically interesting datasets from ground-based O/IR telescopes and making them available to the astronomical community together with the tools for exploring them. The program coordinates NOAO projects that are data intensive, including the handling, pipeline processing, analysis, and archiving of data. These datasets, and the facilities for mining them, will form a significant component of the resources of the National Virtual Observatory, and will be an important part of NOAO's participation in that endeavor. In the longer term, this activity will lead to a data management role in the Large-aperture Synoptic Survey Telescope, a facility that will produce roughly a petabyte of imaging data per year.
DPP is currently planning the infrastructure necessary to process all data from NOAO instruments that are capable of generating surveys of great scientific interest, such as the Mosaic camera, FLAMINGOS, and (later) NEWFIRM. Some of the necessary application and system software was developed to support the NOAO Deep-Wide Field survey, and was made available to the community in IRAF release V2.12. Development and operation of these data reduction pipelines will both decrease the cost of future surveys and allow the rapid and efficient flow of new data into the archive. The pipelines (or their component modules) will also be used for "quick-look" and "quick-reduce" processing at remote observing facilities to enable near real-time evaluation of data quality and transient events, which can feed back into decisions about observing priorities, techniques, or instrument configuration.
DPP is also planning an archive facility that will serve as a repository for these datasets. Phase 1 of the archive was released to the public in April 2002; a fully-engineered archive, capable of scaling to the tens of terabytes of data that will be generated each year will be planned in the coming year. Longer-term planning will be focused on developing the technologies for archiving, data discovery and data mining, and the creation of value-added products and services for instruments whose data rates and volumes will exceed that of current ground or space-based instrumentation by a factor of more than 100.
The technology necessary for managing such large quantities of data is an area of active research in the context of NVO. Likewise, the techniques for extracting meaningful science from large, heterogeneous datasets is an area of endeavor for a small but growing number of researchers at institutions throughout the US and abroad. DPP is participating in these efforts, and is benefiting from the experience gained as a participant in the NOAO Surveys programs.