IPP Data Management Infrastructure
The IPP data management infrastructure consists of a few major components.
- Nebulous This components manages the locations of data files on the distributed cluster of storage nodes. The physical location of a data object may be specified or left to Nebulous to choose. Multiple copies of data files may be automatically saved.
- DVO This tool manages detection of astronomical objects in a coordinated fashion. It provides the link between detections and specific astronomical objects. It also provides the infrastructure for astrometric and photometric calibration.
- Metadata Database This database and supporting tools store the processing results from each of the IPP analysis stages. It is also used to record the IPP system state and to track the analysis data products as they progress through the different steps of the analysis.
- pcontrol This tool manages the parallel processing of the analysis tasks. It distributes tasks to the cluster of computing nodes as those nodes are available. Processing tasks may be optionally assigned to a specific machine, or identify a preference for a specific machine.
- pantasks This program uses the current state of the IPP data products, based on the contents of the Metadata Database, and determines what analysis jobs must be performed at any given time. The tasks are either sent to pcontrol or are run in the background as appropriate.