Data Management


As any other large imaging camera, PAUCam produces a large amount of data that needs specific data-reduction software. Each frame will consist of about 300 MB of information, which translates into a maximum total data volume of 200 GB for a typical observing night. In response to those needs we are developing a new Data Management System (software and hardware), called PAUdm, to process the PAUCam images taken at the WHT and store them in a format suitable for further analysis.



Data Access at PIC


For the PAU Survey the data will be transferred between the WHT facilities on site at La Palma and the Scientific Information Port (PIC) in Barcelona.

The PIC is a large facility located at the Campus of the Universitat Autònoma de Barcelona (UAB), one of the 11 “Tier-1” centers around the globe used for the processing and distribution of the data taken at the Large Hadron Collider (LHC) at CERN.

Homehome-PAU.htmlshapeimage_1_link_0
About Usabout_us.htmlshapeimage_2_link_0
Surveysurvey.htmlshapeimage_3_link_0
Cameracamera.htmlshapeimage_4_link_0
Data Managementshapeimage_5_link_0
Educationeducation.htmlshapeimage_6_link_0
Outreachoutreach.htmlshapeimage_7_link_0
Theorytheory.htmltheory.htmlshapeimage_8_link_0

One of the first stages of data processing is the Community Nightly Processing Pipeline, which will be also available to users of the camera external to PAU. The output of this pipeline are reduced images as well as cosmetic information for every single pixel. The system allows parallel processing at the Image Reduction Pipeline.

The task of transferring enormous amounts of data to the PIC has been already tested in the context of MICE Project.


MICE is a large Dark Matter simulation project that is taking place at the Marenostrum facility of the Barcelona Supercomputer Center (BSC). Since 2009 the data simulated at the Marenostrum computer is transferred to PIC by means of an automated and reliable protocol implemented in the grid infrastructure.


The MICE simulations are used in the PAU Simulation package, PAUSim, which is also an integral part of PAUdm.


[Give examples of data volumes transferred in 2009].