Williams Owen

Abstract Reference: 31181
Identifier: P4.18
Presentation: Poster presentation
Key Theme: 4 Long-term Management of Data Archives

The Role of the Euclid Archive System in the Processing of Euclid and External Data

Authors:
Williams Owen R., Begeman Kor, Belikov Andrey N., Boxhoorn Danny , Droege Bob, McFarland John P., Edwin Valentijn A., Vriend Willem-Jan, Christophe Dabin

The Euclid Archive System (EAS) is a core element of the Science Ground Segment (SGS) of Euclid. Euclid is an ESA M2 mission which will create a 15,000 square degrees space-based survey. Two subsystems of the EAS, the Data Processing System (DPS) and Distributed Storage System (DSS), provide the metadata and data storage for Euclid data processing. These subsystems implement the Euclid Common Data Model and provide numerous services for Euclid Consortium users and SGS subsystems. In addition the EAS-DPS assists in the preparation of the Euclid data releases which are copied to the third EAS subsystem, the ESA-developed and managed Science Archive System (SAS) where they become available for the wider astronomical community. The EAS follows a data-centric approach to data processing where the EAS-DPS is responsible for the centralized metadata storage and the EAS-DSS supports the distributed storage of data files. The EAS-DPS implements the object-oriented Euclid Common Data Model using a relational DBMS for the storage. The EAS-DPS supports the tracing of the lineage of any data item in the system, provides services for the data quality assessment and the data processing orchestration. The EAS-DPS services are built on the forward and backward conversion of Python objects to tables of relational database and extensively use the extreme data lineage of metadata objects which is stored in EAS-DPS. The EAS-DSS is a distributed storage system which is based on a set of storage nodes located in each of the nine Science Data Centers of the Euclid SGS. The storage nodes supports a wide range of solutions from local disk using a unix filesystem to iRODS nodes or Grid storage elements. In this paper the architectural design of EAS-DPS and EAS-DSS are reviewed: the interaction between them and tests of the already implemented components are described.