Shatskaya Marina

Abstract Reference: 29843
Identifier: P4.16
Presentation: Poster presentation
Key Theme: 4 Long-term Management of Data Archives

Storage and processing of big data volume for Space-VLBI projects (Radioastron and Millimetron Missions)

Authors:
Shatskaya Marina, Abramov Alexei, Fedorov Nikolai, Sergey, Likhachev Seliverstov Sergey, Sichev Dmitry

In this report IT support for two Space-VLBI Projects are considered: currently operating Radioastron mission and future Millimetron mission. Radioastron Data Processing center was implemented to collect, process and archive the data and to organize the informational exchange between all participants of this project. More than 40 ground telescopes are involved in joint observations with Radioastron. Digital data from space and ground telescopes is collected in Data Processing Center. Large interest of scientific community around the world in Radioastron mission led to the growth of number and duration of observations, as well as to the growth of number of ground telescopes participating in these observations and the total amount of data. Currently, the volume of stored data is more than 2 PB. Usually in VLBI projects the initial or raw data is being deleted after successful correlation. In Radioastron mission we have made a decision to save and store all raw observational data, because of its uniqueness in order to have a possibility for future reprocessing and re-correlation. This is one of the reasons to expand our data archive. We have organized the data storage, computer complex, high-speed internal and external networks and archive for transferring, processing and archiving all data of Radioastron mission. All these components work as an integrated system. We have done an optimization of our equipment, by improving our operation scheme and dividing the data flows. In this report we pay special attention to the elimination of “bottle necks” in our data processing complex. Additionally, we pay great attention to control the Radioastron mission data transfer and storage reliability. The structure and functions of the Astro Space Center Data Processing Center fulfill the requirements for the Radioastron Mission data processing and have been successfully confirmed during the whole period of our operation. In our next Space-VLBI project – Millimetron mission, the expected volume of scientific information is estimated to be at least 100 PB. It is much more than for Radioastron mission. In this case for Millimetron mission will be organized a separate data center. We believe that our experience in processing, transferring and storing of large data volumes for Radioastron mission will be very useful.