Grange Yan

Abstract Reference: 31171
Identifier: B1
Presentation: BoF
Key Theme: 8 Other

Operations in the era of large distributed telescopes

Authors:
Grange Yan, Vinsen Kevin, Guzman Juan, Parra José, Mol Jan, Magnus Lindsay

The previous generation of astronomical instruments tended to consist of single receivers in the focal point of one or more physical reflectors. Because of this, most astronomical data sets were small enough that the raw data could easily be downloaded and processed on a single machine.

In the last decade, several large, complex Radio Astronomy instruments have been built and the SKA is currently being designed. Many of these instruments have been designed by international teams, and, in the case of LOFAR span an area larger than a single country. Such systems are ICT telescopes and consist mainly of complex software. This causes the main operational issues to be related to the ICT systems and not the telescope hardware. Managing the operations of such telescopes therefore requires an approach that significantly differs from classical telescope operations.

The goal of this session is to bring together members of operational teams responsible for such large-scale ICT telescopes. This gathering will be used to exchange experiences and knowledge between those teams. Also, we consider such a meeting as very valuable input for future instrumentation, especially the SKA and its regional centres.

Abstract Reference: 30841
Identifier: P8.14
Presentation: Poster presentation
Key Theme: 8 Other

Characterising radio telescope software with the Workload Characterisation Framework

Authors:
Grange Yan, Lakhoo Rahim, Petschow Matthias, Wu Chen, Veenboer Bram, Emsley Iain, Dijkema Tammo Jan, Mechev Alex, Mariani Giovanni

Modern low-frequency radio interferometers, such as LOFAR, MWA and in the future the SKA, are comprised of commodity hardware. The core of such instruments is the software used to correlate and process the data — reflected by the community term “software telescope”. Such software telescopes are modular and under constant development by large interdisciplinary research collaborations, working across continents; consequently, obtaining a holistic view of the entire system becomes a challenging task.

Managing a software telescope requires insight into the performance at different levels, from the underlying compute hardware to the software processing pipelines. Profiling of pipelines is used to provide in-depth understanding of software (in)efficiency; however, profiling does not provide a broader understanding of the interaction between the system (i.e. the hardware and operating system) and the software (i.e. the pipelines and their underlying algorithms), nor does it provide insight about the energy consumption. To study the behaviour of these interactions and the energy consumed, the workload needs to be characterised. Workload characterisation is a useful input to R&D (e.g. focusing development efforts, aiding algorithm verification and observing bottlenecks), telescope operations (e.g. scheduling of observations, the mapping resources to software requirements and resource monitoring) and hardware management (e.g. capacity planning and hardware acquisition). To ensure that the characterisation obtained by different developers and/or operational telescopes are comparable and reproducible, the standardisation for gathering metrics and storing results is essential.

We present a modular framework, the Workload Characterisation Framework (WCF), that is developed to obtain, store and compare key characteristics of radio telescope software; at the same time, the framework provides resource information useful for optimising scheduling and mapping resources. The WCF is being developed as part of the SKA Science Data Processor (SDP) design. It is available as open source software and can be used for other research software stacks; radio telescopes being only the first use case.

We discuss the design considerations and show how the WCF is used to gain insights into the behaviour of software. As a demonstration, we discuss the experiences using the framework to characterise a LOFAR pipeline.