LOFAR self-calibration using a blackboard software architecture

One of the major challenges for the self-calibration of the new generation of radio telescopes is to handle the sheer amount of observational data. For LOFAR, an average observation consists of several tens of terabytes of data. Fortunately, many operations can be done in parallel on only part of the data. So, one way to take up this challenge is to employ a large cluster of computers and to distribute both data and computing power. This paper focuses on the architectural design of the LOFAR self-calibration system, which is loosely based on the Blackboard architectural pattern. Key design consideration was to provide maximum scalability by complete separation of the global controller-issuing sequences of commands-on the one side; and the local controllers-controlling the so-called 'kernels' that execute the commands-on the other side. In between, resides a database system that acts as a shared memory for the global and local controllers by storing the commands and the results.