Research Software Engineer - Reading, Berkshire, United Kingdom - Society of Research Software Engineering

Description

We are looking to hire a highly motivated Research Software Engineer (A2)to work on the development of data infrastructure services to support Machine Learning (ML) efforts within the Destination Earth (DestinE) initiative. This role is an integral part of a dynamic team consisting of scientists and software engineers contributing to key activities at ECMWF, including DestinE (this position), Copernicus and core programmes.

At ECMWF, you will find a passionate and diverse community, collectively aiming to bring novel technology and science to the cutting-edge of numerical weather prediction. ECMWF is one of the three organisations entrusted to deliver Destination Earth, a flagship initiative of the European Commission to develop a highly accurate digital model of the Earth on a global scale. In the framework of ECMWF's contribution to the Destination Earth initiative of the European Commission (DestinE), you will work on the Digital Twin Engine (DTE), a set of interoperable software components and services which support end-to-end execution of the Digital Twins.

With the recent breakthrough in in AI-driven weather forecasting, it becomes clear that AI will play a key role in the next generation of forecasting systems. To this end, ECMWF is building a dedicated multi-disciplinary group to tackle these challenges. ECMWF has been the first operational weather centre to publish results of their own global machine learned weather model – the Artificial Intelligence Forecasting System (AIFS). Within DestinE, ECMWF will now develop and deploy workflows of machine-learned Earth-system components of a European foundation model based on existing DestinE traditional simulation and modelling results.

In this role, you will explore and innovate solutions to improve data and related services to support the scientific and technical development of ML models and processing for DestinE. You will develop tools and workflows to assist with assembling and efficiently serving datasets for ML training. You will be expected to bring creative solutions for handling of large datasets across multiple data centres. Given the high resolution of the data being produced, it will be important to efficiently and scalably sub-select and process data for training, and to federate access to these datasets between the data centres, in particular those of EuroHPC where DestinE runs its digital twins.

The role sits in the Data Management Services team, within the Development Section in the Forecasts and Services Department. The primary focus of the team is to create scalable, robust and performant software and services for handling manipulating large volumes of meteorological data. Our software is used heavily in ECMWF's operations and research, through all stages of our workflows from acquisition of observations, through model output on the HPC to archival and dissemination. Special focus is placed on a semantic approach to data handling, such that the technical realities of data handling are abstracted from the scientific software. We expend significant effort to navigate scalability challenges posed by future model upgrades and to exploring new technology and approaches for the future. This supports ECMWF's strategy of producing cutting‐edge science and world-leading weather predictions and monitoring of the Earth system.

In summary, you will be supporting our ML efforts with tailored high-performance, scalable, semantically driven data access.This contract is funded by Destination Earth and will collaborate closely with other activities at Member States and with our partners at ESA and EUMETSAT.

About ECMWF

The European Centre for Medium-Range Weather Forecasts (ECMWF)is a world-leader in weather and environmental forecasting. As an international organisation we serve our members and the wider community with global weather predictions and data that is critical for understanding and solving the climate crisis. We function as a 24/7 research and operational centre with a focus on medium and long-range predictions, holding one of the largest meteorological data archives in the world. The success of our activities builds on the talent of our scientists and experts, strong partnerships with 35 Member and Co-operating States and the international community, some of the most powerful supercomputers in the world, and the use of innovative technologies and machine learning across our operations.ECMWF is a multi-site organisation, with a main office in Reading, UK, a data centre/supercomputer in Bologna, Italy, and a large presence in Bonn, Germany.ECMWF has also developed a strong partnership with the European Union and has been entrusted with the implementation and operation of the Destination Earth Initiative and the Climate Change and Atmosphere Monitoring Services of theCopernicus Programme. Other areas of work include High Performance Computing and the development of digital tools that enable ECMWF to extend provision of data and products covering weather, climate, air quality, fire and flood prediction and monitoring.

See more info about what we do.

The Destination Earth (DestinE) initiative

ECMWF is one of the three entities entrusted to implement the DestinE initiative of the European Commission, alongside with ESA and EUMETSAT as partners. DestinE aims to deploy several highly accurate thematic digital replicas of the Earth, called Digital Twins.The Digital Twins will help monitor and predict environmental change and human impact, in order to develop and test scenarios that would support sustainable development and corresponding European policies for the Green Deal.ECMWF is responsible for the delivery of these digital twins and of the Digital Twin engine, the software infrastructure needed to power them of some of Europe's largest supercomputers, those of the European HPC Joint Undertaking (EuroHPC).

The second phase of DestinE covers the period June 2024 – May 2026, and future phases are foreseen (subject to funding).Phase 2 will focus on early operations with consolidation, maintenance, and continuous evolution of the DestinE system components developed in the first phase. There will also be an enhanced focus on ML activities, including the deployment of workflows of components of a ML model for the Earth system, optimisation of the Digital Twin Engine to enable efficient model training and simulations, and other activities. One key element of the ML activities in phase 2 includes training. This shall build on recent ML training initiatives at ECMWF, including the Massive Open Online Course (MOOC) on ML for Weather and Climate.

(see )

For more information on DestinE, see and

Your responsibilities

• Develop and support tools to handle DestinE meteorological data for ML training.
• Optimise DestinE data delivery workflows for ML training, including from and to non-ECMWF facilities (especially including EuroHPC data centres)
• Explore novel technologies and methodologies to improve ML data generation and access, and integration with ECMWF core and DestinE services.
• Contribute to the ECMWF and DestinE open-source software stack.

What we're looking for

• Excellent analytical and problem-solving skills with a proactive, continuous improvement approach.
• Initiative and ability to work collaboratively with other ECMWF teams but also able to work independently.
• Good interpersonal and communication skills.
• Dedication, passion, and enthusiasm to succeed both individually and across teams of developers.
• Highly organised with the capacity to work on a diverse range of tasks to tight deadlines.
• Ability to collaborate effectively in an international work environment and with people from diverse backgrounds.

Education and Experience

• Advanced university degree (EQ7 level or above) or equivalent professional experience in computer science or engineering, computational science, physics or natural sciences, mathematics, or a related discipline.
• Demonstrated experience developing object-oriented software in C++ and Python
• Experience developing scientific software to process large datasets is an advantage.
• Experience developing interfaces between Python lower-level languages like C++ would be an advantage.
• Experience in system design or network development is an advantage.

Don't be discouraged if you do not meet all the requirements. ECMWF invests in people with training and multiple opportunities for development.

Skills and knowledge

• Competence in Object-Oriented Analysis and Design, preferably in C++ or Python.
• Familiarity with software lifecycle maintenance is highly desirable.
• Demonstrated ability programming in UNIX/Linux systems.
• Ability to write software in a distributed computing or scientific computing environment would be advantageous.
• Candidates must be able to work effectively in English.
• Good knowledge of one of the Centre's other working languages (French or German) would be an advantage.