- At least two years of professional experience as a data engineer with a proven ability to provide effective data solutions in a production setting.
- Knowledge of developing real-time data stream systems (ideally Kafka).
- Proven track record in developing data systems using PySpark and Apache Spark for batch processing.
- Capable of managing data intake from various sources, including data streams, unstructured data, relational databases, and NoSQL databases.
- Extensive knowledge in distributed systems, cloud architecture, and data pipelines.
- Proficiency in Python programming (knowledge of Scala or Rust is a plus).
- Familiarity with ETL principles in contemporary data applications (Dagster, Airflow, Perfect).
- Familiarity with AWS services such as Glue, Redshift, Athena, and S3.
- Proficiency with Terraform, Kubernetes, and ArgoCD (expertise not required; cloud team support available).
- Thorough understanding of software engineering techniques, including version control, continuous integration and delivery (CI/CD), testing, and documentation.
- Well-versed in data lifecycle management, including development, storage, and access.
- Acquainted with Jira, Confluence, and other agile work and collaboration tools.
- A cooperative team member with strong communication skills to interact with various internal and external teams and stakeholders.
- Capable of operating in a fast-paced setting.
- Advocate of continuous development and open-mindedness.
- Proficiency in English, with the ability to work in the EU being a bonus.
- Develop and automate streaming data architecture, supporting the expansion of the data team in an interesting full-time position with plenty of challenges.
- Collaborate closely with other data engineers, scientists, analytics teams, and cloud engineering teams to develop and manage data products across multiple domains, providing scalable and dependable data for internal decision-making and client visibility.
- Play a major role in the development and optimisation of data systems to support our Data Lake/Data Mesh architecture and data processing needs, ensuring the provision of scalable, high-performance data products for the entire organisation and proper data governance.
- Utilise and improve our current AWS-based data platform.
- Work with our tech stack, which includes dbt/DuckDB for transformation, Kafka/RabbitMQ as a streaming platform, Deltalake as a data format, Dagster for managing data assets, and Terraform, Kubernetes, and ArgoCD as the underlying infrastructure.
- Build tools to produce data products and automate the self-serve platform.
- Create dependable and highly scalable data assets using Python, Scala, Spark/PySpark.
- Manage both fresh and current data sources into our pipelines for ELT and reverse ETL.
- Create a data strategy that uses appropriate resources to manage, classify, and locate important data while guaranteeing safe access.
- Use CI/CD, testing, and standard monitoring procedures to guarantee data quality and dependability.
- Continuously seek out and implement new technologies and industry best practices to enhance our data infrastructure.
- Work within the tech team using an agile methodology.
- Understand the business to comprehend the data demands of various internal and external stakeholders and design the best possible solutions.
-
Data Engineer
5 days ago
Higher - AI recruitment London Area, United KingdomAre you an experienced Data Engineer who wants to work at the cutting edge of engineering sustainability? · Our client is on a mission to empower engineers to use AI to overcome their most difficult physics and design challenges and build truly great products. With strong product ...
-
Data Engineer
1 week ago
Understanding Recruitment NFP London Area, United KingdomData Engineer · A brand new and exciting opportunity has recently arisen for a Data Engineer with an excellent knowledge in ETL, Azure Data Factory, Azure Databricks and SQL to join one of the UK's Largest Charity based in London. As the Data Engineer you will be shaping the da ...
-
Data Engineer
3 days ago
Prism Digital London Area, United KingdomLead Azure Data Engineer | PySpark (Python) & Synapse | Tech for Good / Charity · Essential skills required: · Azure – solid experience required of the Azure Data ecosystem · Python - ESSENTIAL as PySpark is used heavily. You will be tested on PySpark · Azure Synapse - ESS ...
-
Data Engineer
5 days ago
Selby Jennings London Area, United KingdomOur client, a prominent multi-strategy hedge fund, is seeking a talented Data Engineer to join their dynamic team in the London office. In this role, you will be a key contributor to building a robust data platform that services a wide range of investment teams and internal clien ...
-
Data Engineer
5 days ago
Amber Resourcing London Area, United KingdomData Engineer · £55,000 - £65,000 per annum + benefits · London · Hybrid (3 days a week in office) · Data Engineer | ETL | Data Pipelines | HealthTech | Beauty | Python | SQL | GCP | Azure | AWS | BigQuery | Cloud Functions | Pub/Sub | DataFlow | Data Studio | Shopify | Migration ...
-
Data Engineer
5 days ago
TRM-International London Area, United KingdomTRM-International are hiring on behalf of a Wholesale financial Institute. We are seeking a Data Infrastructure Engineer to develop and maintain their enterprise Data Platform. The platform aims at serving data processing, data consumption, business intelligence analytical tools, ...
-
Data Engineer
1 week ago
Travelex London Area, United KingdomJob Title: Data Engineer · Job Type: Full Time, Permanent · Working location: London, Hybrid · Role Purpose · At Travelex we are developing modern data services, which will be at the heart of our relationship with our customers. Our data architecture is becoming more serv ...
-
Data Engineer
5 days ago
Selby Jennings London Area, United KingdomWorking with a high performing multi strategy hedge fund that are looking to hire into their London team which is made up of data sourcing experts, data scientists, engineers and specialists. Adding an engineer to the team. · Mission of this engineer being to deliver financial an ...
-
Data Engineer
1 week ago
Axtria - Ingenious Insights London Area, United KingdomIntroduction · Axtria is a global provider of cloud software and data analytics to the Life Sciences industry. We help Life Sciences companies transform the product commercialization journey to drive sales growth and improve healthcare outcomes for patients. We are acutely awar ...
-
Data Engineer
5 days ago
Miki Travel Europe London Area, United KingdomData Engineer · Embark on a Rewarding Journey with MIKI Travel: Where Your Career Takes Flight. At MIKI Travel, our 55+ years of expertise make us a trailblazing wholesale travel provider, curating unforgettable global adventures. As a premier private travel company with a prese ...
-
Data Engineer
5 days ago
Mastek London Area, United KingdomJob Description: Data Engineer · Location: [UK] · SC Cleared: Required · Job Type: Full-Time · Experience: 12-14 years · Job description: · We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The Data Engineer will be responsible for designing, b ...
-
Data Engineer
5 days ago
Taylor Root London Area, United KingdomAn international Insurance company which is headquartered in the US, is looking to hire a Data Science Manager do join its Analytics team in London. · This team is well established and has been using leading technologies for the past few years. · Job Responsibilities: · Collabora ...
-
Data Engineer
5 days ago
Synchro London Area, United KingdomPosition: Data Engineer · Location: Clapham, hybrid working (2 days per week in the office) · Salary: £65,000 - £75,000 · Synchro is partnering exclusively with a rapidly growing online digital e-commerce business, now recognised as one of the UK's leading brands in their sector. ...
-
Data Engineer
5 days ago
Immersum London Area, United KingdomData Engineer - Exchange trading platform · Location: London, 3 days per week in the office · Salary range: £60,000-£65,000 + benefits · Purpose: Design, build, and maintain scalable data architectures · Tech: Python (NumPy, Pandas), SQL, ETL, Cloud (AWS, Azure or GCP), Snowflake ...
-
Data Engineer
1 week ago
Seargin UK & Ireland London Area, United KingdomData Engineers needed for 6-12 month project based in London paying £450 a day inside IR35. This is an Urgent role an ideally we need the data engineers to be able to start asap or within 3 weeks as a maximum. All data engineers have to able to work in London 3 days a week, have ...
-
Data Engineer
5 days ago
Wildfarmed London Area, United KingdomAre you making the most of being alive? · Do you feel optimistic about the future? · Have you been waiting for the perfect opportunity to put your skills towards a project that could change the world? · Do you want to wake up every day feeling full of life? · If so - hi, we're Wi ...
-
Data Engineer
5 days ago
Qurated Network London Area, United KingdomOur client is a leading provider of specialist governance and compliance solutions for the asset management industry and are looking for a Data Engineer to join their team. Partnering with asset managers and institutional investors they are currently serving assets under manageme ...
-
Data Engineer
5 days ago
Venquis London Area, United KingdomMy leading London Market client are seeking an experienced Data Migration Analyst/ Data Engineer to join the Integrations team. They are dedicated to optimising our data migration processes as we integrate newly acquired businesses. You will be migrating data from various sources ...
-
Data Engineer
3 weeks ago
Talent London Area, United KingdomData Engineer – SAS / SQL/ Power BI · Sponsorship is not available with this position * · Role – Data Engineer · Salary – Up to £75k DOE · Location – Manchester or London (Hybrid) · A well established and award-winning Data consultancy are looking to add a Senior Data Con ...
-
Data Engineer
3 days ago
TRM-International London Area, United KingdomTRM-International are hiring on behalf of a Wholesale financial Institute . We are seeking a Data Infrastructure Engineer to develop and maintain their enterprise Data Platform. The platform aims at serving data processing, data consumption, business intelligence analytical tools ...
Data Engineer - London Area, United Kingdom - Vallum Associates
Description
Requirements
What the Role Entails
As a growing company with expanding data demands, scalable, dependable, and efficient data architecture is required. We are now seeking a Data Engineer to join our Data team to help build, upgrade, and enhance our current data architecture.
Additional Responsibilities
This position offers the opportunity to work with a bright and varied team to develop and improve our data platform for data processing, access, gathering, and monitoring, building the future of our data infrastructure.
