Lead Data Engineer Azure - London, United Kingdom - Jefferson Frank

Jefferson Frank
Jefferson Frank
Verified Company
London, United Kingdom

1 month ago

Tom O´Connor

Posted by:

Tom O´Connor

beBee Recruiter
Description

Lead Data Engineer - Azure, Hybrid, East London - DataLakes, PySpark, SQL, Azure, Python, AWS, Databricks, Agile

Role Overview

Role & Responsibilities

  • Collaborate with Big Data Solution Architects to design, prototype, implement, and optimize data ingestion pipelines so that data is shared effectively across various business systems.
  • Build ETL/ELT and Ingestion pipelines and design optimal data storage and analytics solutions using cloud and onperm technologies.
  • Ensure the design, code and procedural aspects of the solution are production ready, in terms of operational, security and compliance standards.
  • Participate in daytoday project and product delivery status meetings, and provide technical support for faster resolution of issues.

Skills and Experience

  • Demonstrable
    design & development experience and experience with
    big data technologies like
    Spark/Flink and Kafka
  • Proficient in
    Python, PySpark, or Java/Scala.

Hands-on experience with some of the following technologies:


  • Azure/AWS
  • Data Lake Projects:
  • SQL
  • ETL/ELT
  • Databricks
  • Spring/Guice or any other DI framework,
  • RESTful Web Services.
  • Proficient in querying and manipulating data from various DB (relational and big data).
  • Experience of writing effective and maintainable unit and integration tests for ingestion pipelines.
  • Experience of using static analysis and code quality tools and building CI/CD pipelines.
If you are interested and meet the above requirements please send your
latest CV and call
More jobs from Jefferson Frank
See all jobs of Jefferson Frank