Data Engineer - London, United Kingdom - Vallum Associates

Description

Job Description

My client is a leading TechFin company that is revolutionising finance through innovative technology solutions.

Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse.

Responsibilities:

• Develop robust data ingestion pipelines.
• Implement data transformation processes for analysis.
• Ensure seamless integration with Databricks Lakehouse.
• Implement security and compliance best practices.
• Collaborate on a scalable data lake architecture.
• Monitor and optimise data pipelines.
• Maintain thorough documentation.
• Plan for future scalability.
• Collaborate with cross-functional teams.

Required Skills:

• Proficiency in Azure services (SQL Database, Data Lake Storage, Databricks, CosmosDB).
• Understanding of data modelling and governance principles.
• Experience with Python, Databricks, and ETL tools.
• Familiarity with Spark, Kafka, and Azure service integration.
• Skills in Power BI or Azure Data Explorer for visualisation.
• Knowledge of data security and compliance.
• Strong analytical and communication skills.
• Understanding of CI/CD practices.
• Familiarity with Azure Machine Learning and.NET is a plus.

Benefits:

• Competitive salary & Day rate (looking for one contractor and one permanent position)
• Excellent benefits package, including pension, holidays, and insurance.
• Flexible working arrangements.