- Extensive experience working as a Data Engineer
- Experience designing and building Databricks data products
- Strong programming skills in languages such as Python (PySpark preferred), Scala, or SQL
- Experience in owning, designing and implementing data pipelines ingesting enterprise levels of data volume. In-depth knowledge of data engineering concepts, data integration, and ELT/ETL processes.
- Ability to write production-grade code including automated testing and able to deploy code via CI/CD platforms (e.g. Github Actions, Jenkins)
- Proficiency in working with distributed computing frameworks, such as Apache Spark and data modelling, database systems, and SQL optimisation techniques
- Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and associated services (e.g., S3, Redshift, BigQuery).
- Knowledge of the UK broadcast industry & broadcast/OTT advertising market and digital marketing and advertising industry
- Familiar with:
- Data design techniques for classification and regression as well as knowledge of A/B testing, experimental design, and general statistical modelling
- Architecture disciplines such as Data Mesh Architecture, Data Architecture, BI Architecture and Enterprise Architecture
- Data governance, data privacy, and security principles.
- Tools across other data management capabilities – data engineering, data governance, data quality, metadata management and master data management
- Designing and building out data visualisation solutions based on technology such as: Tableau, AWS QuickSight, Looker, ThoughtSpot
- Exceptional communication skills, delivery focused with a technical outlook and ability to influence stakeholders at all levels.
Senior Data Engineer - London, United Kingdom - ITV
Description
Senior Data Engineer
Data Insights
Permanent
Office location: ITV White City
Hiring range: £70,000 - £85,000
Your work matters to millions.
Shaping culture is in the DNA of ITV. So, it's not surprising that you'll find us in every home in the UK, our productions are famous all over the world and we're at the forefront of the digital streaming revolution.
When you join us, you enter a fun working environment. With opportunities to learn, to grow and make a real difference. Small enough that your impact's felt in the business, but big enough that your impact reaches millions of people.
Come develop your skills, change TV and the course of your career. Don't just watch it. Be part of it. Join ITV.
Your impact sends ripples.
Our data team is responsible for helping us realise the value of data. That could mean anything from up-skilling and training our people, to increasing our revenue through experimentation and personalisation. The list is truly endless. The Innovation strand consists of Data Scientists, Operational Researchers and Behavioural Scientists. As a mix of number crunchers and behavioural experts, they put their curiosity to work in influencing the business. As we enter the fourth industrial revolution, anyone who joins the team will have the opportunity to work on genuinely groundbreaking innovations so that we can continue to blend science into a creative business.
As a Senior Data Engineer, you will be responsible for developing, maintaining, and optimising our data pipeline infrastructure using our proprietary central data platform (Symphony), which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis. You will be comfortable working independently as well as within a team. You will lead development initiatives by providing solution designs accompanied with reliable work estimations. You will be instrumental in bringing the team together and defining what success looks like from a technical perspective.
Responsibilities of this role include:
· Ownership and delivery of the design, development, and maintenance of scalable data pipelines, infrastructure and data products using Databricks.
· Assist and collaborate with the ITV commercial teams and data product owners to define data product requirements and visualisation needs
· Develop and optimise Extract, Transform, Load (ETL/ELT) processes to ensure the smooth flow of data from various sources into the data platform. Perform data cleansing, validation, and enrichment to support accurate and reliable data analysis.
· Collaborating with data scientists, analysts and stakeholders capturing and defining data requirements, designing appropriate data models and architectures whilst optimising data storage formats and partitioning for efficient data retrieval and analysis.
· Monitor and optimise data pipelines and workflows to ensure high performance and reliability. Identify and resolve performance bottlenecks, data quality issues, and data integration challenges.
· Define the data quality characteristics of your data products and work with the QA team to implement automated quality processes to measure data correctness and report outcomes through our observability infrastructure
· Implement data governance practices and ensure compliance with ITV data privacy and security regulations. Establish data access controls, encryption mechanisms, and data retention policies. Execute and comply with ITV architecture governance processes.
· Work closely with cross-functional teams, including data scientists, analysts, and software engineers, to understand their requirements and provide data engineering support. Document data pipelines, workflows, data presentation implementations and technical specifications for future reference. Communicate complex solutions in a clear and understandable way to both technologists and business stakeholders
· Stay up-to-date with industry trends and emerging technologies in the data engineering domain.
Skills you'll need (minimum criteria)
Other things we're looking for (key criteria)