Senior Data Engineer - Manchester Area, United Kingdom - Mobysoft

Description

Senior Data Engineer

Mobysoft is one of the fastest growing SaaS providers in the UK and has been shortlisted in the "Top 50 fastest growing technology companies in the North" for four successive years. Mobysoft provides predictive analytical software that has helped our customers deliver tens of millions of pounds in savings. Such is our success; customer numbers have doubled in little over two years and as a result we're looking for great people to join our exciting team.

We have also just won AI Project of the year at the Real I.T Awards 2024- one of the many awards we are proud of

Role overview

We are an ambitious, customer-centric Data & Insights team, dedicated to developing a new generation of data products that unlock significant value for the social housing sector. We operate with a focused product lens, driven by curiosity and a commitment to technical excellence.

Data Engineering is foundational to this mission, ensuring synchronized, curated, trusted, dynamic data is available at speed and scale for our clients, data analysts,data scientists, and stakeholders.

As such we are looking for a most excellent Senior Data Engineer (SDE) professional to join our dynamic team, and working with our most excellent existing SDE Bartek

What will you be doing?

Bartek is redeveloping our teams data platform from a legacy system to one based on AWS EMR, with Amazon RDS and DynamoDB ingestion converted to Parquet files, interrogatable through Spark and MapReduce. This modern platform will support rapid data insight generation, data experiments for new product development, our live Machine Learning solutions and live client facing dashboards.

To complete this initial work (stage one), together you will ensure efficient, robust data flows (including job scheduling and the ability to easily ingest 3rd party data), a reproducible/resilient platform via infrastructure as code, peer reviewed scripts, appropriate and documented processes in support of effective information security/data governance.

Once up and running, your next focus (stage two) will be around helping to further develop our data reliability, including building out an active metadata platform, developing data quality monitoring and designing/implementing service level objectives (SLOs) and service level indicators (SLIs).

Stage two will ensure you get to know the data really well, which will be invaluable for stage three - building out an API led feature factory to support data analytics/science, developing (API led) data flows that support our dashboards and building capabilities to work effectively with unstructured data.

During this time we will rely on your interest and knowledge to help steer our direction and technology choices. After stage three alongside you helping to keep the show on the road, there will be many new challenges reliant on your application.

These include ingesting and understanding new data sources, evolving our data approach (e.g., to a fully fledged data fabric), implementing containers to streamline MLOps and exploring how we can sensibly and safely utilise artificial intelligence (AI) to turbo charge our work (e.g., smart data quality monitoring, meta data curation and service optimisation).

In short there will be plenty to keep you interested, your technical and power skills developing and a real chance to drive innovation and change. As part of this we'll also ask that you liaise with our clients (directly or through events, conferences, webinars etc.), to help support their data driven journeys.

Please note this is a professional, technical role, with no line management responsibilities. There will however be multiple opportunities to lead data analytics workstreams

Why should you consider this role?

Alongside the work opportunities as described your development will be supported, you will be sensibly renumerated, we will provide a compelling benefits package and we are a great bunch of folks to work with - though we would say that

What qualifications and experience are we after?

• The successful person will ideally have a Masters, merit and above in Computer Science, Software Engineering, Big Data, Information Systems, or similarly related and quantitative discipline.
• AWS Certified Data Engineer, Associate certification.
• Circa 4-5+ years of serious commercial experience working primarily in an AWS Cloud environment using approaches/tooling similar to ours (see technical skills), delivering scalable, performant, reliable solutions.
• Data reliability/observability, data governance and information security credentials

What technical skills are required

• Ability to implement and use modern data platform concepts, frameworks and tooling, covering all data orchestration aspects from ETL/ELT data flows (including building out source-to-target mappings) to testing and service optimisation.)
• Good familiarity with our developing key services/applications - AmazonRDS, Amazon DynamoDB, AWS Glue, MapReduce, Hive, Spark, YARN, Airflow.
• Ability to work with a range of structured, semi-structured and unstructured file formats including Parquet, json, csv, pdf, jpg.
• Accomplished data wrangling and scripting abilities, with advanced knowledge of Python, Linux shell scripting, SQL and NoSQL.
• Tools and methods to develop comprehensive data reliability and active metadata solutions.
• Ability to work with and develop APIs (including data transformations).
• Ability to deliver data deidentification and anonymisation solutions.
• Understanding of Cloud security frameworks (specifically on the AWS Cloud) including appropriate data encryption.

What experience and technical skills are desirable

• Utilising AI within data engineering to drive performance.
• Knowledge of implementing, managing and utilising graph databases (e.g., using technologies such as Neo4j).
• MLOps experience including familiarity with Docker and Kubernetes.
• Full data engineering cycle knowledge (tools and skills) for stream data.

Benefits

Competitive salary and rewards package including: - Private Health care , 4 x salary Life cover, 25 days annual leave, increasing to 28 after 3 years service ,salary sacrifice pension scheme and much more including investment and people and training ..

Inclusion

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, colour, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process-please contact us to request accommodation. We offer hybrid roles, so we're happy to discuss flexible working

If you are interested and would like to know more then please apply ASAP to or via our careers page.