Spark Architect - Sheffield, United Kingdom - PRACYVA

Description

Job Description

Spark Architect / SME

Contract Role- 6 months to begin with & its extendable

Location: Sheffield, UK (min 3 days onsite)

• Context: Legacy ETL code for example DataStage is being refactored into PySpark using Prophecy low-code no-code and available converters.
• Converted code is causing failures/performance issues.

Skills:

• Spark Architecture – component understanding around Spark Data Integration (PySpark, scripting, variable setting etc.), Spark SQL, Spark Explain plans.
• Spark SME – Be able to analyse Spark code failures through Spark Plans and make correcting recommendations.
• Spark SME – Be able to review PySpark and Spark SQL jobs and make performance improvement recommendations.
• Spark – SME Be able to understand Data Frames / Resilient Distributed Data Sets and understand any memory related problems and make corrective recommendations.
• Monitoring – Be able to monitor Spark jobs using wider tools such as Grafana to see whether there are Cluster level failures.
• Cloudera (CDP) – Knowledge of understanding how Cloudera Spark is set up and how the run time libraries are used by PySpark code.
• Prophecy – High level understanding of Low-Code No-Code prophecy set up and its use to generate PySpark code.