MSD

Lead Data Engineer

16 January 2025
Apply Now
Deadline date:
£28000 - £67000 / year

Job Description

Job Description

Position Overview: We are seeking a skilled and experienced Lead Data Engineer to join our growing data team. This role will involve designing, building, and maintaining scalable data pipelines and ETL processes. You will lead a team of data engineers and collaborate with data scientists, analysts, and stakeholders across the organization to deliver solutions that drive business insights.

Key Responsibilities:

  • Lead the design, development, and implementation of data pipelines and ETL workflows using Matillion and Databricks.
  • Architect and implement data solutions in AWS, including but not limited to Amazon S3, AWS Glue, Amazon Redshift, and Amazon RDS.
  • Manage and optimize workflow orchestration using Apache Airflow to ensure reliable data processing and job scheduling.
  • Collaborate with cross-functional teams to gather requirements and translate them into scalable data architecture and process designs.
  • Mentor and guide junior data engineers, providing technical leadership and fostering a culture of continuous improvement and innovation.
  • Ensure data quality and integrity by implementing best practices in data governance and validation.
  • Monitor performance, troubleshoot issues, and optimize data systems for efficiency and scalability.
  • Stay abreast of industry trends and emerging technologies to ensure continuous improvement of the data engineering practices.

Qualifications:

  • Bachelor’s degree in Computer Science or related Engineering, or a related field.
  • 10+ years of experience in data engineering or related roles, with a proven track record of leading technical teams.
  • Extensive experience with AWS services, particularly in data storage, data processing, and database management.
  • Hands-on experience with Matillion for ETL processes and data integration.
  • Proficient in using Databricks for data analysis and processing with Apache Spark.
  • Strong knowledge of Apache Airflow for workflow orchestration and scheduling.
  • Experience with SQL and data modeling principles; familiarity with NoSQL databases is a plus.
  • Excellent problem-solving skills and ability to work in a fast-paced, agile environment.
  • Strong communication skills to effectively interact with technical and non-technical stakeholders.

Preferred Skills:

  • Experience with Python, Pyspark, and SQL for data processing and scripting.
  • Experience in AWS services,Databricks,Airflow & Matillion
  • Familiarity with data visualization tools such as Power BI.
  • Knowledge of machine learning practices and frameworks.

Current Employees apply HERE

Current Contingent Workers apply HERE

Search Firm Representatives Please Read Carefully 
Merck & Co., Inc., Rahway, NJ, USA, also known as Merck Sharp & Dohme LLC, Rahway, NJ, USA, does not accept unsolicited assistance from search firms for employment opportunities. All CVs / resumes submitted by search firms to any employee at our company without a valid written search agreement in place for this position will be deemed the sole property of our company.  No fee will be paid in the event a candidate is hired by our company as a result of an agency referral where no pre-existing agreement is in place. Where agency agreements are in place, introductions are position specific. Please, no phone calls or emails. 

Employee Status:

Regular

Relocation:

VISA Sponsorship:

Travel Requirements:

Flexible Work Arrangements:

Hybrid

Shift:

Valid Driving License:

Hazardous Material(s):

Required Skills:

Business Intelligence (BI), Database Administration, Data Engineering, Data Management, Data Modeling, Data Visualization, Information Management, Information Technology (IT) Infrastructure, Network Infrastructures, Software Development

Preferred Skills:

Job Posting End Date:

01/30/2025

*A job posting is effective until 11:59:59PM on the day BEFORE the listed job posting end date. Please ensure you apply to a job posting no later than the day BEFORE the job posting end date.