The Crown Estate

Data Engineer

8 November 2024
Full Time
Apply Now
Deadline date:
£109000 / year

Job Description

Are you a data enthusiast ready to make an impact?

Join us in building the new Crown Estate Azure data platform! As a Data Engineer, you’ll deliver technical solutions and create end-to-end analytical systems that are highly available, scalable, stable, secure, and cost-effective. Your main goal will be to build, deploy and automate data pipelines using the Azure cloud platform, including IAC (Bicep), Data Factory, Python/PySpark and Databricks. You’ll support the Senior Data Engineer in making data usable, monitoring its quality, and delivering data services that drive The Crown Estate’s strategy.

Key Responsibilities:

  • Design, implement, and operate large-scale, high-performance data pipelines for analytics and data science.
  • Apply best practices in development, test design/automation, CI/CD, and documentation.
  • Develop, monitor, and maintain data pipelines, ensuring consistent availability and quality.
  • Analyse data structures, identify performance changes, and suggest improvements.
  • Understand and meet stakeholder requirements.
  • Ensure information security at all times.

Experience and Skills:

  • Degree in Computer Science, Engineering, Mathematics, or related field, or equivalent industry experience.
  • Proven experience in developing data ingestion, processing, and analytical pipelines for big data, relational databases, NoSQL, and data warehouse solutions.
  • Extensive hands-on experience implementing data pipelines & services using Azure: Data Factory, Functions, Synapse, Azure SQL DB, Event Hubs/Grids, Databricks, Unity Data Catalog, Cosmos Db, etc.
  • Hands-on experience in Big Data technologies such as PySpark, Python, SQL, Databricks, ADLS/Storage options, Lakehouse design, File formats (JSON/Delta/Parquet), Powershell, and streaming technologies.
  • Well versed in DevOps, IaC (Bicep/Terraform) and CI/CD deployments, experience working with Developer tools such as Azure DevOps, VS Code, GitHub, etc.
  • Experience in building scalable and performant API data services.
  • Awareness with Azure AI technologies and tools: openAI, ML, Cognitive Services, AI Studio
  • Excellent stakeholder management skills at all levels.

Nice-to-Have Certifications:

  • DP-200 Implementing an Azure Data Solution / DP-201 Designing an Azure Data Solution is desirable but not essential

Key Business Capabilities:

  • Continuous Improvement: Lead in adopting industry best practices and staying updated with technical and compliance requirements.
  • Communication: Strong interpersonal skills, especially under pressure. Ability to explain technical topics in simple terms.
  • Technology and Data: Expertise in developing and implementing high-quality technology solutions and processes in a matrixed organisation.
  • Customer Excellence: Commitment to continuous improvement and challenging the status quo to enhance business growth and customer experience.
  • Collaboration: Team player with the ability to influence and align stakeholders and partners, build consensus, and support major initiatives.

Join us and play a pivotal role in driving our mission forward with cutting-edge data solutions!

Advert Closes: Thursday 14th November 

Related Jobs