Barry Callebaut
Expert Data Engineer
Job Description
About the role
- Barry Callebaut Digital (BC Digital) is on a mission to lead the digital revolution in the chocolate industry, and we’re looking for an Expert Data Engineer, to help us build a foundation for actionable insights, based on robust, scalable, and efficient data pipelines. Reporting to the Head of Data Engineering, your focus will lie on ensuring data availability for analytical needs, through the transformation and provisioning of data from various source systems and domains across Barry Callebaut. As part of the central Data Engineering unit, you will be responsible for managing and coordinating external and distributed engineers, while providing community support, training, and governance to drive best practices across the engineering community within BC.
Key responsibilities include
- Design and build robust and scalable data pipelines to load and transform data from various source systems
- Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and build the right solutions
- Implement validation processes to ensure data is accurate, consistent, and accessible
- Build automation for repetitive tasks related to data analytics
- Monitor and improve pipeline performance, identifying and resolving bottlenecks
- Perform root cause analysis and implement corrective measures
- Document data engineering processes, systems, and procedures
- Identify opportunities to optimize pipelines for performance and cost efficiency
- Manage and coordinate BC’s external and distributed engineers, ensuring effective collaboration and integration
- Provide community support, training, and governance to drive best practices across the data engineering community
- Lead the development and implementation of global frameworks and patterns for data ingestion and pipelines, working closely with all other internal and external Data Engineer roles
About you
- Advanced knowledge of data engineering processes, with a minimum of 7 years experience (senior)
- Comprehensive understanding of data engineering patterns and best practices for pipeline orchestration
- Extensive experience in developing data analytics solutions on Azure
- Strong background in designing and building efficient, reliable, and automated data pipelines, ETL workflows, data warehousing, and Big Data processing, with experience in respective technologies like Airflow, dbt, etc.
- Proficiency in Spark and Databricks Lakehouse technologies
- Hands-on expertise with Python or PySpark
- In-depth knowledge of Azure Stack components like Azure Databricks, Azure Data Factory, Azure Synapse, and ADLS Gen2 from a data engineering perspective
- Familiarity with utilizing DevOps and DataOps methodologies to enhance development and deployment practices (e.g., CI/CD)
- Experience in managing and coordinating data engineers, providing community support, training, and governance to drive best practices
- Expertise in leading the development and implementation of global frameworks and patterns
- Excellent problem-solving skills and a structured way of working
- Collaborates well across diverse and globally distributed teams, with the ability to build and maintain positive relationships across different levels and functions of the organization
- Is a true team player, supporting their colleagues by sharing knowledge and experience and committing to the teams’ joint success
- Is capable of effective communication, conveying complex problems clearly and persuasively to internal and external stakeholders
- Is open to try and learn new technologies and skills
- Has an independent and self-driven personality, taking responsibility and owning tasks
- Independent and self-driven personality with strong analytical skills and techniques
- Is capable of guiding diverse teams and driving change collaboratively on a global scale
- Can create internal and external partnerships/networks across the organization with technical and non-technical stakeholders
