Lead Data Engineer-Digital Banking Kotak 811-Regional Sales

Job Title: Lead Data Engineer

Job Description:

As a Lead Data Engineer, you will be responsible for overseeing the design, development, and maintenance of data infrastructure and solutions within @Kotak811. 

• You will lead a team of data engineers, collaborate with stakeholders, and provide technical leadership to drive the success of our data initiatives. 
• Your deep understanding of data engineering principles, advanced technical skills, and strategic mindset will be critical in shaping our data strategy and driving innovation.

Responsibilities:

1. Data Strategy and Architecture:
   1. Define and drive the data engineering strategy, aligning it with the organization’s goals and objectives.
   2. Design and architect scalable, secure, and performant data solutions
   3. Evaluate emerging technologies, tools, and frameworks to enhance the data engineering ecosystem.
   4. Collaborate with stakeholders to identify data requirements and translate them into technical solutions.
2. Team Leadership and Management:
   1. Lead and manage a team of data engineers, providing technical guidance and mentorship.
   2. Set clear goals, monitor progress, and provide regular feedback to team members.
   3. Foster a collaborative and inclusive team culture, promoting knowledge sharing and professional development.
   4. Drive recruitment efforts to attract and retain top data engineering talent.

1. Data Engineering Project Management:
   1. Lead end-to-end data engineering projects, from requirements gathering to solution delivery.
   2. Collaborate with cross-functional teams, including data scientists, business analysts, and software engineers, to ensure successful project outcomes.
   3. Define project scope, timelines, and resource allocation, and manage project risks and issues.
   4. Monitor project progress and ensure adherence to quality standards and best practices.
2. Data Infrastructure and Governance:
   1. Oversee the development and maintenance of scalable data infrastructure, including databases, data warehouses, and data lakes.
   2. Implement and enforce data governance policies, standards, and best practices.
   3. Collaborate with data governance teams to ensure compliance with data privacy and security regulations.
   4. Continuously assess and optimise data infrastructure for performance, scalability, and cost efficiency.
3. MLOps[1] 
   1. Lead and manage a team of data engineers to design, develop, and maintain MLOps infrastructure and processes for real-time consumer applications.
   2. Collaborate with data scientists and software engineers to define requirements and translate them into scalable MLOps pipelines.
   3. Architect and develop infrastructure for model training, deployment, and monitoring.
   4. Implement efficient and reliable model deployment strategies using containerization and orchestration technologies like Docker and Kubernetes.
   5. Design and implement scalable data pipelines for real-time data ingestion, preprocessing, and feature engineering.
   6. Optimise model performance, latency, and scalability for real-time inference in consumer applications
4. Technical Leadership and Innovation:
   1.  Stay abreast of emerging trends, technologies, and best practices in data engineering.
   2.  Provide technical expertise and guidance to solve complex data engineering challenges.
   3.  Drive innovation by identifying opportunities to leverage advanced analytics, machine learning, or AI techniques in data engineering solutions.
   4.  Foster a culture of continuous improvement and drive process efficiencies within the data engineering team.

Qualifications:

1. 8-10 years of experience in data engineering(including designing and implementing data solutions), model deployment, and MLOps with Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
2. Experience with Enterprise Business Intelligence Platform/Data platform sizing, tuning, optimization and system landscape integration in large-scale, enterprise deployments.
   1. Deep understanding of data engineering principles, data architecture, and data integration patterns.
   2. Experience with big data technologies and frameworks such as Apache Spark, Hadoop, or similar.
   3. Candidate who understands the security frameworks, best practices, treat data as assets in EDW ecosystem
3. Experience working extensively in multi-petabyte DW environment
4. Experience in engineering large-scale systems in a product environment
5. Strong leadership and people management skills, with the ability to lead and inspire a team.
6. Excellent communication and stakeholder management skills.
7. Strategic mindset and ability to align data engineering initiatives with business objectives.
8. Demonstrated ability to drive innovation and foster a culture of continuous improvement.

@[email protected] – Going too far. ?

_Assigned to Anshul Pahwa_