Sr Technical Lead

Location- Bangalore or Chennai , Immediate joiners

Role:

We are looking for an experienced Azure Databricks Data Engineer. This role involves designing and implementing ETL pipelines using Azure Databricks, PySpark, Azure SQL, and other Azure services. Should have strong background in data engineering and ETL processes.

Required Qualifications:

• Experience: 3+ years of experience in data engineering, with a focus on Azure Databricks and PySpark.
• Technical Skills:
  • Proficiency in Azure Databricks, PySpark, and Azure Data Factory.
  • Strong knowledge of SQL and experience with Azure SQL Database.
  • Experience with Delta Lake for scalable and reliable data lake solutions.
  • Familiarity with data warehousing concepts and ETL processes.
  • Knowledge of data integration tools and techniques within the Azure ecosystem.
• Soft Skills: Excellent problem-solving skills, attention to detail, and the ability to work in a fast-paced environment.

Responsibilities:

• ETL Pipeline Development: Design, develop, and maintain scalable ETL pipelines using Azure Databricks and PySpark to process large volumes of data.
• Data Transformation: Perform complex data transformations and aggregations, ensuring data quality and consistency.
• Data Integration: Integrate various data sources, including ADO, SNOW, and third-party APIs, into a unified data platform.
• Performance Optimization: Optimize ETL processes for performance, scalability, and cost-effectiveness using best practices in Azure Databricks and PySpark.
• Collaboration: Work closely with stakeholders to understand data requirements and deliver actionable insights.
• Automation: Automate data pipeline workflows using Azure Data Factory and ensure robust scheduling and monitoring.
• Documentation: Maintain comprehensive documentation of data workflows, processes, and systems.
• Compliance and Security: Ensure all data processing complies with security and data governance standards.