Lead Data Engineer

• Implement enterprise data lake, develop data transformation & conversion strategies from multiple source systems, federated Datahouses/Data Lakes with data retention & archiving strategies along with regulatory requirements
• Build data pipelines to implement Analytics. AI/ML use cases in real time and batch
• Create data consumption layer for BI/visualization tools
• Establish data governance, data security and data cleansing procedures to ensure Data Quality & Compliance
• Mange cloud consumption cost through optimization of services

Skill and Experience

• 4+ years of experience in cloud platforms of Azure\GCP
• 3+ experience in Databricks
• Experience in building data pipelines (batch and real time) and handling variety of data sources including semi-structured and unstructured data sources (image/video, GPS. IoT)
• Strong expertise in ETL/ELT processes, data lake, Datawarehouse, data mesh
• Excellent knowledge of python, SQL, cloud DWHs (Synapse, BigQuery), managed DBs (CloudSQL, Azure DBs), PostgreSQL
• Hands on experience in Databricks (DLT, CDC, Lakeflow Connect), Unity catalogue
• Knowledge of open table formats (Delta, Iceberg) and tools (Trino, Flink, Presto)
• Good knowledge of scheduling and orchestration tools like Airflow, Compose
• Expertise in Azure services like ADLS, ADF, Functions, Event Grid, Stream Analytics
• Expertise in GCP services like GCS, Dataflow, Dataproc, Cloud Functions, Cloud Run
• Experience in developing and deploying API endpoints on API gateways/APIGEE
• Experience in handling streaming data and messaging tools (Kafka, Pub/Sub, Service Bus Messaging, Event Hub)
• Experience in deploying workloads on managed Kubernetes clusters
• Strong analytical & problem-solving skills with the ability to think critically & creatively about complex business problems
• Ability to work independently & collaboratively and to manage multiple projects & priorities simultaneously
• Strong communication skills
• High level of collaboration & stakeholder management

Education Qualification

• Bachelor’s degree in engineering
• Professional certification in Azure, GCP, Databricks