Lead Product Software Development Engineer

We are seeking a highly experienced Lead Developer, QA, and Operations Strategist with a minimum of 10 years of experience in Azure, SQL, ADF, PySpark, Databricks, and Python frameworks for test automation. The ideal candidate will help establish high-quality data standards, automate testing processes, and lead data operations within Azure environments. This hybrid role spans development, QA, and operational strategy, focusing on automation and advanced data solutions.

Key Responsibilities:

Lead Development, QA, and Operations Strategy

• Establish and maintain best practices for data lake frameworks, ensuring data quality within Azure-based architectures.
• Design and implement solutions using Azure Data Factory (ADF), Databricks, and PySpark for data processing and integration.
• Drive automation of data pipelines and testing strategies for performance and scalability in Azure environments.
• Develop and manage test automation frameworks using Python.

Automation & Testing

• Automate ETL and API testing for data lakes and data warehouses, with a focus on Azure solutions.
• Design, create, and execute high-quality reusable manual and automated test cases to ensure complete test coverage.
• Build, enhance, and maintain test automation frameworks using Python for functional, integration, and regression testing.
• Leverage automated test frameworks and continuous build tools like Azure DevOps to streamline the development and testing lifecycle.
• Validate and test data using SQL, writing complex queries to ensure data integrity.
• Support and enhance automated and manual testing processes, including unit, system, performance, and load testing.

Azure & Data Expertise

• Build and maintain scalable data pipelines using Azure Data Factory (ADF), Databricks, and PySpark to ensure efficient data movement and processing.
• Optimize data workflows within Azure to handle large datasets, ensuring data quality and reliability.
• Automate data validation processes using SQL and Python to ensure accuracy and consistency.

Python Automation Frameworks

• Develop and maintain test automation scripts using Python to automate functional, regression, and performance testing.
• Implement Python-based frameworks for automated testing of data pipelines, APIs, and backend systems.
• Continuously improve automation tools and frameworks to meet evolving organizational needs.

SQL Expertise

• Write and optimize complex SQL queries, including analytical functions, for data validation and testing purposes.
• Use SQL to create and validate test cases, ensuring robust data integrity across all projects.

Collaboration

• Collaborate with cross-functional teams including data engineers, developers, and QA professionals to align development and testing strategies.
• Mentor and guide junior staff members in development, QA, and test automation best practices.
• Manage individual career development, and engage in discussions with leadership on personal growth opportunities.

Continuous Integration/Deployment (CI/CD)

• Ensure code quality through continuous integration and deployment pipelines within Azure DevOps.
• Develop and maintain automated test suites within the CI/CD pipelines to validate code changes and ensure high-quality releases.
• Identify and resolve potential vulnerabilities, edge cases, and system failure points to improve system robustness.

Qualifications:

• Minimum of 8 years of experience in development and QA roles, with a focus on cloud-based data solutions.
• Strong expertise in Azure, including Azure Data Factory (ADF), Databricks, and PySpark.
• Proficiency in developing and maintaining Python-based test automation frameworks.
• Expertise in writing complex SQL queries, including analytical functions, for data validation and optimization.
• Extensive experience in automating ETL and API testing for data lakes and data warehouses.
• Proven leadership skills in mentoring junior staff and guiding their development.

Preferred Skills:

• Strong experience in both development and QA, with a focus on Azure-based data operations.
• Expertise in managing test automation frameworks using Python and integrating them with cloud environments.
• Experience with test case management and defect tracking tools within Azure DevOps.
• Ability to solve complex problems independently and work effectively in a fast-paced environment.

This role provides an exciting opportunity to lead automation, development, and data quality strategies within an Azure-based cloud environment. If you’re passionate about data automation, cloud solutions, and driving high-quality outcomes, we invite you to apply!