Data Engineer

Project Role : Data Engineer
Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
Must have skills : Advanced Embedded System Engineering
Good to have skills : Big Data Analysis Tool and Techniques, Data Mining, Cloud Data Architecture, Azure Cloud, Spark Applications
Minimum 5 year(s) of experience is required
Educational Qualification : Bachelor of Engineering degree in Electronics / Electronics Communication Engineering/Mechatronics/

Experience
• Above 3 – 6 years of experience in setting up data pipelines to injest streaming and batch data from many sources
• Experience on optimized data collection procedures
• Good experience in performing extract, transform and load (ETL) processes to make the data more usable with cloud infrastructures
• Expertise in data storage structures, data mining, and data cleansing
• Hands-on experience using Big Data components (Hadoop, Spark) & programming languages.
• Experience in using, or migrating to cloud databases
• Proficient in relational database design and development
• Proficient in Data warehousing concepts and up to date Data Integration patterns/technologies
• Experience in Azure data lake, Azure SQL Database, Data Bricks, Azure SQL Data warehouse and Azure Data factory.
• Experience in Developing Spark applications using Spark – SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights.
• Systematic problem-solving approach with strong communication skills and a sense of ownership and drive

Skills
• Technical Skills

• Strong Experience in Azure Cloud
• Experience with distributed data/computing tools: Map/Reduce, Hadoop, Hive, Spark
• Strong knowledge in Azure Analytics Products like Data Factory, HD Insights, Azure Data Lake Storage, Azure SQL Datawarehouse & Data Bricks
• Strong in Python Programming Skills
• Strong experience in networking and security measures
• Good knowledge in using NoSQL databases like MongoDB, PostgreSQL and Cassandra running huge volumes of data
• Proficient in with SQl Databases like MSSql, Postgres.
• Expertise in Data Warehousing – Hadoop, MapReduce, HIVE, PIG, Apache Spark, Kafka
• Good understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, Driver Node, Worker Node, Stages, Executors and Tasks.
• Good understanding of Big Data Hadoop and Yarn architecture along with various Hadoop Demons such as Job Tracker, Task Tracker, Name Node, Data Node, Resource/Cluster Manager, and Kafka (distributed stream-processing).

Job Responsibilities

• Building and maintaining the data architecture for data science projects.
• Extract Transform and Load data from sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics.
• Data Ingestion to one or more Azure Services – (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Data bricks.
• Define table schema and quickly adapt with the pipeline.
• Improve data foundational procedures, integrating new data management technologies and softwares into the existing system, building data collection pipelines
• Identify valuable data sources and automate collection processes
• Implement concepts of programming such as data structures, error handing, data manipulation and I/O processing. Implements database concepts and practices including definition and query language
• Implement adequate networking and security measures for the data pipeline.
• Implement monitoring solution for the data pipeline.
• Develop Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Support the design, and implement data engineering solutions
• Maintain excellent documentation for understanding and accessing data storage
• Work independently as well as in teams to deliver transformative solutions to clients
• Be proactive and constantly pay attention to the scalability, performance and availability of our systems
• Establishes privacy/security hierarchy and regulates access
• Undertake preprocessing of structured and unstructured data
• Support in collecting, organizing, and interpreting data along with fellow colleagues.
• Collaborate with engineering and product development teams

Bachelor of Engineering degree in Electronics / Electronics Communication Engineering/Mechatronics/

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Accenture is committed to providing veteran employment opportunities to our service men and women.