Senior Data Engineer – Hadoop, Spark, Kafka, Hive

Company Description

Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to connect the world through the most innovative, convenient, reliable, and secure payments network, enabling individuals, businesses, and economies to thrive while driven by a common purpose – to uplift everyone, everywhere by being the best way to pay and be paid.

Make an impact with a purpose-driven industry leader. Join us today and experience Life at Visa.

Job Description

At Visa, the Corporate Information Technology, Billing & Incentives Platforms team, enables Visa’s revenue growth through flexible pricing engines and global revenue platforms built on next-generation technologies. This includes managing system requirements, evaluating cutting-edge technologies, design, development, integration, implementation, and maintenance of corporate revenue applications. The team works closely with business owners of these services to deliver customer developed solutions, as well as implement industry leading packaged software. This team has embarked on a major transformational journey to build and implement best of breed revenue and billing applications to transform our business as well as technology.  

This position is for a candidate with solid background in development and production, who will be able to navigate both data engineering and data science disciplines to build effective engineering solutions which will improve the current framework and help to speed up the processes. This position is ideal for an engineer who is passionate about solving challenging business problems and building applications that provide excellent user experience.

The candidate will be extensively involved in hands-on activities including POCs, design, documentation, development, and testing of new functionality. Candidate must be flexible and willing to switch tasks based on team’s needs. 

Key Responsibilities

• Responsible for the design, development, and implementation

• Work on development of new products iteratively by building quick POCs and converting ideas into real products.

• Work closely with architects, business, and technical stakeholders to develop high quality products and services that meet business requirements and expectations while applying the latest available tools and technology.

• Develop code to ensure deliverables are on time, within budget, and with good code quality.

• Have a passion for delivering zero defect code and be responsible for ensuring the team’s deliverables meet or exceed the prescribed defect SLA.

• Coordinate Continuous Integration activities, testing automation frameworks, and other related items in addition to contributing core product code.

• Present technical solutions, capabilities, considerations, and features in business terms. Effectively communicate status, issues, and risks in a precise and timely manner

• Identify opportunities for further enhancements and refinements to standards and processes.

• Mentor junior team members, develop departmental procedures and best practices standards.

• With active engagement, collaboration, effective communication, quality, integrity, and reliable delivery, develop and maintain a trusted and valued relationship with the team, customers, and business partners.

This is a hybrid position. Hybrid employees can alternate time between both remote and office. Employees in hybrid roles are expected to work from the office 2-3 set days a week (determined by leadership/site), with a general guidepost of being in the office 50% or more of the time based on business needs.

Qualifications

Basic Qualifications:

• 4+ years of relevant work experience with a Bachelor’s Degree or at least 2 years of work experience with an Advanced degree (e.g. Masters, MBA, JD, MD) or 0 years of work experience with a PhD, OR 7+ years of relevant work experience

Preferred Qualifications:

• 4 or more years of work experience with a Bachelor’s Degree or 2 or more years of relevant experience with an Advanced Degree (e.g. Masters, MBA, JD, MD)
• Min of 4 years of software development experience (with a concentration in data centric initiatives), with demonstrated expertise in leveraging standard development best practice methodologies
• Minimum of 4 years of hands-on experience in building large-scale applications or Framework using open-source technologies like Hadoop, Spark, Scala, Hive, Kafka
• Hands-on expertise in Oozie, Tuber
• Must have experience with Unix/Shell or Python scripting
• Experienced with highly distributed, scalable, concurrent, and low latency systems working with one or more of the following database technologies: MySQL, Oracle , Mongo
• Experienced with building SQL for extracting, aggregating, and processing big data Pipelines using Hadoop, EMR & NoSQL Databases.
• Hands on experience in Version control using SVN, Git, Bitbucket
• Experience in Automating data processing using Control-M, ESP Series 
• Experience in building Automation Framework is a plus
• Experience working in an Agile and Test-Driven Development environment
• Should be self- sufficient in Testing the Framework and Workflows as well
• Strong knowledge of API development is highly desired
• Strategic thinker and good business acumen to orient data engineering to the business needs of internal and external clients
• Demonstrated intellectual and analytical rigor, strong attention to detail, team oriented, energetic, collaborative, diplomatic, and flexible style
• Previous exposure to financial services is a plus

Additional Information

Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.