Senior Data Engineer (BIG DATA / HADOOP)
Visa, Foster City CA, US
Everywhere you want to be
Cybersecurity is at the beating heart of our culture. Our diligence and expertise is what makes us the undisputed leader in electronic payments. We’ve made it our priority to create a top-tier Security Architecture team, poised to defend us against any potential cyber threats. CyberSecurity Architecture team is looking for a Data Engineer with Hadoop admin experience. Candidate will be a part of the Visa Security Analytics Engineering team and involved with administering Hadoop-ecosystem and Data Engineering activities. The position will be based at Visa's headquarters in Foster City, California.
The successful candidate will be working on cutting-edge technologies to perform statistical profiling, inference, classification, clustering and predictive analysis. As a key member of the Data Engineering team, you will be working on large datasets of Security Data and implement Data mining, Data parsing contributing to data availability building machine learning models to help derive new insights to defend against cyber-attacks.
You will be working with world-class operation teams including Cyber Defense team and Data Science team to create awesome analytics for security and other business units. You'll need to have excellent communication and cross-group collaboration skills, be able to make forward progress despite ambiguous circumstances, be a self-starter, a quick learner of new technologies and have experience in Data engineering to maintain DataQuality and Integrity.
As a Data Engineer, you will be responsible to establish processes, automations, structures and big data systems based on business and technical requirements to channel multiple requirements, route appropriately and plan proper big data technology using combination of open source and vendor supported big data technologies, databases, and other applicable big data technologies as required.
- Interact and aid to provide data for other investigative teams within Visa on time sensitive, critical investigations.
- Work with cross functional teams to fully understand business requirements and desired business outcomes
- Assist in scoping and designing analytic data assets, implementing modeled attributes and contributing to brainstorming sessions
- Build and maintain a robust data engineering process to develop and implement self-serve data and tools for Visa’s data scientists
- Find opportunities to create, automate and scale repeatable analyses or build self-service tools for business users
- Execute data engineering projects ranging from small to large either individually or as part of a project team
- Experience on working with very large data sets and knowledge of building programs that leverage Massively Parallel Processing (MPP) Data warehouse platforms.
- The engineer will be working on Big Data technologies like Hadoop and tools and security related datasets doing Data parsing, Data mining activities and writing parsers.
- Responsible for loading data from several disparate datasets, documentation and debugging applications.
- 5 years of work experience with a Bachelor’s Degree or an Advanced Degree such as Computer Science. (e.g. Masters, MBA, JD, MD, or PhD).
- 4+ years of software development experience (with a concentration in data centric initiatives), with demonstrated expertise in leveraging standard development best practice methodologies.
- Experience with Data Engineering with production pipelines, utilizing data engineering techniques that enable statistical modeling solutions to solve business problems.
Experience with various Opensource Hadoop distribution systems and technology stack like Spark, oozie, airflow etc.
- Extensive experience with big data technologies (Hadoop, Hive, Druid, etc.) tools for large scale data processing, data transformation and machine learning pipelines.
- Ability to create and manage big data pipeline using Syslog-ng, kafka, flume, Airflow etc.
- Hands-on expertise with Java or Python, Spark and any scripting languages. Go lang and Scala is a plus.
- Experience with highly distributed, scalable, concurrent and low latency systems working with one or more of the following database technologies: MySQL, Postgres SQL, NoSQL DB, Elastic Search.
- Experience with data visualization and business intelligence tools like Power BI, Tableau, Microstrategy, or other programs.
- Experience in developing Rest API.
- Experience with Continuous Integration and Automated Test tools such as Jenkins, Artifactory, Git, Ansible.
- Experience working in an Agile and Test-Driven Development environment.
- Demonstrated intellectual and analytical rigor, strong attention to detail, team oriented, energetic, collaborative, diplomatic, and flexible style.
- Previous exposure to financial services, credit cards or Security analytics is a plus, but not required.
This position requires the incumbent to travel for work “0-5%” of the time
This position will be performed in an office setting. The position will require the incumbent to sit and stand at a desk, communicate in person and by telephone, frequently operate standard office equipment, such as telephones and computers, reach with hands and arms, and bend or lift up to 25 pounds.
Visa will consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.
Visa Inc. (NYSE: V) is the world’s leader in digital payments. Our mission is to connect the world through the most innovative, reliable and secure payment network - enabling individuals, businesses and economies to thrive. Our advanced global processing network, VisaNet, provides secure and reliable payments around the world, and is capable of handling more than 65,000 transaction messages a second. The company’s relentless focus on innovation is a catalyst for the rapid growth of connected commerce on any device, and a driving force behind the dream of a cashless future for everyone, everywhere. As the world moves from analog to digital, Visa is applying our brand, products, people, network and scale to reshape the future of commerce.
Want to learn more about Visa? Visit Visa's website.
Lyft is your friend with a car, whenever you need one