• Senior Data Lake Engineer

    ID
    2018-7268
    Job Locations
    US-CA-San Francisco
    Category
    Engineering
  • Overview

    This position requires knowledge and experience with Data Lake technologies, particularly in AMAZON WEB SERVICES such as EMR, SPARK, HADOOP/ HIVE and experience in Database, Data warehouse and Data Lake technologies both on premise and in the cloud. Requirements include strong background in analysis, software/database design, development, testing, and deployment. While strategic thinking and coordination are important elements of this role, this individual will be required to take a very hands-on role with requirements, design, development, testing, deliverable preparation and other related activities.

    Responsibilities

    • Work on end-to-end (design, development, maintenance and evolution) implementation of Data lake solutions.
    • Works as a liaison between business and technology groups while maintaining alignment with the Bank’s security and information technology policies.
    • Drive Data Lake EMR/SPARK/HIVE technical excellence and implementation of best practices.
    • Perform prioritized planning and execution of Data Lake AMAZON WEB SERVICES EMR/SPARK development and release schedules while iteratively defining & refining processes.
    • Manages Data Lake AMAZON WEB SERVICES EMR/SPARK development workflow including reviewing requests, dev releases to test and final production deployment packages.
    • Responsible for research, delivery, implementation and support of Data Lake technical solutions.
    • Implements Hive data catalog, new features and fixes bugs and maintains testing and automation where appropriate.
    • Defines and executes test cases for any production deployment for enterprise use cases

    Qualifications

    Experience & Education
    • Bachelor’s degree in Computer Science or related degree preferred
    • 5 + years of experience delivering enterprise solutions to financial services organizations as a hands-on engineer developing complex systems architecture
    • 5+ years of Data Lakes, Data Warehouses and related technology experience
    • AMAZON WEB SERVICES EMR/Hadoop/SPARK implementation experience a must
    • Experience working with variety of AMAZON WEB SERVICES technologies
    • Three or more years of professional experience as a hands-on developer.
    • Two or more years scripting in a Linux environment, Bash experience preferred
    • 2 or more years of experience with SQL, Spark (Scala), and Hadoop/Hive
    • Experience with AMAZON WEB SERVICES (S3, Lambda, EMR, and EC2)
    • Knowledge of encryption and hashing techniques preferred
    • Demonstrated experience with shell scripting
    • Knowledge of version control software
    • Knowledge of web technologies and protocols (HTTP, SSL, JavaScript)
    • Ability to diagnose and troubleshoot problems in a multi-tiered environment

    Professional Skills/Competency
    • Demonstrate an understanding of data lake and warehouse systems at enterprise scale.
    • Experience developing on the AMAZON WEB SERVICES platform - Should be able to install/configure software, start/stop services, troubleshoot issues
    • Experience problem solving and interacting with business and other technology partners in a corporate setting
    • Exercise communication, presentation and organizational skills
    • Work with governance models and guidelines in the data management area
    • Identify and solve for points of integration with other technology and tools.
    • Develop best practices, presentations, training materials or documentation on data lake use within enterprise
    • Train and develop expertise in the data lake

    Personal Skills/Competency
    • A positive, enthusiastic attitude and the ambition to tackle challenging issues
    • Looks toward the broadest possible view of an issue/ challenge; can easily pose future scenarios; can think globally about all aspects of the Bank; can discuss multiple considerations of an issue and forecast them into the future; understands how the Bank works, competes, serves clients, and generates shareholder value
    • Negotiates skillfully in tough situations with internal and external groups; can be both direct and diplomatic
    • Can quickly find common ground and solve problems for the good of all; is a team player and encourages collaboration

    Mental/Physical Requirement
    • The ability to learn and comprehend basic instructions; understand the meanings of words and respond effectively; and perform basic arithmetic accurately and quickly.
    • Vision must be sufficient to read data reports, manuals and computer screens.
    • Hearing must be sufficient to understand a conversation at a normal volume, including telephone calls and in person.
    • Speech must be coherent to clearly convey or exchange information, including the giving and receiving of assignments and/or directions.
    • Position involves sitting most of the time, but may involve walking or standing for brief periods of time.
    • Must be able to travel in a limited capacity.

    Options

    Sorry the Share function is not working properly at this moment. Please refresh the page and try again later.
    Share on your newsfeed