Senior Data Engineer

You will be a member of the Data Group: Data Analytics, Data Science, Machine Learning and Data Engineering.  We are a force multiplier, owning the data, analysis, and knowledge infrastructure that enables ourselves and our teammates to move faster and smarter.

The Data Engineering team’s mission is to ingest data, build ELT pipelines and create services and tools for others to use data more efficiently. You will work on developing and enhancing our data warehouse,  defining processes for data monitoring and alerting as well as maintaining data integrity in our data ecosystem. You will work with cross functional teams and internal stakeholders to define requirements and build solutions to meet the requirements. You will work with other engineers to ensure that our data platform and infrastructure are scalable and reliable.

Responsibilities:

• Work on high impact projects that improve data availability and quality, and provide reliable access to data for the rest of the business
• Design, architect and support new and existing ELT pipelines, using tools like Snowflake, Fivetran, dbt and the AWS stack
• Assemble large, complex data sets that meet functional and non-functional business requirements
• Be responsible for ingesting data into Snowflake data lake and warehouse and providing tools and frameworks for operating on that data
• Analyze, debug and correct issues with data pipelines
• Communicate strategies and processes around data modeling and architecture to the data engineering as well as other teams
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, loading, and transformation of data from a wide variety of data sources using SQL, Fivetran, dbt, Python and AWS technologies
• Build a reliable and scalable metrics and monitoring system to support proactive handling and communication of issues with all aspects of our data pipelines
• Working with both Structured and semi-structured data (JSON and XML) raw data

Requirements:

• 5 years of experience, which includes at least 3 years of experience implementing complex ELT pipelines preferably in connection with Snowflake and extract and load tools like Fivetran or Stitch along with transformation tools like dbt
• Experience writing complex SQL and ELT processes
• Exceptional coding and design skills, particularly in Python and SQL
• Worked with large data volumes, including processing, transforming and transporting large-scale data
• Have hands-on experience with AWS and services like EC2, SQS, Lambda, S3, etc.
• Experience with DAG-based orchestration tools like Airflow
• Experience with infrastructure-as-code tools, especially Terraform
• Have a strong understanding and usage of algorithms and data structures
• Strong Bash shell scripting and Linux command-line experience