Senior Data Engineer​

We are looking for a Senior Data Engineer to join a team to work on data pipeline development. You would work closely with small, highly collaborative Operations and Development teams to architect and deploy high-impact security solutions.

You must be able to work US time zones (UTC-8 to UTC-5/UTC-7 to UTC-4).

Job Responsibilities

• Build data management pipelines from heterogeneous data sources into Data Marts and other repositories for subsequent analysis and data mining
• Code, test, and support Python-based services, implementing ETL/ELT, data cleansing, etc.
• Integrate with internal (on-prem) and external services utilizing various approaches like RESTful APIs, DMS, distributed event streaming etc., develop shippable code, documentation, and unit test new features
• Collaborate with Quality, Product, and other Engineering teams
• Provide code reviews, design feedback, demos, technical requirements & documentation
• Scope projects, provide accurate estimates for reliable delivery of projects
• Facilitate discovery sessions to map out business processes.
• Collaborate with data architects, business users and create comprehensive functional specifications, and process flow diagrams that bridge the gap between business needs and data engineering implementation.
• Independently explore backend systems and database schemas to determine the feasibility of business requirements.
• Trace data lineage to understand how a metric is calculated in the source system versus how the business defines it, resolving discrepancies before they reach the dashboard

Key Applicant Requirements

• Professional development experience using Python, including an understanding of functional programming: 5+ years
• Cloud infrastructure experience in AWS, Docker: 3+ years
• Experience in Distributed event streaming and process orchestration and with modern data stack.
• Infrastructure As Code (Terraform) experience: 1+ years
• Strong proficiency in Python and SQL.
• Proficient with relational databases such as Oracle, MySQL, PostgreSQL
• Experience with AWS services such as RDS, Redshift, S3, EMR, MWAA (Airflow) and MSK, ECS, SNS, SQS etc.
• Experience with designing and building Data Marts and ETL/ELT
• Proficient in data profiling and analysis techniques utilizing SQL and/or Python.
• Proficient with Jira, Confluence, and GIT for version control.
• Hands-on experience with Agile/Scrum/Kanban
• Excellent written & verbal communication skills, working asynchronously and independently in a remote environment.
• Ability to coordinate between teams.
• Data visualization dashboarding knowledge/experience.

Pluses

• Design, implementation, and/or maintenance of CICD, Bash scripts
• Experience with using AWS EMR Serverless

Interview Process
We will assist you with preparation, including mock interviews, and coaching to succeed! The steps typically are:

• Prescreen with recruiters
• Technical interview with development team

Originally posted on Himalayas