Senior Data Engineer

We are seeking a Senior Data Engineer to work directly with agencies, enabling them to reach their desired development delivery efficiency and system resiliency through optimized Continuous Integration/Continuous Deployment(CI/CD) and Site Reliability Engineering(SRE) practices.

Asa forward-deployed engineer, you’ll partner with agencies to solve complex challenges, gather insights, and drive continuous improvement in our product offerings. This role combines deep technical expertise with a commitment to customer enablement and feedback, ensuring that our Core Engineering Products evolve to address the real-world needs of our customers.

As a Senior Data Engineer, youwill be working on:

·      Translate data requirements from business users into technical specifications.

·      Collaborate with partner agency’s IT teams on technology stack, infrastructure and security alignment.

·      Build out data product as part of a data team:

·      Architect and build ingestion pipelines to collect, clean, merge, and harmonize data from different source systems.

·      Day-to-day monitoring of databases and ETL systems, e.g., database capacity planning and maintenance, monitoring, and performance tuning; diagnose issues and deploy measures to prevent recurrence; ensure maximum database uptime;

·      Construct, test, and update useful and reusable data models based on data needs of end users.

·      Design and build secure mechanisms for end users and systems to access data in data warehouse.

·      Research, propose and develop new technologies and processes to improve agency data infrastructure.

·      Collaborate with data stewards to establish and enforce data governance policies, best practices and procedures.

·      Maintain data catalogue to document data assets, meta data and lineage.

·      Implement data quality checks and validation processes to ensure data accuracy and consistency.

·       Implement and enforce data security best practices, including access control, encryption, and data masking, to safeguard sensitive data.

What we are looking for:

·      A Bachelor’s Degree, preferably in Computer Science, Software Engineering, Information Technology, or related disciplines.

·      Deep understanding of system design, data structure and algorithms, data modelling, data access, and data storage.

·      Demonstrated ability in using cloud technologies such as AWS, Azure, and Google Cloud.

·      Experience with Databricks.

·      Experience in designing, building, and maintaining batch and real-time data pipelines.

·      Experience with orchestration frameworks such as Airflow, Azure Data Factory.

·      Proficiency in working with Python, Shell Scripts, and SQL.

Preferred requirements:

·      Familiarity with building and using CI/CD pipelines.

·      Familiarity with DevOps tools such as Docker, Git, Terraform.

·      Experience with implementing technical processes to enforce data security, data quality, and data governance.

·      Familiarity with government systems and government's policies relating to data governance, data management, data infrastructure, and data security.

·      Experience in Climate and Weather domains will be an advantage.