Data Engineer

We are hiring a hands-on Data Engineer to join our India-based team and help build the next generation of our data delivery platform. Working closely with our senior and principal engineers and with engineering leadership across North America and India, you will design, build, and operate the pipelines that move and transform millions of data points every day.

We are deliberately moving away from a locked-in, vendor-heavy stack toward a flexible, largely open-source architecture that keeps our options open. You will be in the engine room of that re-architecture — writing code, designing schemas, tuning queries, and helping prove out new tools before we adopt them at scale.

This is a build role. You bring strong hands-on data pipeline experience and the judgment to use the right open-source tool for the job, while taking technical direction on the broader platform strategy from our senior engineers. We expect you to get into the details and understand the “how” and “why” behind every pipeline.

What You’ll Do

• Build and operate pipelines: Design, build, and optimize robust ETL/ELT pipelines that move and transform millions of daily data points reliably and efficiently.
• Work hands-on with the open-source stack: Build and maintain production data workflows using tools such as dbt, ClickHouse, and open-source orchestration frameworks (Airflow, Dagster, or similar).
• Engineer and tune databases: Write and optimize complex SQL across SQL Server and Postgres — schema design, indexing, performance tuning, query optimization, and root-cause analysis.
• Support platform modernization: Contribute to the hands-on migration away from Azure Databricks toward a more open, flexible, AWS-based stack, minimizing disruption to high-volume daily data delivery.
• Prototype and evaluate: Help build proofs-of-concept to benchmark new tools and patterns, and feed clear results back into the team’s adoption decisions.
• Work with AI-assisted tooling: Use AI coding assistants and well-crafted prompts (e.g., GitHub Copilot, Claude, ChatGPT) to accelerate pipeline development, SQL generation, debugging, testing, and documentation — always reviewing and validating the output before it ships.
• Safeguard data quality and reliability: Implement testing, validation, monitoring, observability, and CI/CD practices so data stays accurate and pipelines stay healthy at scale.
• Integrate across systems: Understand the systems upstream and downstream of your pipelines, from ingestion through to client-facing platforms, to ensure clean, end-to-end data delivery.
• Collaborate across a distributed team: Partner with engineers in North America and India, participate in code reviews, and learn the nuances of the Advertising and Market Research domain.

Requirements

• Experience: 7–8+ years in data engineering / ETL, with a strong track record as a hands-on engineer building and operating production data pipelines.
• Core databases: Strong SQL and RDBMS skills with solid, hands-on experience in SQL Server and Postgres (schema design, performance tuning, complex query optimization).
• Open-source and modern stack: Hands-on experience with tools such as dbt, ClickHouse, and open-source pipeline / orchestration tools (e.g., Airflow, Dagster), with the judgment to choose the right tool for the job.
• Programming: Strong proficiency in Python (or a similar language) for building and automating data pipelines.
• AI-assisted development: Hands-on experience using AI coding assistants and effective prompting techniques to work more efficiently, with the judgment to verify, test, and refine AI-generated code and queries.
• Cloud: Hands-on experience with a major cloud platform; AWS strongly preferred, as we are standardizing on AWS as we move off Azure Databricks.
• Engineering fundamentals: Comfortable with Git, code reviews, and writing tested, maintainable, well-documented code.
• Education: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Bonus Points (Nice to Haves)

• Data warehousing: Experience with cloud data warehouses or lakehouse patterns (Snowflake, BigQuery, Redshift, or similar).
• Streaming and real-time: Experience with streaming / real-time data technologies such as Kafka.
• Infrastructure: Familiarity with containerization (Docker, Kubernetes) and infrastructure-as-code (e.g., Terraform).
• Domain expertise: Previous experience in Advertising, Media, or Market Research.

Originally posted on Himalayas