Role Summary
We're hiring experienced Data Engineers to design, develop, and maintain scalable, high-performance data pipelines and enterprise data warehouse (EDW) solutions for our various clientele projects. You will work closely with Solution Architects, Data Analysts, Product Owners, and QA teams in an Agile delivery environment to support business intelligence and data integration needs.
Key Responsibilities
Data Solution Design & Development:
- Develop and maintain ETL/ELT code using Teradata SQL, Informatica Power Center, Apache Spark, QueryGrid, and Trino.
- Design and build data pipelines and orchestration using enterprise scheduling tools (e.g., Control-M, Autosys).
- Create automation frameworks using shell scripts, BTEQ, and GCFR for end-to-end data workflows.
- Build and optimize frameworks for Change Data Capture (CDC) using Java, integrating SAP (Oracle) data into Teradata.
- Develop and maintain Master Data Management (MDM) applications for data uploads, validation, and approval workflows.
Performance Optimization & Automation:
- Optimize and tune high-complexity data applications to ensure efficient resource usage.
- Integrate ETL/ELT frameworks with control and monitoring solutions for robust automation.
Integration & Data Management:
- Develop and support data integration processes between various enterprise systems and the data warehouse.
- Implement and maintain reference data management and data quality controls.
Agile Delivery & Collaboration:
- Participate in Agile sprint activities, including planning, development, testing, and deployment.
- Collaborate with cross-functional teams to translate business requirements into technical solutions.
- Document technical designs, deployment procedures, and operational playbooks.
Testing, Deployment & Support:
- Perform unit testing, support system integration testing, and resolve defects.
- Support deployment activities across development, SIT, UAT, staging, and production environments.
- Troubleshoot and resolve issues to ensure data platform stability and reliability.
Mandatory Skills & Experience
- Degree in Computer Science, Information Technology, Engineering, or related discipline.
- Minimum 3 years of experience in data engineering, ETL/ELT development, and data pipeline orchestration.
- Strong hands-on experience with:
- Teradata SQL, Informatica Power Center, Apache Spark, QueryGrid, Trino
- Shell scripting, BTEQ, GCFR
- Enterprise scheduling/orchestration tools (e.g., Control-M, Autosys)
- Java for building CDC frameworks
- Deep understanding of data integration, data warehousing, and MDM concepts.
- Experience with Hadoop ecosystem tools (Hive, Impala, Spark, Kafka, Iceberg, Ranger, Atlas, NiFi, Flink).
- Familiarity with Agile delivery methodologies and tools (JIRA, Confluence).
- Experience with version control and CI/CD pipelines.
- Strong analytical, problem-solving, and collaboration skills.
Preferred/Advantageous Skills & Experience
- Experience with SAP (Oracle) data integration and OLR log processing.
- Exposure to large-scale enterprise data platforms or digital transformation projects.
- Experience integrating with monitoring and control systems.
- Certification in relevant data engineering or cloud technologies.
- Strong sense of ownership and continuous learning mindset.
Technical Stack / Domain Knowledge
- ETL/ELT Development: Teradata SQL, Informatica Power Center, Apache Spark, QueryGrid, Trino
- Data Pipeline & Orchestration: Control-M, Autosys, Shell scripting, BTEQ, GCFR
- Integration & CDC: Java, SAP/Oracle OLR logs, Custom CDC frameworks
- Platform Expertise: Teradata, Hadoop ecosystem (Hive, Impala, Spark, Kafka, Iceberg, Ranger, Atlas, NiFi, Flink)
- MDM & Data Quality: Reference data management, validation, workflow automation
- Agile Delivery: JIRA, Confluence, Sprint activities
- Version Control & DevOps: Git, CI/CD pipelines, deployment automation
- Testing & Support: Unit testing, integration testing, defect resolution, platform troubleshooting
- Documentation: Technical design, deployment procedures, operational playbooks