Data Engineer Job Responsibilities:
- Work with stakeholders to understand data sources and Data, Analytics and Reporting team strategy in supporting within our on-premises environment and enterprise AWS cloud solution
- Work closely with Data, Analytics and Reporting Data Management and Data Governance teams to ensure all industry standards and best practices are met
- Ensure metadata and data lineage is captured and compatible with enterprise metadata and data management tools and processes
- Run quality assurance and data integrity checks to ensure accurate reporting and data records
- Ensure ETL pipelines are produced with the highest quality standards, metadata and validated for completeness and accuracy
- Develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity.
- Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.
- Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
- Writes unit/integration tests, contributes to engineering wiki, and documents work.
- Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
- Defines company data assets (data models), spark, sparkSQL jobs to populate data models.
- Designs data integrations and data quality framework.
- Designs and evaluates open source and vendor tools for data lineage.
- Works closely with all business units and engineering teams to develop strategy for long term data platform architecture.
- Focusing on structured problem solving
- Phenomenal communication and business awareness
- Working with ETL tools, Querying languages, and data repositories
- Support of technical Data Management Solutions
- Provide support to the development and testing teams to resolve data issues
- Experience in database, storage, collection and aggregation models, techniques, and technologies – and how to apply them in business
- Working knowledge of source code control tool such as GIT
- Knowledge about file formats (e.g. XML, CSV, JSON), databases (e.g. Redshift, Oracle) and different type of connectivity is also very useful.
- Working experience with the following Cloud platforms is a plus: Amazon Web Services, Google Cloud Platform, Azure
- Working experience with data modeling, relational modeling, and dimensional modeling
- The interpersonal skills: You have a way of speaking that engages your audience and instills confidence and credibility. You know how to leverage communication tools and methodologies. You can build relationships internal and external team members, positioning yourself as a trusted advisor. You are always looking for ways to improve processes, and you always ensure your communications have been received and are clearly understood. Your commitment and focus influence those around you to do better.
Position: Data Engineer
Term: Fulltime Permanent
Industry: eServices – Civil Infrastructure
Location: GTA, ON
About you
Do you love making sure that information is accessible and easy to use? So do we.
You are a data engineer who knows how to find, store and present a range of information from
different sources so that everyone can access what they need quickly and simply, and use it
effectively.
About the job
As a Data Engineer, you know the importance of data to the business. You design and set up
solutions that bring together information from a variety of sources, to enable analysis and
decision-making.
Originally posted on Himalayas