About the Role
As a mid-level data engineer (DE II) you will responsible for designing, building, and maintaining the infrastructure required to collect, process, and store large amounts of data. You will collaborate with data scientists, business intelligence, and other stakeholders to ensure data is correct, accessible, reliable, and optimized for performance.
Responsibilities:
Produce reliable data for stakeholders by reviewing collected data APIs, troubleshooting, and processing data into the datamart.
Propose and collect requirements from users for data pipeline creation, sourcing data based on needs, and establishing data pipelines, cataloging, and formatting.
Execute the creation of machine learning pipelines, gathering user requirements, sourcing data, developing pipelines, and cataloging destinations for formatted data.
Develop APIs by collecting requirements, sourcing relevant data, creating pipelines, and cataloging and formatting data accordingly.
Improve structured planning for model deployment through research, testing, and documenting technical specifications (TSD).
Maintain discussions with stakeholders, provide documentation, and collect feedback.
Optimize and implement integrations effectively based on stakeholder feedback and testing results.
Design, build, and manage scalable data pipelines and ETL processes.
Develop and maintain the data infrastructure (e.g., data warehouses, data lakes).
Monitor and maintain data pipelines to ensure real-time and batch data processing.
Requirements
At least have 3+ years experience in the same position
Strong experience with SQL and relational databases (e.g., MySQL, PostgreSQL).
Experience with ETL tools and data pipeline frameworks (e.g., Apache Airflow, AWS Glue).
Proficiency in a programming language such as Python, Java, or Scala.
Experience with cloud platforms (AWS, Azure, GCP) and related data services.
Experience with version control (e.g., Git) and CI/CD pipelines.
Strong problem-solving and troubleshooting skills.
Familiarity with data governance, security, and privacy principles.