Data Engineer
Prefer a full account with a password? Sign Up to Apply
Job Description
• Design, develop, deploy and maintain reliable ETL/ELT batch & streaming data pipelines, ingest structured/semi-structured/unstructured data from multiple source systems (DB, API, log, Kafka etc.)
• Perform dimensional data modeling, build and iterate data warehouse / data lake, define table schema, partition strategy, data layers (ODS/DWD/DWS/ADS)51job
• Write & optimize complex SQL, tune query performance, reduce resource cost and improve pipeline stability and SLA
• Implement data quality rules, anomaly detection, monitoring & alerting, troubleshoot data lineage and data consistency issues
• Use workflow orchestration tools to schedule, manage and monitor data jobs (Airflow/Dagster)
• Collaborate with analysts, DS and business stakeholders to translate business requirements into data solutions, build reusable datasets and data APIs
• Participate in data governance: metadata management, data security, access control, documentation for schema and pipelines51job
• Evaluate and adopt cloud data technologies (Snowflake/BigQuery/Redshift), continuously optimize storage and compute cost
Requirements
Function: Information Technology
Experience Level: Associate