Chih Han Yu

Global Data Warehouse Automated Ingestion Platform (Sep. 2019 to Current)

• Web platform can automatically deploy batch "EL" pipelines on Apache NiFi, ingest data from RDBMS/File to GCP BigQuery, Snowflake and Hive.
• Performed both backend services (Java) and frontend development (Angular) for features to support multiple sources and target types.
• Managed ~9000 ETLs built by this platform to Snowflake/GCP/Hive which processes over than terabytes of data per day across global facilities.
• Designed with detailed SA/SD documentation, CICD, solid code review, version control and change management process.
• Performed training and demo documents and sessions for better user coworking and communication.
• Additional Roles: Scrum Master / Operation Chief / Team Coach

Global Data Warehouse Migration - GCP Migration / Hadoop Retirement (Dec. 2020 to Current)

• Designed data landing structure and peformed POC to adapt new platform (GCP) for GDW Automated Ingestion Platform.
• Redesigned ETLs for Manufacturing Photo image calculation with Spark in Scala, Cloud Composer and Cloud DataProc.

Global Data Warehouse Migration - Teradata Retirement (Mar. 2019 to Aug. 2019)

• Performed existing ETLs migration from Teradata to Snowflake with no impact.

Global Manufacturing SCADA/IoT Data Ingestion Project (Dec. 2017 to Aug. 2019)

• Designed real-time ETLs on Apache NiFi for handling large volume of sensors data to Hadoop Hive.
• Performed requirements clarification, ETL refactor, performance tuning and operation maintenance.

Gaming Log Collection Platform

• Designed and built-up ELK Stack (Elasticsearch / Logstash / Kibana) environment within 1 month.
• Designed and build-up log collecting system from RDBMS to Elasticsearch by Logstash.
• Open Source Project Contribution: Apache Spark (Bugfix), Elastalert (ElasticSearch 3rd Party Module Bugfix)

Gaming Log Analysis Platform

• Designed and developed ETLs to process, transfer and clean data with Python.
• Built-up data visualization dashboard with Grafana and Kibana for developer team and company business customer.

Manufacturing Big Data Platform Project

• Designed and built-up Big Data Platform POC environment in on-prem machine alone within 2 weeks, including Hadoop ecosystem and SMACK ecosystem (Spark, Mesos, Akka, Cassandra, Kafka).
• Designed and developed ETL for manufacturing data preprocess and analysis by Spark Streaming, Kafka and Cassandra.

IoT Data Acquisition Project

• Developed Web RESTful API with Spring Boot and Cassandra.

300mm Fab In-line Quality Control MES System Administrator

• Managed Quality Control systems, including server, databases, ETLs and web application.
• User requirements coordinator between Fab users and system developers.
• Executed fast troubleshooting, server support, fast recovery in high pressure environment urgently.
• Maintained service release schedule for MES related system.
• Performed scheduled MES system work, testing and repairs.
• Offered automation tools for fast recovery archiving data to support Customer complaint event, processing speed is 24 times faster than before.