Embedded Files
#FinTech #BigData #QuantitativeTrading
Programming language: Python, Java, Scala (base on Java)
Engineering technology: Hadoop、Spark、K8S、AWS、GCP
- Work Experience -
- Cathay Financial Holdings Digital Data Development Center,
- Data Science R&D Department,
- Data Science Engineer, Feb 2023 ~ Now
Developed an end-to-end data service leveraging data migration for all company.
Developed database interface plugins for MongoDB, Oracle, and AWS S3, leveraging Apache Spark for distributed data reading, transformation, and transmission. Transformed single-stream data transmission into a distributed ETL process, improving data transfer efficiency by over 60%.
Developed validation features for data types, numerical and categorical data, record counts, unique values, and time ranges, reducing the data error rate in the ETL process by over 80%.
Developed ETL pipeline allowing users to add, modify, delete, and convert data during migration.
Designed and developed RESTful APIs to retrieve unstructured data from AWS S3, addressing storage limitations in MongoDB and Oracle.
Implemented proxy redirection functionality to facilitate cross-domain data transmission, resolving internal and external network connectivity issues. Reduced data landing occurrences, increasing ETL efficiency by 40%.
Implemented Kubernetes (K8s) automated testing and a GitLab CI/CD environment to accelerate developer testing and improve efficiency. Built a scalable testing environment, reducing cloud costs by 20%.
Built an end-to-end AI Sepsis Detection service, currently serving at Cathay General Hospital.
Developed a data preprocessing pipeline to handle missing values, outliers, and differentiate between discrete and continuous data, and employed Stratified Shuffle Split to augment data.
Developed an XGBoost model that reduced diagnostic errors by 33% compared to human diagnosis.
Implement feature engineering to identify key features, enhancing model recall and precision.
Deployed the XGBoost model on the hospital’s server using Kubernetes (K8s) for operation.
Developed the CaFe federated learning framework to address data silo issues by integrating financial data from various institutions, reducing fraudulent transaction flows by 20%.
Deployed the Hadoop ecosystem (Spark, Yarn, HDFS, etc.) in a K8S environment to provide data storage, resource management, and migration functionalities.
Built an end-to-end testing process (smoke, black-box, stress, stability tests) to improve product quality and architecture.
Built a biomimetic internal testing environment to enable developers to test products.
Set up a big data Hadoop environment using AWS EMR, RDS, and EC2.
Built OKD/CRC testing environments on GCP.
- National Chung-Shan Institute of Science and Technology,
- Intern, July 2020 ~ July 2021
Developed AI for unmanned combat aircraft to provide simulated enemy training for pilots.
Developed AI for unmanned combat aircraft to provide simulated enemy training for pilots.
Created data preprocessing functions to handle missing values, outliers, and distinguish between discrete and continuous data.
Trained unmanned combat aircraft models using techniques such as LSTM, neural networks, and reinforcement learning
Built a data visualization website using JavaScript and PHP to improve research efficiency and expedite feature collection
- Chaoyang University of Technology
- Information Service Center
- Web Engineer, Aug 2017 ~ Nov 2018
Developed the Chaoyang Electronic Newsletter website, handling user login, message boards, and admin portal functionalities.
Built the Chaoyang USR project website using Django, incorporating JavaScript, Ajax, and PostgreSQL
- Side Project -
Web3 Risk Hedging System (2022/01/30)
Web3 Risk Hedging System (2022/01/30)
An automated futures-spot arbitrage system integrating DeFi and CeFi.
Interacts with on-chain smart contracts through web3.py and periodically monitors the status of on-chain assets and profits; monitors market fluctuations and trading in real-time through exchange Restful APIs, and records real-time and expected profits.
USDT Exchange Rate Grid System (2022/06/22)
USDT Exchange Rate Grid System (2022/06/22)
Integrates with Taiwanese exchanges for automated trading of exchange rates.
Utilizes grid trading strategies for automated low buy and high sell, integrating with the MAX exchange API to offer proportional or arithmetic trading methods. Incorporates tkinter for a user interface to easily monitor profits and order statuses.
(This functionality was independently developed and provided by MAX on 2023/08/08.)
Cryptocurrency Arbitrage System (2023/11/13)
Cryptocurrency Arbitrage System (2023/11/13)
Monitors price differences across exchanges and performs arbitrage trading.
Developed in Python with version control via GitHub, integrating APIs from Taiwan's three major exchanges, and using async for parallel processing to enhance arbitrage efficiency. Regularly monitors profits and sends profit reports and market price fluctuations via email. Ultimately deployed on AWS EC2 for real-time automated operation.
Automated Trendline Trading System (2022/01/17)
Automated Trendline Trading System (2022/01/17)
Integrated an automated trendline strategy trading system with Binance.
Utilized a trendline strategy to automate entry and exit points for buy and sell orders. Connected to the Binance API to batch filter all tradable cryptocurrencies, enabling 24/7 automated trading. The system notifies users of entry and exit points and updates them on profit/loss status for each trade.
Exchange Rate Rebalancing System (2022/07/13)
Exchange Rate Rebalancing System (2022/07/13)
Integrates with Taiwanese exchanges for automated rebalancing of exchange rates.
Employs rebalancing to automatically equalize asset values while earning rewards on demand deposits, integrating with the MAX exchange API. Utilizes tkinter for a user interface to facilitate the observation of profits and coin holding statuses.
Oh!DogCat Stray Animal Shelter Platform (2019/11/24)
Oh!DogCat Stray Animal Shelter Platform (2019/11/24)
Visualize and view stray animal data across Taiwan / LINE Bot for instant reporting
Web：Developed using JavaScript and PHP, regularly scraping opendata to the Firebase database for data cleaning and processing, and finally hosting the visualized website on the Herokuapp platform for users to browse.
(Due to the expiration of Herokuapp's free period, the website is no longer accessible, you can watch the youtube video )
Line bot：Developed using Google Script to integrate with LINE Message for creating a chatbot, providing functions such as stray animal reporting, lost pet reporting, nearby animal hospital notifications, and stray animal sheltering. Precision push notifications were added in the lost report to improve the recovery effect, and all uploaded data were linked with the Imgur API as an image database. (It's still available)
- Education -
- Chaoyang University of Technology
- Master's Degree in Information Management, 2020 ~ 2022
Thesis: A Low-Risk, High-Profit Cryptocurrency Arbitrage System Based on Hedging
Independently completed an AI decision-making screening system.
Independently developed an automated quantitative trading system: Trend Quantitative Grid and Pattern Grid.
- Chaoyang University of Technology
- Bachelor's Degree in Information Management, 2016 ~ 2020
Developed the Oh!DogCat Stray Animal Shelter Platform, responsible for backend programming, data analysis, and database development.
Developed the Oh!DogCat Stray Animal LINE Official Chatbot, handling data processing, backend programming, and database development.
Independently designed and developed a web game, completing the planning, story, scenes, prototype, implementation, and testing within one month.
Developed a Stray Animal Shelter App using Swift.
Participated in the organization team for the 23rd Competitive Cheerleading Squad, completing planning, fundraising, theme selection, personnel distribution, and formations within four months and achieving victory.
Independently developed an online stock quote system using Jupyter Notebook.
- Zhongshan Industrial and Commercial School
- Department of Multimedia Technology, 2012 ~ 2016
Ranked 10th in the National Business Skills Competition in Programming in 2015.
First place in the Kaohsiung Digital Programming Contest.
Leader of the Elite Society, guiding junior and senior high school students in activities and explaining the content.
Executive Director of the Alumni Association Activities, planning graduation events, ceremonies, and yearbook design and production.
Holds a certificate in Software Application (Level 2), Accounting Information (Level 3), and Web Design (Level 3).
Thank you for reading this and have a nice day.
I am jui-yuan Liu , an explorer focusing on #data #finance #business
Page updated
Google Sites
Report abuse