How Apache Airflow Better Manages Machine Learning Pipelines

2 years ago thenewstack.io

Summary: This is a summary of an article originally published by The New Stack. Read the full original article here →

VANCOUVER — What is apparent with https://airflow.apache.org/, the open-source project for building pipelines in machine learning? The experience is getting even easier, as illustrated in a discussion on The New Stack Makers with three technologists from https://aws.amazon.com/?utm_content=inline-mention

It is well-suited to https://thenewstack.io/ for building pipelines, managing data, training models, and deploying them. Airflow is generic enough for the whole pipeline in machine learning.

A recent project included writing and implementing https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow (Airflow Improvement Proposal), which modifies and updates the Executor interface in Airflow.

DevOps Articles