An Approach Of Scaling Airflow To A Corporate Level

Sat 15 Jul 2017 by Tianlong Song Tags Big Data

The last post on Airflow provides step-by-step instructions on how to build an Airflow cluster from scratch. It could serve the development purpose well, but lacks critical features to work in prod, e.g., CI/CD compliance, resource monitoring, service recovery, and so on.

I have been leading the efforts ...

read more

A Guide On How To Build An Airflow Server/Cluster

Sun 23 Oct 2016 by Tianlong Song Tags Big Data

Airflow is an open-source platform to author, schedule and monitor workflows and data pipelines. When you have periodical jobs, which most likely involve various data transfer and/or show dependencies on each other, you should consider Airflow. This blog post briefly introduces Airflow, and provides the instructions to build an ...

read more