![]() ![]() ![]() ![]() ![]() Either the networking specifics go over the Data engineer's head and/or the data pipeline IAM permissions and DAG idempotency go over the DevOps engineer's head. Responsibilities overlap and both roles are traditionally ill-equipped to come to consensus. As I got more hands-on with infrastructure/networking, I was performing two jobs: Data and DevOps engineer. Too often, I've done all this work in my local desktop airflow environment only to find out the DAGs don't work in a Kubernetes deployment or vice versa. It is a painful exercise to setup secure airflow environments with parity(local desktop, dev, qa, prod). SAME AIRFLOW DATA PIPELINES | WHEREVER YOU RUN THEM airflow-toolkit □Īny Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND a Google Cloud Composer Airflow environment with working example DAGs across both ✨ Motivations ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |