Install Python dependencies to docker-compose cluster without re-build images

  1. Python python .pth file
  2. Docker volume
  3. and pip with flag:
pip install --target=

Test DAG

from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator

from slack import *

with DAG(dag_id='dependency_dag',
start_date=datetime(2019, 12, 17),
schedule_interval=None) as dag:
DummyOperator(task_id='dummy')
import error in Airflow UI

Python python .pth file

  1. understand where leave site-packages directory that used by our Python interpret (that run our Apache Airflow (or your server) in docker container)
  2. decide in that folder we will install our packages and define it as volume in docker-compose .yml
  3. put inside container — .pth file with our path to packages folder: add to Dockerfile 2 lines: 1st create dir for packages inside container, 2nd copy .pth file
/usr/local/airflow/packages

site-packages folder

  1. define in DAG simple python code like:
import sys
print(sys.path)
site-packages in Apache Airflow container
docker exec -it #container_id /bin/bash
/usr/local/lib/python3.7/site-packages

Docker volume

- ./airflow_files/packages:/usr/local/airflow/packages
volumes:
- ./airflow_files/dags:/usr/local/airflow/dags
- ./airflow_files/logs:/usr/local/airflow/logs
- ./airflow_files/packages:/usr/local/airflow/packages

Dockerfile

RUN mkdir /usr/local/airflow/packages
COPY ./packages.pth /usr/local/lib/python3.7/site-packages
FROM puckel/docker-airflow:1.10.6
RUN pip install --user psycopg2-binary
ENV AIRFLOW_HOME=/usr/local/airflow
RUN mkdir /usr/local/airflow/packages
COPY ./packages.pth /usr/local/lib/python3.7/site-packages
COPY ./airflow.cfg /usr/local/airflow/airflow.cfg

.dockerignore

airflow_files/
data/
docs/
LICENSE
README.md
airflow/packages added to site-packages

pip install with ‘target’ flag

pip install --target=./airflow_files/packages slackclient
pip install with target flag
packages installed in target folder
no import error UI

--

--

--

Python Dev \\ https://github.com/xnuinside \\ https://twitter.com/xnuinside. Currently open to remote https://www.linkedin.com/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Cloud Computing Applications

Zero Operations vision

Improving GCE boot times with custom images

Horus Community Update #1

Next higher palindrome using the same set of digits

Write your own tools: python + slack webhook + crontab

How to maintain development environments on ECS cost-effectively

What's the Difference Between the Front-End and Back-End?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Iuliia Volkova

Iuliia Volkova

Python Dev \\ https://github.com/xnuinside \\ https://twitter.com/xnuinside. Currently open to remote https://www.linkedin.com/

More from Medium

Simplify python asynchronous API testing

How to run PostgreSql using Docker and Python

Bulk Boto3 (bulkboto3): Python package for fast and parallel transferring a bulk of files to S3…

Work locally with Azure Machine Learning from docker images