Airflow | Airflow 설치, Airflow docker compose로 설치하기

 

Airflow 설치 방법에는 크게 두가지가 있다.

1. docker compose로 쉽고 간단하게 설치하기

2. 가상환경 내에 airflow 설치하여 환경 셋팅하기

 

우선 간단한 docker compose로 설치하는 방법부터 정리하려고 한다.

 

 

1. docker install

Requirements

• Linux (Ubuntu 20.04 +)

Install Docker using the apt

Install Docker Engine on Ubuntu

• 참고 링크

[Docker, Ubuntu 22.04] 도커 설치부터 Nvidia-toolkit까지

docker install using apt

 

Update the apt package index.

sudo apt-get update

 

install packages to allow apt to use a repository over HTTPS.

sudo apt-get install ca-certificates curl gnupg

 

Add Docker’s official GPG key.

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

 

Use the following command to set up the repository.

echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

 

Update the apt package index.

sudo apt-get update

 

Install Docker Engine, containerd, and Docker Compose.

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

- Install latest version

apt-cache madison docker-ce | awk '{ print $3 }'

VERSION_STRING=5:20.10.13~3-0~ubuntu-jammy
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

- Install specific version

 

Verify that the Docker Engine installation is successful by running the hello-world image.

sudo systemctl status docker
sudo docker run hello-world

 

Manage Docker as a non-root user

sudo groupadd docker

sudo usermod -a -G docker $USER

newgrp docker

sudo reboot

docker run hello-world

 

 

docker with GPU

rerequisites

• NVIDIA Graphic Driver

Setup the package repository and the GPG key

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

 

Install the nvidia-container-toolkit package (and dependencies) after updating the package listing.

sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit-base nvidia-container-toolkit

 

Configure the Docker daemon to recognize the NVIDIA Container Runtime.

sudo nvidia-ctk runtime configure --runtime=docker

 

Restart the Docker daemon to complete the installation after setting the default runtime

sudo systemctl restart docker

 

At this point, a working setup can be tested by running a base CUDA container

sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

 

2. docker compose(2.14.0 ver 이상) install

sudo curl -L "https://github.com/docker/compose/releases/download/v2.4.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

docker-compose -v      # 버전이 출력되면 제대로 설치 된 것

 

3. airflow를 설치 할 폴더 생성 후 해당 폴더로 이동

mkdir airflow

cd airflow

 

4. docker-compose.yaml 파일 다운로드

sudo curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.yaml'

→ docker-compose.yaml 파일에는 아래와 같은 내용들이 저장되어 있다.

• postgres, redis, airflow-webserver, airflow-scheduler, airflow-worker, airflow-triggerer, airflow-init, airflow-cli, flower

→ airflow-webserver default port : 8080

→ 마운트 된 볼륨 경로

 

5. 컨테이너 작업을 vm 인스턴스에 마운트 시킬 폴더 생성

mkdir -p ./dags ./logs ./plugins

 

6. user ID 받아 올 환경 변수 설정

echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env

 

7. db 초기화 및 최초 유저 계정 생성

docker-compose up airflow-init

db 초기화가 완료 되면 아래와 같이 id/pw가 airflow인 최초 계정이 생기게 된다.

 

8. airflow 실행

docker-compose up

→ 각 docker container 생성 확인

 

→ 외부IP:8080 접속 시 로그인 창이 뜨고 최초 생성 된 id : airflow / pw : airflow를 입력하면 로그인이 된다.

 

→  로그인에 성공하면 모든 예제 DAG가 대시보드에 뜨게 된다.