Airflow 설치 방법에는 크게 두가지가 있다.
1. docker compose로 쉽고 간단하게 설치하기
2. 가상환경 내에 airflow 설치하여 환경 셋팅하기
우선 간단한 docker compose로 설치하는 방법부터 정리하려고 한다.
1. docker install
Requirements
- Linux (Ubuntu 20.04 +)
Install Docker using the apt
Install Docker Engine on Ubuntu
[Docker, Ubuntu 22.04] 도커 설치부터 Nvidia-toolkit까지
딥러닝 서버를 만들기 위한 도커 설정 과정Ubuntu server를 설치한다.Docker를 설치한다Nvidia driver를 설치한다Nvidia-container tooklit을 설치한다. 도커 설치에 참고한 공식문서 링크입니다. https://docs.docke
artiper.tistory.com
docker install using apt
Update the apt package index.
sudo apt-get update
install packages to allow apt to use a repository over HTTPS.
sudo apt-get install ca-certificates curl gnupg
Add Docker’s official GPG key.
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
Use the following command to set up the repository.
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Update the apt package index.
sudo apt-get update
Install Docker Engine, containerd, and Docker Compose.
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
- Install latest version
apt-cache madison docker-ce | awk '{ print $3 }'
VERSION_STRING=5:20.10.13~3-0~ubuntu-jammy
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin
- Install specific version
Verify that the Docker Engine installation is successful by running the hello-world image.
sudo systemctl status docker
sudo docker run hello-world
Manage Docker as a non-root user
sudo groupadd docker
sudo usermod -a -G docker $USER
newgrp docker
sudo reboot
docker run hello-world
docker with GPU
rerequisites
- NVIDIA Graphic Driver
Setup the package repository and the GPG key
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
Install the nvidia-container-toolkit package (and dependencies) after updating the package listing.
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit-base nvidia-container-toolkit
Configure the Docker daemon to recognize the NVIDIA Container Runtime.
sudo nvidia-ctk runtime configure --runtime=docker
Restart the Docker daemon to complete the installation after setting the default runtime
sudo systemctl restart docker
At this point, a working setup can be tested by running a base CUDA container
sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi
2. docker compose(2.14.0 ver 이상) install
sudo curl -L "https://github.com/docker/compose/releases/download/v2.4.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose -v # 버전이 출력되면 제대로 설치 된 것
3. airflow를 설치 할 폴더 생성 후 해당 폴더로 이동
mkdir airflow
cd airflow
4. docker-compose.yaml 파일 다운로드
sudo curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.0/docker-compose.yaml'
→ docker-compose.yaml 파일에는 아래와 같은 내용들이 저장되어 있다.
- postgres, redis, airflow-webserver, airflow-scheduler, airflow-worker, airflow-triggerer, airflow-init, airflow-cli, flower
→ airflow-webserver default port : 8080
→ 마운트 된 볼륨 경로

5. 컨테이너 작업을 vm 인스턴스에 마운트 시킬 폴더 생성
mkdir -p ./dags ./logs ./plugins
6. user ID 받아 올 환경 변수 설정
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
7. db 초기화 및 최초 유저 계정 생성
docker-compose up airflow-init
db 초기화가 완료 되면 아래와 같이 id/pw가 airflow인 최초 계정이 생기게 된다.

8. airflow 실행
docker-compose up

→ 각 docker container 생성 확인

→ 외부IP:8080 접속 시 로그인 창이 뜨고 최초 생성 된 id : airflow / pw : airflow를 입력하면 로그인이 된다.

→ 로그인에 성공하면 모든 예제 DAG가 대시보드에 뜨게 된다.