Подготовка инфраструктуры
Требования
-
GitLab;
-
Linux-хост для установки GitLab Runner, имеющий:
-
Bash;
-
Git версии 2.18.0 или выше;
-
GPG.
-
Установка GitLab Runner
Установите GitLab Runner на выделенный для него хост, следуя официальным инструкциям.
Установка Buildah
Для установки Buildah выполните следующие инструкции на хосте для GitLab Runner:
- Установите пакет Buildah, следуя официальным инструкциям, но не производите его дальнейшую настройку. Если для вашего дистрибутива нет готовых пакетов Buildah, используйте следующие инструкции:
-
Установите пакеты, предоставляющие программы
newuidmap
иnewgidmap
. -
Убедитесь, что программы
newuidmap
иnewgidmap
имеют корректные права:sudo setcap cap_setuid+ep /usr/bin/newuidmap sudo setcap cap_setgid+ep /usr/bin/newgidmap sudo chmod u-s,g-s /usr/bin/newuidmap /usr/bin/newgidmap
-
Установите пакет, предоставляющий файлы
/etc/subuid
и/etc/subgid
. -
Убедитесь, что в файлах
/etc/subuid
и/etc/subgid
имеется строка видаgitlab-runner:1000000:65536
, где:-
gitlab-runner
— имя пользователя GitLab Runner; -
1000000
— первый subUID/subGID в выделяемом диапазоне; -
65536
— размер диапазона subUIDs/subGIDs (минимум65536
).
Избегайте коллизий с другими диапазонами, если они имеются. Изменение файлов может потребовать перезагрузки. Подробнее в
man subuid
иman subgid
. -
-
(Для Linux 5.12 и ниже) Установите пакет, предоставляющий программу
fuse-overlayfs
. -
Убедитесь, что путь
/home/gitlab-runner/.local/share/containers
создан, и пользовательgitlab-runner
имеет доступ на чтение и запись. -
Команда
sysctl -ne kernel.unprivileged_userns_clone
НЕ должна вернуть0
, а иначе выполнитеecho 'kernel.unprivileged_userns_clone = 1' | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
. -
Команда
sysctl -n user.max_user_namespaces
должна вернуть15000
или больше, а иначе выполнитеecho 'user.max_user_namespaces = 15000' | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
.
Установка werf
Для установки werf на хосте для GitLab Runner выполните:
curl -sSL https://werf.io/install.sh | bash -s -- --ci
Регистрация GitLab Runner
Для регистрации GitLab Runner в GitLab следуйте официальным инструкциям, указав Shell в качестве executor’а. При желании после регистрации произведите дополнительную конфигурацию GitLab Runner’а.
Конфигурация container registry
Включите сборщик мусора вашего container registry.
Подготовка системы к кроссплатформенной сборке (опционально)
Данный шаг требуется только для сборки образов для платформ, отличных от платформы системы, где запущен werf.
Регистрируем в системе эмуляторы с помощью образа qemu-user-static:
docker run --restart=always --name=qemu-user-static -d --privileged --entrypoint=/bin/sh multiarch/qemu-user-static -c "/register --reset -p yes && tail -f /dev/null"
Настройка проекта
Настройка проекта GitLab
-
Включите требование удачно выполненного pipeline для merge requests.
-
Включите возможность автоматически отменять лишние pipelines.
-
Создайте и сохраните access token для очистки ненужных образов из container registry со следующей конфигурацией:
-
Token name:
werf-images-cleanup
; -
Role:
developer
; -
Scopes:
api
.
-
-
В переменных проекта добавьте следующие переменные:
-
Версия werf:
-
Key:
WERF_VERSION
; -
Value:
2 stable
;
-
-
Access token для очистки ненужных образов:
-
Key:
WERF_IMAGES_CLEANUP_PASSWORD
; -
Value:
<сохранённый "werf-images-cleanup" access token>
; -
Protect variable:
yes
; -
Mask variable:
yes
.
-
-
-
Добавьте плановое задание на каждую ночь для очистки ненужных образов в container registry, указав ветку
main
/master
в качестве Target branch.
Конфигурация CI/CD проекта
Так может выглядеть репозиторий, использующий werf для сборки и развертывания:
/.werf-deploy-report.json
/.werf-build-report.json
/.werf_secret_key
stages:
- build
- test
- review
- qa
- staging
- staging-smoke
- prod
- prod-smoke
- cleanup
variables:
WERF_BUILDAH_MODE: auto
ENABLE_ALL_IMAGES: "false"
ENABLE_APPS_IMAGES: "false"
ENABLE_IMAGE_SOURCES: "false"
WERF_REQUIRE_BUILT_IMAGES: "true"
workflow:
rules:
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push"
when: never
- if: !reference [.rules, if, mr]
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: !reference [.rules, if, tag]
- if: !reference [.rules, if, main]
default:
tags:
- "<GitLab Runner tag>"
.scripts:
setup_werf: |
source "$(~/bin/trdl use werf ${WERF_VERSION:?})"
source "$(werf ci-env gitlab --as-file)"
.rules:
if:
main: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
main_schedule: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "schedule"
main_no_schedule: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"
mr: $CI_PIPELINE_SOURCE == "merge_request_event"
mr_no_schedule: $CI_PIPELINE_SOURCE == "merge_request_event"&& $CI_PIPELINE_SOURCE != "schedule"
main_or_mr: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_PIPELINE_SOURCE == "merge_request_event"
main_or_mr_no_schedule: ($CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_PIPELINE_SOURCE == "merge_request_event") && $CI_PIPELINE_SOURCE != "schedule"
tag: $CI_COMMIT_TAG
schedule: $CI_PIPELINE_SOURCE == "schedule"
no_schedule: $CI_PIPELINE_SOURCE != "schedule"
.artifacts:
deploy_report:
paths:
- .werf-deploy-report.json
when: always
.dismiss:
script:
- werf dismiss --with-namespace --use-deploy-report
environment:
action: stop
interruptible: false
timeout: null
.build:
stage: build
variables:
ENABLE_ALL_IMAGES: "true"
before_script:
- !reference [.scripts, setup_werf]
timeout: 30m
rules:
- if: !reference [.rules, if, main_or_mr_no_schedule]
images:apps:build:
extends:
- .build
script:
- werf build app
image:sources:build:
extends:
- .build
script:
- werf build sources
.test:simple:
stage: test
needs:
- image:sources:build
variables:
ENABLE_IMAGE_SOURCES: "true"
before_script:
- !reference [.scripts, setup_werf]
rules:
- if: !reference [.rules, if, main_or_mr_no_schedule]
app:lint:
extends:
- .test:simple
script:
- werf kube-run sources -- go-task lint
timeout: 5m
app:unit:
extends:
- .test:simple
script:
- werf kube-run sources -- go-task test:unit
coverage: '/Code coverage is \d+\.\d+/'
timeout: 15m
app:integration:
extends:
- .test:simple
script:
- werf kube-run sources -- go-task test:integration
coverage: '/Code coverage is \d+\.\d+/'
timeout: 30m
.test:complex:
stage: test
needs:
- images:apps:build
- image:sources:build
variables:
ENABLE_ALL_IMAGES: "true"
WERF_SET_TAGS_RELEASE: "tags.release=true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf converge --save-deploy-report
resource_group: ${CI_ENVIRONMENT_SLUG}
artifacts: !reference [.artifacts, deploy_report]
e2e:short:
extends:
- .test:complex
variables:
WERF_SET_TAGS_E2E_SHORT: "tags.e2e-short=true"
environment:
name: e2e-short/${CI_PIPELINE_IID}
on_stop: e2e:short:remove
interruptible: true
rules:
- if: !reference [.rules, if, mr_no_schedule]
e2e:short:remove:
extends:
- e2e:short
- .dismiss
needs:
- e2e:short
rules:
- if: !reference [.rules, if, mr_no_schedule]
when: always
allow_failure: true
e2e:long:
extends:
- .test:complex
variables:
WERF_SET_TAGS_E2E_LONG: "tags.e2e-long=true"
environment:
name: e2e-long/${CI_PIPELINE_IID}
on_stop: e2e:long:remove
rules:
- if: !reference [.rules, if, main_no_schedule]
e2e:long:remove:
extends:
- e2e:long
- .dismiss
needs:
- e2e:long
rules:
- if: !reference [.rules, if, main_no_schedule]
when: always
allow_failure: true
performance:
extends:
- .test:complex
variables:
WERF_SET_TAGS_PERFORMANCE: "tags.performance=true"
environment:
name: performance/${CI_PIPELINE_IID}
on_stop: performance:remove
resource_group: performance
rules:
- if: !reference [.rules, if, main_no_schedule]
performance:remove:
extends:
- performance
- .dismiss
needs:
- performance
rules:
- if: !reference [.rules, if, main_no_schedule]
when: always
allow_failure: true
review:
stage: review
needs:
- images:apps:build
- image:sources:build
variables:
ENABLE_ALL_IMAGES: "true"
WERF_SET_TAGS_RELEASE: "tags.release=true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf converge --save-deploy-report
environment:
name: review/${CI_MERGE_REQUEST_IID}
on_stop: review:remove
auto_stop_in: 3 days
timeout: 30m
interruptible: true
resource_group: ${CI_ENVIRONMENT_SLUG}
artifacts: !reference [.artifacts, deploy_report]
rules:
- if: !reference [.rules, if, mr_no_schedule]
review:remove:
extends:
- review
- .dismiss
needs:
- review
rules:
- if: !reference [.rules, if, mr_no_schedule]
when: manual
allow_failure: true
.qa:
stage: qa
needs:
- app:lint
- app:unit
- app:integration
- job: e2e:long
artifacts: false
- job: performance
artifacts: false
variables:
ENABLE_ALL_IMAGES: "true"
WERF_SET_TAGS_RELEASE: "tags.release=true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf converge --save-deploy-report
environment:
auto_stop_in: 3 days
timeout: 30m
resource_group: ${CI_ENVIRONMENT_SLUG}
artifacts: !reference [.artifacts, deploy_report]
rules:
- if: !reference [.rules, if, main_no_schedule]
when: manual
qa:eu:
extends:
- .qa
environment:
name: qa-eu/${CI_PIPELINE_IID}
on_stop: qa:eu:remove
qa:eu:remove:
extends:
- qa:eu
- .dismiss
needs:
- qa:eu
rules:
- if: !reference [.rules, if, main_no_schedule]
when: manual
allow_failure: true
qa:us:
extends:
- .qa
environment:
name: qa-us/${CI_PIPELINE_IID}
on_stop: qa:us:remove
qa:us:remove:
extends:
- qa:us
- .dismiss
needs:
- qa:us
rules:
- if: !reference [.rules, if, main_no_schedule]
when: manual
allow_failure: true
.staging:
stage: staging
needs:
- app:lint
- app:unit
- app:integration
- job: e2e:long
artifacts: false
- job: performance
artifacts: false
variables:
ENABLE_ALL_IMAGES: "true"
WERF_SET_TAGS_RELEASE: "tags.release=true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf converge
resource_group: ${CI_ENVIRONMENT_SLUG}
rules:
- if: !reference [.rules, if, main_no_schedule]
when: manual
staging:eu:
extends:
- .staging
environment:
name: staging-eu
staging:us:
extends:
- .staging
environment:
name: staging-us
.prod:
stage: prod
variables:
ENABLE_ALL_IMAGES: "true"
WERF_SET_TAGS_RELEASE: "tags.release=true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf converge
resource_group: ${CI_ENVIRONMENT_SLUG}
rules:
- if: !reference [.rules, if, main_no_schedule]
when: manual
prod:eu:
extends:
- .prod
needs:
- staging:eu:smoke
environment:
name: prod-eu
prod:us:
extends:
- .prod
needs:
- staging:us:smoke
environment:
name: prod-us
.smoke:
variables:
ENABLE_IMAGE_SOURCES: "true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf kube-run sources -- go-task test:smoke
timeout: 20m
rules:
- if: !reference [.rules, if, main_no_schedule]
staging:eu:smoke:
extends:
- .smoke
stage: staging-smoke
needs:
- staging:eu
staging:us:smoke:
extends:
- .smoke
stage: staging-smoke
needs:
- staging:us
prod:eu:smoke:
extends:
- .smoke
stage: prod-smoke
needs:
- prod:eu
prod:us:smoke:
extends:
- .smoke
stage: prod-smoke
needs:
- prod:us
images:cleanup:
stage: cleanup
variables:
ENABLE_ALL_IMAGES: "true"
before_script:
- !reference [.scripts, setup_werf]
script:
- werf cr login -u nobody -p "${WERF_IMAGES_CLEANUP_PASSWORD:?}" "${WERF_REPO:?}"
- werf cleanup
resource_group: cleanup-images
rules:
- if: !reference [.rules, if, main_schedule]
dependencies:
- name: app
version: 1.0.0
tags:
- app
- apps
- release
- all
export-values: &export-values
- parent: werf
child: werf
- name: database
version: 1.0.0
condition: database.enabled
tags:
- database
- infra
- release
- all
export-values: *export-values
- name: e2e-short
version: 1.0.0
tags:
- e2e-short
- tests
- all
export-values: *export-values
- name: e2e-long
version: 1.0.0
tags:
- e2e-long
- tests
- all
export-values: *export-values
- name: performance
version: 1.0.0
tags:
- performance
- tests
- all
export-values: *export-values
apiVersion: v2
name: app
version: 1.0.0
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ $.Chart.Name }}
spec:
selector:
matchLabels:
app: {{ $.Chart.Name }}
template:
metadata:
labels:
app: {{ $.Chart.Name }}
spec:
imagePullSecrets:
- name: registrysecret
containers:
- name: {{ $.Chart.Name }}
image: {{ index $.Values.werf.image $.Chart.Name }}
ports:
- containerPort: 80
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ $.Chart.Name }}
spec:
ingressClassName: nginx
rules:
- host: "{{ $.Release.Name }}.{{ $.Chart.Name }}.example.org"
http:
paths:
- backend:
service:
name: {{ $.Chart.Name }}
port:
number: 80
path: /
pathType: Prefix
{{ if $.Release.IsInstall }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $.Chart.Name }}-init-db-{{ $.Release.Revision }}
annotations:
werf.io/weight: "-20"
spec:
template:
spec:
imagePullSecrets:
- name: registrysecret
restartPolicy: Never
containers:
- name: init-db
image: {{ $.Values.werf.image.sources }}
command: ["go-task", "db:init"]
{{ end }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $.Chart.Name }}-migrate-db-{{ $.Release.Revision }}
annotations:
werf.io/weight: "-10"
spec:
template:
spec:
imagePullSecrets:
- name: registrysecret
restartPolicy: Never
containers:
- name: migrate-db
image: {{ $.Values.werf.image.sources }}
command: ["go-task", "db:migrate"]
apiVersion: v1
kind: Service
metadata:
name: {{ $.Chart.Name }}
spec:
ports:
- port: 80
selector:
app: {{ $.Chart.Name }}
apiVersion: v2
name: database
version: 1.0.0
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
annotations:
werf.io/weight: "-30"
spec:
serviceName: db
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: db
image: alpine:3.17
command: ["tail", "-f", /dev/null]
apiVersion: v2
name: e2e-long
version: 1.0.0
apiVersion: batch/v1
kind: Job
metadata:
name: test-e2e-long-{{ $.Release.Revision }}
annotations:
werf.io/weight: "10"
spec:
template:
spec:
imagePullSecrets:
- name: registrysecret
restartPolicy: Never
containers:
- name: test-e2e-long
image: {{ $.Values.werf.image.sources }}
command: ["go-task", "test:e2e:long"]
apiVersion: v2
name: e2e-short
version: 1.0.0
apiVersion: batch/v1
kind: Job
metadata:
name: test-e2e-short-{{ $.Release.Revision }}
annotations:
werf.io/weight: "10"
spec:
template:
spec:
imagePullSecrets:
- name: registrysecret
restartPolicy: Never
containers:
- name: test-e2e-short
image: {{ $.Values.werf.image.sources }}
command: ["go-task", "test:e2e:short"]
apiVersion: v2
name: performance
version: 1.0.0
apiVersion: batch/v1
kind: Job
metadata:
name: test-performance-{{ $.Release.Revision }}
annotations:
werf.io/weight: "10"
spec:
template:
spec:
imagePullSecrets:
- name: registrysecret
restartPolicy: Never
containers:
- name: test-performance
image: {{ $.Values.werf.image.sources }}
command: ["go-task", "test:performance"]
dependencies:
- name: app
repository: ""
version: 1.0.0
- name: database
repository: ""
version: 1.0.0
- name: e2e-short
repository: ""
version: 1.0.0
- name: e2e-long
repository: ""
version: 1.0.0
- name: performance
repository: ""
version: 1.0.0
digest: sha256:efce1b655fce8c8bf7f1739dc34a98fed3dad7943cef39bd6829aeaee34c03e9
generated: "2023-02-08T12:17:50.931015996+03:00"
{{ $chartsTagNames := dict }}
{{- range $.Chart.Dependencies }}
{{- if .Alias }}
{{- $_ := set $chartsTagNames .Alias .Tags }}
{{- else }}
{{- $_ := set $chartsTagNames .Name .Tags }}
{{- end }}
{{- end }}
{{- range $chartsTagNames.app }}
{{- if dig . false $.Values.tags }}
Application: https://{{ $.Release.Name }}.app.example.org
{{- break }}
{{- end }}
{{- end }}
tags:
all: false
version: "3"
tasks:
build:
cmds:
# Here be building your app:
- cp src/main.sh app.sh
run:
cmds:
# Here be running your app:
- ./app.sh
lint:
cmds:
# Here be your linter:
- echo Lint completed.
test:unit:
cmds:
# Here be your unit tests:
- echo Unit tests completed. Code coverage is 12.34%
test:integration:
cmds:
# Here be your integration tests:
- echo Integration tests completed. Code coverage is 43.21%
test:e2e:short:
cmds:
# Here be your short E2E tests:
- echo Short E2E tests completed.
test:e2e:long:
cmds:
# Here be your long E2E tests:
- echo Long E2E tests completed.
test:performance:
cmds:
# Here be your performance tests:
- echo Performance tests completed.
test:smoke:
cmds:
# Here be your smoke tests:
- echo Smoke tests completed.
db:init:
cmds:
# Here be your DB initialization:
- echo DB initialized.
db:migrate:
cmds:
# Here be your DB migrations:
- echo DB migrated.
FROM alpine:3.17 as builder
WORKDIR /app
RUN apk add go-task
COPY . .
RUN go-task build
FROM alpine:3.17
WORKDIR /app
RUN apk add nmap-ncat go-task
COPY --from=builder /app/app.sh /app/Taskfile.yaml ./
CMD ["go-task", "run"]
FROM alpine:3.17
WORKDIR /src
RUN apk add go-task
COPY . .
#!/bin/sh
while true; do
printf "HTTP/1.1 200 OK\n\nHello world.\n" | ncat -lp 80
done
giterminismConfigVersion: 1
config:
goTemplateRendering:
allowEnvVariables:
- ENABLE_ALL_IMAGES
- ENABLE_APPS_IMAGES
- ENABLE_IMAGE_SOURCES
configVersion: 1
project: myproject
{{- if or (env "ENABLE_ALL_IMAGES" | eq "true") (env "ENABLE_APPS_IMAGES" | eq "true") }}
---
image: app
dockerfile: ./app.Dockerfile
{{- end }}
{{- if or (env "ENABLE_ALL_IMAGES" | eq "true") (env "ENABLE_IMAGE_SOURCES" | eq "true") }}
---
image: sources
dockerfile: ./sources.Dockerfile
{{- end }}
Дополнительно:
- Добавьте для
werf cleanup
опции авторизации в container registry, следуя инструкциям.