Create a git-sync container image to be used with airflow
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	BTullis
	Jun 28 2024, 4:22 PM

Description

We have been discussing in T368033: Design a suitable DAG deployment method what sort of system we might use for DAG management for Airflow under Kubernetes.

It looks like we may want to use a git-sync sidecar method, so for this we would need a container image with this functionality.

The upstream git-sync project is here: https://github.com/kubernetes/git-sync

We may want to use this project and build a version with blubber/kokkuri

Bear in mind that we may find other uses for git-sync, such as T347421: [NEEDS GROOMING] schema services should be moved to k8s.

Related Objects
Search...

Status	Assigned	Task
Open	None	T362788 Migrate Airflow to the dse-k8s cluster
Open	None	T364387 Adapt Airflow auth and DAG deployment method
Open	brouberol	T368033 Design a suitable DAG deployment method
Resolved	bking	T368757 Create a git-sync container image to be used with airflow

Event Timeline

BTullis created this task.Jun 28 2024, 4:22 PM

Gehel triaged this task as High priority.Jul 9 2024, 8:10 AM

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.

Gehel edited projects, added Data-Platform-SRE (2024.07.08 - 2024.07.28); removed Data-Platform-SRE.

bking changed the task status from Open to In Progress.Jul 12 2024, 1:32 PM

bking claimed this task.

bking subscribed.

I have the git-sync binary building locally on a Linux machine via the repo-supplied build script . Next step is to get it working with blubber.

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/2

blubber: add first draft of blubberfile

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/2

blubber: add first draft of blubberfile

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/3

blubber: add multiple build stages

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/3

blubber: add multiple build stages

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/4

blubber: Add additional pipeline steps

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/4

blubber: Add additional pipeline steps

I forgot to tag this task, but I also have an MR up to add the new repo to the trusted runners.

BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.07.08 - 2024.07.28) board.Jul 15 2024, 1:36 PM

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/5

Configure the build pipeline to use a specific version of git-sync

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/5

Configure the build pipeline to use a specific version of git-sync

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/6

Add .gitlab-ci.yml (gitlab CI config)

bking closed https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/6

Add .gitlab-ci.yml (gitlab CI config)

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2024, 3:31 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/7

Add .gitlab-ci.yml (gitlab CI config), use wget instead of git for fetching source code, use envvars

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/7

Add .gitlab-ci.yml (gitlab CI config), use wget instead of git for fetching source code, use envvars

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2024, 6:31 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/8

Having some pipeline timeouts.

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/8

Having some pipeline timeouts.

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/9

Add a sleep command to the production stage, just to

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/9

Add a sleep command to the production stage, just to

Maintenance_bot removed a project: Patch-For-Review.Jul 24 2024, 9:31 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/10

Explicitly set the proxy URLs for wget

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/10

Explicitly set the proxy URLs for wget

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2024, 1:31 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/11

Remove proxy vars

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/11

Remove proxy vars

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2024, 2:31 PM

bking edited projects, added Data-Platform-SRE (2024.07.29 - 2024.08.16); removed Data-Platform-SRE (2024.07.08 - 2024.07.28).Jul 31 2024, 2:47 PM

bking moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.07.29 - 2024.08.16) board.

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/12

Add version tag to image

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/12

Add version tag to image

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 4:33 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/13

gitlab-ci.yml: hardcode version var

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/13

gitlab-ci.yml: hardcode version var

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 5:30 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/14

gitlab-ci: try hardcoding version in PUBLISH_IMAGE_EXTRA_TAGS

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/14

gitlab-ci: try hardcoding version in PUBLISH_IMAGE_EXTRA_TAGS

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 6:31 PM

bking opened https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/15

production build step: add git package

bking merged https://gitlab.wikimedia.org/repos/data-engineering/git-sync/-/merge_requests/15

production build step: add git package

tested by running the following command within the docker container: /usr/local/bin/git-sync --branch=main --depth=1 --dest=repo --max-sync-failures=0 --ref=main --repo=https://gitlab.wikimedia.org/repos/releng/kokkuri.git --rev=HEAD --root=/home/git-sync

In production, we will probably want to invoke git-sync in a similar fashion as this github comment .

At this point, I believe the image is done but will need review from team...hmm, I think there's a Phab status for that ;) ....

bking moved this task from In Progress to Needs Review on the Data-Platform-SRE (2024.07.29 - 2024.08.16) board.Wed, Aug 7, 1:47 PM

Maintenance_bot removed a project: Patch-For-Review.Wed, Aug 7, 2:31 PM

BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.Wed, Aug 7, 3:13 PM

Per Slack conversation with @BTullis , he has reviewed the image and is happy with it for the time being. Side note, I had some issues discovering info about existing WMF-hosted docker registry images during the course of this ticket. I created ticket T371549 for this. Resolving this ticket, as the AC is complete.

Yep, I executed:

btullis@marlin:~$ docker run -it docker-registry.wikimedia.org/repos/data-engineering/git-sync
runuser@9bcf787d48ad:/home/git-sync$ git-sync --one-time --ref=main --depth=1 --link=repo --max-failures=0 --repo=https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags --root=/home/runuser

Logs were as follows:

{"logger":"","ts":"2024-08-07 15:53:42.035888","caller":{"file":"main.go","line":568},"level":0,"msg":"starting up","version":"4.2.4","pid":433,"uid":900,"gid":900,"home":"/home/runuser","flags":["--depth=1","--link=repo","--max-failures=0","--one-time=true","--ref=main","--repo=https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags","--root=/home/runuser"]}
{"logger":"","ts":"2024-08-07 15:53:42.037091","caller":{"file":"main.go","line":673},"level":0,"msg":"git version","version":"git version 2.39.2"}
{"logger":"","ts":"2024-08-07 15:53:43.120806","caller":{"file":"main.go","line":1682},"level":0,"msg":"update required","ref":"main","local":"5569f857581fa58facff457891caaa20956a43ef","remote":"5569f857581fa58facff457891caaa20956a43ef","syncCount":0}
{"logger":"","ts":"2024-08-07 15:53:43.149370","caller":{"file":"main.go","line":1728},"level":0,"msg":"updated successfully","ref":"main","remote":"5569f857581fa58facff457891caaa20956a43ef","syncCount":1}
{"logger":"","ts":"2024-08-07 15:53:43.149459","caller":{"file":"main.go","line":906},"level":0,"msg":"exiting after one sync","status":0}

Looks good to me.

brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2024.07.29 - 2024.08.16) board.Mon, Aug 12, 2:08 PM

Gehel edited parent tasks, added: T368033: Design a suitable DAG deployment method; removed: T364387: Adapt Airflow auth and DAG deployment method.Fri, Aug 16, 3:04 PM

Create a git-sync container image to be used with airflowClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Create a git-sync container image to be used with airflow
Closed, ResolvedPublic
Actions

Related Objects
Search...