PRs for running the init jobs:

• https://github.com/wmde/wbaas-deploy/pull/1065
• https://github.com/wmde/wbaas-deploy/pull/1066

It appears that the image we custom build from an older bitnami starting image doesn't seem to be compatible with the latest charts. This means that the node roles aren't correctly assigned. We talked about this a little and drew up a similar options list to the one drawn up before:

Option 1 we stick with 6 now we can't update mediawiki / we're now running an unsupported version -F

Option 2 we keep using the latest bitnami chart and build an older image that is compatible with it (and licesned correctly)

Option 3 we go back to using a historic elasticco built chart, fork it for k8s comptibility and also continue building an older image that is compatible and open licensed - F

Option 4 we use the new bitnami chart and build our custom image off a new image and add the plugins to this (we enter some license minefield)

Option 5 OpenSearch???

Option 6 Revisit product requirements - try to find alternative that isn't wikibase cirrussearch

paper trail: here are the last charts from bitnami and elastic that explicitly supporting 7.10.2:

• https://github.com/bitnami/charts/commit/e1256a97267835165339433c9bfd146ab9ce4691
• https://github.com/elastic/helm-charts/commit/f01036e04be9aa62afa52f052c4e600c9e3e1da7

Aaand... now that I'm off for a week something really needs to be added to this ticket so that there is a chance other people can make some headway in may absence given that there was no opportunity for some in person handover.

After looking at this I invested (probably too much) time in trying to build something analogous to option 2.

My strategy was to lightly tweak the Dockerfile used by the latest bitnami image to load an older (FOSS) version of the code. From this image I then derived an image in the same way we currently do with the repo at https://github.com/wbstack/elasticsearch.

I then attempted somewhat repeatedly to use this with the modern chart and hoped to end with a working cluster.

The lite tweak to the bitnami containers repo was as at: https://github.com/tarrow/containers (basically using the FOSS tarball extracted with the elasticsearch.yml removed) as the source but leaving all the bitnami special sauce present. This file must not be present or the special bitnami scripts do not generate the config but instead assume the baked in one is what you want.

I then spent much time debugging the libelasticsearch.sh script to try to determine if the reasons my cluster was falling over was bad config, transient, or other things.

n.b. removing the liveness probes is very useful to prevent pods getting killed before their time (or before you can inspect them). Adding something like:

coordinating:
  livenessProbe:
    enabled: false

The current hurdle I was stuck on (although probably not far from success) was preventing the generated config from including references to xpack this being a non-FOSS plugin that we don't provide or load (but causes it to bail early) even though all the settings there were to ensure these features (security and ml) were disabled.

Currently untested before I really needed to leave was the following addition to the values file:

extraConfig:
  # T309378
  action.auto_create_index: false
  xpack:

in the hope that leaving this xpack key empty would allow the nodes to boot. Maybe this is a place to start.

I was repeatedly considering and making half-hearted attempts at Option 3 / fork it in the simplest way and write my own custom minimalist chart that would be easy for us to maintain.

Courage mon brave