@pschichtel Do you only have one KES server for production?

@jiuker production has 3, backup has 1

@jiuker production has 3, backup has 1

Does the 3 KES server have the same keys for production? @pschichtel

They are all connected to the same vault (with a dedicated V2 KV engine for minio), so I'd assume so. How can I check?

Could you check the key site-replicator-0's value have changed always? @pschichtel

Not sure what you mean

Not sure what you mean

Check for overlapping value assignments between two clients

Sorry for being confused!

Could you check the key site-replicator-0's value have changed always?

By key, do you mean a KES key or an access key/secret access key? There is no "site-replicator-0" KES key, so I assume access key. What do you mean by "value" then?

Check for overlapping value assignments between two clients

What do you mean by "value assignments"? And what clients?

I just checked with mcli again (mcli admin user svcacct info production site-replicator-0) and every now and then I get mcli: <ERROR> Unable to get information of the specified service account. The specified service account is not found (Specified service account does not exist)., so there must be an instance that doesn't have the account I guess. Didn't notice it yesterday weirdly. If it doesn't fail with said error, then it returns this consistently:

AccessKey: site-replicator-0
ParentUser: root-user
Status: on
Name: 
Description: 
Policy: implied
Expiration: no-expiry

I only found a strange case where when I open two minio login pages in one browser at the same time, there have one said that {"detailedMessage":"Access Denied.","message":"invalid session"} , but I can access it normally with two browsers. I don't know if it's similar to your case? @pschichtel

@jiuker I have the occasional case, where the page after login stays blank, because /api/v1/session returns invalid session / The Access Key Id you provided does not exist in our records.. I don't need a two tabs/browsers for that.

I think you case sounds like a race condition on the shared cookies/localStorage/sessionStorage between the browser tabs, that are not shared between browsers.

@jiuker I have the occasional case, where the page after login stays blank, because /api/v1/session returns invalid session / The Access Key Id you provided does not exist in our records.. I don't need a two tabs/browsers for that.

I think you case sounds like a race condition on the shared cookies/localStorage/sessionStorage between the browser tabs, that are not shared between browsers.

Yeah. Will return back to login page for /api/v1/session return that {"detailedMessage":"Access Denied.","message":"invalid session"} when fresh the page. It looks like your case. So I guess that could be a console issue. What's the page you view when that happen. @pschichtel

@jiuker I don't think it is limited to a specific page, I've seen it happen on several different pages.

So I guess that could be a console issue.

I'm not so sure anymore, because I get errors with mcli too, that doesn't go through the console, right?

We can't reproduce any of the issues reported here.

How can I properly clear replication settings from both sites? then I could test the production cluster without site replication and see if that helps.

I just noticed, even the replication rules on buckets are completely inconsistent from refresh to refresh.

@poornas thanks, I'll try that next week.

Bucket versioning is also affected. it seems like everything somehow related to site replication is completely inconsistent between the nodes of the production cluster. It also seems to have gotten worse since I last checked last week.

@poornas I removed the backup site from replication and it's all fine now. Should the site-replicator-0 account disappear? Or should I clean that account up before re-enabling replication?

Remove site-replication and site-replicator-0 account disappear. You cannot remove that account, it's interval account.

Are you saying it should automatically disappear after removing site-replication? Because it hasn't so far, neither in the production site nor in the backup site. Both sites don't have any other replication rules.

So I'll delete the service accounts to have a clean state.

Are you saying it should automatically disappear after removing site-replication? Because it hasn't so far, neither in the production site nor in the backup site. Both sites don't have any other replication rules.

So I'll delete the service accounts to have a clean state.

Yeah. It should disappear. If not, you can try delete it for I didn't reproduce your case.

I removed the accounts, I'll upgrade both instances to latest now and then setup replication again in the evening.

I remember @harshavardhana saying something about this in a past issue: The /v1/service-accounts endpoint is rather slow (400-900 ms "wait" time in the browser) given that this is a small cluster (5 nodes) and only 3 service accounts exist and my connection is basically local, this feels noticeable slow in the UI. This is still the case even after disabling replication. Is the timing within a normal range or would this be worth investigating? I originally thought this is caused by the replication problem, but apparently it isn't.

I'd argue the operator would better stop and enter an error state, when a correct upgrade is not possible.

How can we find the cause of the Tar file extraction failed for file index: 2, with: EOF error? As I said: It happens in completely independent setups with different networks and different host systems (different hardware, different kernel, different distro), however they are both in k0s and they both use a similarly configured tenant created through the helm chart, the values:

tenant:
  image:
    repository: quay.io/minio/minio
    tag: 'RELEASE.2024-06-11T03-13-30Z'
  name: tenant
  configuration:
    name: credentials
  pools:
  - name: main
    servers: 5
    volumesPerServer: 1
    storageClassName: ''
    size: 123123123
    labels:
      velero.io/exclude-from-backup: "true"
  metrics:
    enabled: true
  certificate:
    requestAutoCert: true
  env:
  - name: MINIO_OPERATOR_TLS_ENABLE
    value: "off"
  - name: MINIO_DOMAIN
    value: "minio.example.org"
  - name: MINIO_BROWSER_REDIRECT_URL
    value: "https://console.minio.example.org"
  - name: MINIO_SERVER_URL
    value: "https://minio.example.org"
  log:
    disabled: true
  prometheus:
    disabled: true
  prometheusOperator: true

also there is no NetworkPolicy and also no outbound firewall in general, so there is no reason why the operator shouldn't be able to download and distribute the minio binaries.

@harshavardhana todays minio release seems to have a few changes relevant to this issue (especially the binary validation). is it worth testing with this version or would the operator likely still fail to update the binary?

@harshavardhana todays minio release seems to have a few changes relevant to this issue (especially the binary validation). is it worth testing with this version or would the operator likely still fail to update the binary?

Not really that seems to be a different problem, I will investigate it later coming week.

OK, great. Tag me if you need something tested.

I think k0s is causing these problems, what is your container runtime?

k0s' default: containerd/runc, nothing fancy. The affected clusters are on k0s 1.29.x and 1.30.x

This is the k0sctl config I use to deploy the single-node home lab:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: ...
      user: ...
      port: ...
      keyPath: ...
    role: single
    privateInterface: ...
    privateAddress: &ip ...
    # This currently won't work: https://github.com/k0sproject/k0sctl/issues/476
    # installFlags: ['--kubelet-extra-args="--cpu-manager-policy=static"']
    installFlags: ['--profile=enable-swap']
  k0s:
    version: v1.30.2+k0s.0
    dynamicConfig: false
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s
      spec:
        api:
          address: *ip
          externalAddress: ...
          k0sApiPort: 9443
          port: 6443
          sans:
          - ...
          - *ip
          - 127.0.0.1
        extensions:
          helm:
            charts: null
            repositories: null
          storage: {}
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        network:
          calico: null
          dualStack: {}
          kubeProxy:
            mode: iptables
          kuberouter:
            autoMTU: true
            mtu: 0
            # removing these will break networking
            peerRouterASNs: ""
            peerRouterIPs: ""
          podCIDR: 10.244.0.0/16
          provider: kuberouter
          serviceCIDR: 10.96.0.0/12
        storage:
          etcd:
            externalCluster: null
            peerAddress: *ip
          type: etcd
        workerProfiles:
        - name: enable-swap
          values:
            memorySwap:
              swapBehavior: LimitedSwap
            featureGates:
              NodeSwap: true

most of the values are defaults

can you also check by exec into operator pod at /tmp/webhook/v1/update/ and then do du -sh /tmp/webhook/v1/update/* and ls -ltr /tmp/webhook/v1/update/

oh looks like this is not kept back, after the fetch()

Hopefully with this change we can do some more deeper investigation.

@harshavardhana if you have a rough guide on how to deploy that change as a custom container I'd happily through that on my home lab

@harshavardhana if you have a rough guide on how to deploy that change as a custom container I'd happily through that on my home lab

What is your current operator version?

5.0.15

I guess we are waiting for the operator 6 release here, right?

@pschichtel yes

however I would suggest upgrading to the latest release to fix few things where it now waits properly.

operator 6.0 would handle the container behavior.

I upgraded the operator to 5.0.16 and upgrade minio to RELEASE.2024-07-04T14-25-45Z. 5.0.16 indeed doesn't carry your changes, the same issue with the same log line still occurs, but after the upgrade the cluster still seems fine so far. So your changes seem to have helped with the symptoms

I upgraded the operator to 5.0.16 and upgrade minio to RELEASE.2024-07-04T14-25-45Z. 5.0.16 indeed doesn't carry your changes, the same issue with the same log line still occurs, but after the upgrade the cluster still seems fine so far. So your changes seem to have helped with the symptoms

yeah they will definitely help however v6.0.0 would not cause the same problem as v5.0.16

I upgraded the operator to 5.0.16 and upgrade minio to RELEASE.2024-07-04T14-25-45Z. 5.0.16 indeed doesn't carry your changes, the same issue with the same log line still occurs, but after the upgrade the cluster still seems fine so far. So your changes seem to have helped with the symptoms

so this issue can be closed ?

Yeah we can close this and I promise to verify that operator 6 indeed performs the upgrade correctly or I'll file a new issue with the additional information from the logs. Would I file the new issue here or in the operator repo?

Yeah we can close this and I promise to verify that operator 6 indeed performs the upgrade correctly or I'll file a new issue with the additional information from the logs. Would I file the new issue here or in the operator repo?

in the operator repo please @pschichtel thanks for your patience on this the last couple of months.