[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wrong state of XR #4757

Closed
NikitaCloudRuntime opened this issue Oct 9, 2023 · 6 comments
Closed

Wrong state of XR #4757

NikitaCloudRuntime opened this issue Oct 9, 2023 · 6 comments
Labels
bug Something isn't working

Comments

@NikitaCloudRuntime
Copy link
NikitaCloudRuntime commented Oct 9, 2023 •
edited
Loading

What happened?

Wrong state of XR

How can we reproduce it?

In order to spin up a new EKS cluster on already existing networking infrastructure(AWS VPC) we prepared test Composite and XRD(see attachment)
eks-composition.yaml.txt
eks-definition.yaml.txt
eks-claim.yaml.txt

After we apply XRC, everything gets created but state of the XR is stated wrong

NAME             SYNCED   READY   COMPOSITION             AGE
hellofresh-eks   True     False   eks.aws.hellofresh.io   3h29m

State of all managed resources included into Composition is ready

k get nodegroups.eks.aws.upbound.io 
NAME                   READY   SYNCED   EXTERNAL-NAME          AGE
hellofresh-eks-h482g   True    True     hellofresh-eks-h482g   3h37m

k get clusters.eks.aws.upbound.io                              
NAME                   READY   SYNCED   EXTERNAL-NAME          AGE
hellofresh-eks-4xwgt   True    True     hellofresh-eks-4xwgt   3h37m

k get roles.iam.aws.upbound.io 
NAME                   READY   SYNCED   EXTERNAL-NAME          AGE
hellofresh-eks-4gm6r   True    True     hellofresh-eks-4gm6r   3h37m
hellofresh-eks-cbtc5   True    True     hellofresh-eks-cbtc5   3h37m

k get RolePolicyAttachment
NAME                   READY   SYNCED   EXTERNAL-NAME                                     AGE
hellofresh-eks-5f6cx   True    True     hellofresh-eks-cbtc5-20231009085914219400000004   3h38m
hellofresh-eks-7pgkn   True    True     hellofresh-eks-cbtc5-20231009085914247400000005   3h38m
hellofresh-eks-dps9h   True    True     hellofresh-eks-4gm6r-20231009085914144600000003   3h38m
hellofresh-eks-g7dpp   True    True     hellofresh-eks-cbtc5-20231009085914115600000001   3h38m
hellofresh-eks-mpqxz   True    True     hellofresh-eks-cbtc5-20231009085914130000000002   3h38m

k get vpc-datasource
NAME                   SYNCED   READY   COMPOSITION                        AGE
hellofresh-eks-sdshc   True     True    vpc-datasource.aws.hellofresh.io   3h38m
team-a-vpc-tf          True     True    vpc-datasource.aws.hellofresh.io   4h21m

k get cluster.eks.aws.upbound.io                                                                                                                                                                                                                             
NAME                   READY   SYNCED   EXTERNAL-NAME          AGE
hellofresh-eks-4xwgt   True    True     hellofresh-eks-4xwgt   3h38m

k get ClusterAuth
NAME                   READY   SYNCED   EXTERNAL-NAME          AGE
hellofresh-eks-5j5md   True    True     hellofresh-eks-5j5md   3h38m

k get NodeGroup
NAME                   READY   SYNCED   EXTERNAL-NAME          AGE
hellofresh-eks-h482g   True    True     hellofresh-eks-h482g   3h38m

k get addon
NAME                   READY   SYNCED   EXTERNAL-NAME                             AGE
hellofresh-eks-7sjn2   True    True     hellofresh-eks-4xwgt:aws-ebs-csi-driver   3h39m

k get OpenIDConnectProvider
NAME                   READY   SYNCED   EXTERNAL-NAME                                                                                                  AGE
hellofresh-eks-7mxgn   True    True     arn:aws:iam::784668270227:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/DF6C23BDE530ABBD0AEBEFBE57729B9B   3h27m

k get providerconfigs.kubernetes.crossplane.io                        
NAME             AGE
hellofresh-eks   3h39m

k get Object
NAME                   KIND        PROVIDERCONFIG   SYNCED   READY   AGE
hellofresh-eks-bcmc4   ConfigMap   hellofresh-eks   True     True    3h39m
hellofresh-eks-ctcq5   ConfigMap   hellofresh-eks   True     True    133m

I assume it happens because state of the Kubernetes providerConfig we use to update aws-auth config map of the new EKS cluster doesn't represent anything

apiVersion: kubernetes.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  annotations:
    crossplane.io/composition-resource-name: kubernetes-provider-config
  creationTimestamp: "2023-10-09T08:58:43Z"
  finalizers:
  - in-use.crossplane.io
  generateName: hellofresh-eks-
  generation: 1
  labels:
    crossplane.io/claim-name: ""
    crossplane.io/claim-namespace: ""
    crossplane.io/composite: hellofresh-eks
  name: hellofresh-eks
  ownerReferences:
  - apiVersion: aws.hellofresh.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: eks
    name: hellofresh-eks
    uid: 9d89b96f-dd1b-4887-85a9-79718a30bfb7
  resourceVersion: "12926672"
  uid: 4eac6e20-43ed-4219-b0af-f419a933c4e8
spec:
  credentials:
    secretRef:
      key: kubeconfig
      name: hellofresh-eks
      namespace: default
    source: Secret
status:
  users: 2

State of the XR

k describe eks hellofresh-eks 
Name:         hellofresh-eks
Namespace:    
Labels:       crossplane.io/composite=hellofresh-eks
Annotations:  uptest.upbound.io/pre-delete-hook: testhooks/delete-release.sh
API Version:  aws.hellofresh.io/v1alpha1
Kind:         eks
Metadata:
  Creation Timestamp:  2023-10-09T08:58:43Z
  Finalizers:
    composite.apiextensions.crossplane.io
  Generation:  26
  Managed Fields:
    API Version:  aws.hellofresh.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"composite.apiextensions.crossplane.io":
        f:labels:
          .:
          f:crossplane.io/composite:
      f:spec:
        f:compositionRef:
          .:
          f:name:
        f:compositionRevisionRef:
          .:
          f:name:
        f:compositionUpdatePolicy:
        f:resourceRefs:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2023-10-09T11:44:46Z
    API Version:  aws.hellofresh.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:uptest.upbound.io/pre-delete-hook:
      f:spec:
        .:
        f:id:
        f:parameters:
          .:
          f:awsAuth:
            .:
            f:mapRoles:
          f:nodes:
            .:
            f:count:
            f:size:
          f:version:
          f:vpcName:
        f:writeConnectionSecretToRef:
          .:
          f:name:
          f:namespace:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2023-10-09T11:44:46Z
    API Version:  aws.hellofresh.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:connectionDetails:
          .:
          f:lastPublishedTime:
        f:eks:
          .:
          f:accountId:
          f:oidc:
          f:oidcUri:
        f:ngRoleArn:
        f:subnetIds:
    Manager:         Go-http-client
    Operation:       Update
    Subresource:     status
    Time:            2023-10-09T12:31:47Z
  Resource Version:  12966256
  UID:               9d89b96f-dd1b-4887-85a9-79718a30bfb7
Spec:
  Composition Ref:
    Name:  eks.aws.hellofresh.io
  Composition Revision Ref:
    Name:                     eks.aws.hellofresh.io-9a0f89d
  Composition Update Policy:  Automatic
  Id:                         team-a
  Parameters:
    Aws Auth:
      Map Roles:  - groups:
  - system:masters
  rolearn: arn:aws:iam::**:role/**
  username: **

    Nodes:
      Count:   3
      Size:    small
    Version:   1.27
    Vpc Name:  crossplane-team-a
  Resource Refs:
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         Role
    Name:         hellofresh-eks-4gm6r
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         RolePolicyAttachment
    Name:         hellofresh-eks-dps9h
    API Version:  aws.hellofresh.io/v1alpha1
    Kind:         vpc-datasource
    Name:         hellofresh-eks-sdshc
    API Version:  eks.aws.upbound.io/v1beta1
    Kind:         Cluster
    Name:         hellofresh-eks-4xwgt
    API Version:  eks.aws.upbound.io/v1beta1
    Kind:         ClusterAuth
    Name:         hellofresh-eks-5j5md
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         Role
    Name:         hellofresh-eks-cbtc5
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         RolePolicyAttachment
    Name:         hellofresh-eks-7pgkn
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         RolePolicyAttachment
    Name:         hellofresh-eks-mpqxz
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         RolePolicyAttachment
    Name:         hellofresh-eks-5f6cx
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         RolePolicyAttachment
    Name:         hellofresh-eks-g7dpp
    API Version:  eks.aws.upbound.io/v1beta1
    Kind:         NodeGroup
    Name:         hellofresh-eks-h482g
    API Version:  eks.aws.upbound.io/v1beta1
    Kind:         Addon
    Name:         hellofresh-eks-7sjn2
    API Version:  iam.aws.upbound.io/v1beta1
    Kind:         OpenIDConnectProvider
    Name:         hellofresh-eks-7mxgn
    API Version:  kubernetes.crossplane.io/v1alpha1
    Kind:         ProviderConfig
    Name:         hellofresh-eks
    API Version:  kubernetes.crossplane.io/v1alpha1
    Kind:         Object
    Name:         hellofresh-eks-bcmc4
    API Version:  kubernetes.crossplane.io/v1alpha1
    Kind:         Object
    Name:         hellofresh-eks-ctcq5
  Write Connection Secret To Ref:
    Name:       hellofresh-eks
    Namespace:  default
Status:
  Conditions:
    Last Transition Time:  2023-10-09T08:58:43Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
    Last Transition Time:  2023-10-09T08:58:43Z
    Reason:                Creating
    Status:                False
    Type:                  Ready
  Connection Details:
    Last Published Time:  2023-10-09T12:31:47Z
  Eks:
    Account Id:  **
    Oidc:        https://oidc.eks.eu-west-1.amazonaws.com/id/**
    Oidc Uri:    oidc.eks.eu-west-1.amazonaws.com/id/**
  Ng Role Arn:   arn:aws:iam::***:role/hellofresh-eks-cbtc5
  Subnet Ids:
    subnet-01a0faa90ec8d8ab4
    subnet-05d37a7fa7e2a177b
Events:
  Type    Reason             Age                     From                                                             Message
  ----    ------             ----                    ----                                                             -------
  Normal  SelectComposition  2m1s (x266 over 3h42m)  defined/compositeresourcedefinition.apiextensions.crossplane.io  Successfully selected composition

What environment did it happen in?

  • Crossplane Version: v1.13.2
  • AWS provider versions: v0.41.0
  • Kubernetes provider version: v0.9.0
  • Kubernetes Version: 1.25
  • Kubernetes Distribution: EKS
@NikitaCloudRuntime NikitaCloudRuntime added the bug Something isn't working label Oct 9, 2023
@phisco
Copy link
Contributor
phisco commented Oct 9, 2023

What do you mean by "wrong"? that the XR is still not ready?

@bobh66
Copy link
Contributor
bobh66 commented Oct 9, 2023

If you create ProviderConfig resources in a Composite you can override the healthchecks so that the state of the ProviderConfig is ignored:

    readinessChecks:
    - type: None

which should allow the Composite to show as Ready = True.

@bobh66
Copy link
Contributor
bobh66 commented Oct 9, 2023

@phisco - should we add the Ready condition to ProviderConfig and always set it to True so we can avoid this issue? It seems like if we can create it from a Composition it should follow the same API as everything else.

@phisco
Copy link
Contributor
phisco commented Oct 9, 2023

I guess that would make sense, @bobh66, WDYT @turkenh @negz? Any other known resource people usually hit this issue with?

Strange it's not mentioning it's waiting for the ProviderConfig in the XR's condition though 🤔

@haarchri
Copy link
Contributor
haarchri commented Oct 9, 2023 •
edited
Loading

@bobh66 is right you need to add this:

    readinessChecks:
    - type: None

https://github.com/upbound/configuration-caas/blob/main/apis/aws/eks/composition.yaml#L349-L350

cross ref: crossplane/crossplane-runtime#418

@phisco in RC-1.14 we have new condition - to show that XR is waiting for ProviderConfig ex.

@NikitaCloudRuntime
Copy link
Author

@haarchri that solved the problem, thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bobh66 @phisco @haarchri @NikitaCloudRuntime