[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Design Document for Cross-Resource Dependencies #4100

Closed
turkenh opened this issue May 24, 2023 · 4 comments
Closed

Design Document for Cross-Resource Dependencies #4100

turkenh opened this issue May 24, 2023 · 4 comments
Assignees
Labels
design enhancement New feature or request
Milestone

Comments

@turkenh
Copy link
Member
turkenh commented May 24, 2023

What problem are you facing?

As the project continues to evolve, we have identified two prominent patterns of dependencies between managed resources:

  • Referencing arbitrary fields from another resource, also known as generic resource references. For instance, a subnet that requires the ID of a VPC.
  • Depending on the existence of another resource for successful operation, also known as deletion ordering. An example of this is a Helm release deployed into a Kubernetes cluster managed by Crossplane.

After initial discussions and investigations (#1770, #3393 and this doc) we have come to believe that both types of dependencies could be addressed through a unified design by introducing a new type as proposed here.

How could Crossplane help solve your problem?

Write a design doc for cross-resource dependencies.

@turkenh turkenh added the enhancement New feature or request label May 24, 2023
@turkenh turkenh self-assigned this May 24, 2023
@turkenh turkenh added the roadmap Issues that have priority and are included in the roadmap, or are candidates to add to the roadmap label May 24, 2023
@jbw976 jbw976 added this to the v1.13 milestone May 24, 2023
@turkenh
Copy link
Member Author
turkenh commented May 30, 2023

I've put some thinking on what an API would look like for a generic type that would solve both the deletion ordering and generic referencing starting from the one proposed here.

# A Reference used only for deletion ordering.
kind: Reference
metadata:
  name: release-uses-cluster
spec:
  # A 'blocking' reference prevents all of the 'to' (Cluster) resources from being
  # deleted until the 'from' resource (the Release) is gone.
  deletion: Blocking
  from:
    apiVersion: helm.crossplane.io/v1beta1
    kind: Release
    name: my-prometheus-chart
  to:
  - apiVersion: eks.upbound.io/v1beta1
    kind: Cluster
    name: my-cluster
# A reference used only to derive field values.
apiVersion: crossplane.io/v1alpha1
type: Reference
metadata:
  name: subnet-to-vpc
spec:
  # A 'non-blocking' reference is the default. The VPC can be deleted before
  # the Subnet.
  deletion: NonBlocking
  from:
    apiVersion: network.aws.crossplane.io/v1alpha2
    kind: Subnet
    name: my-subnet
    # Note the optional fieldPath is specified here and below.
    fieldPath: spec.forProvider.vpcId
  to:
  # An array to support the case where a reference is derived from multiple
  # fields in multiple objects? This would likely end up pretty close to
  # Composition patches, with transforms etc. We'd probably want to support
  # selectors here too, not only explicit references.
  - apiVersion: network.aws.crossplane.io/v1alpha2
    kind: VPC
    name: my-vpc
    fieldPath: metadata.annotations[crossplane.io/external-name]

The first issue I was having was the confusion with the reversed direction of the from/to fields compared to the one that my brain used to (from the composition patches). Especially when I want to include transforms, then it gets more confusing, e.g. spec.to transforms into spec.from.

When I imagine the future of this API with similar support as composition patches with transforms, I ended up introducing a new type field to support functionality similar to Combine typed patches and used this as an advantage to fix confusion about direction mentioned above.

apiVersion: apiextensions.crossplane.io/v1alpha1
type: Reference
metadata:
  name: subnet-to-vpc
spec:
  type: PatchesFieldPath
  patchesFieldPath:
    from:
      apiVersion: network.aws.crossplane.io/v1alpha2
      kind: VPC
      name: my-vpc
      fieldPath: metadata.annotations[crossplane.io/external-name]
    to:
      apiVersion: network.aws.crossplane.io/v1alpha2
      kind: Subnet
      name: my-subnet
      # Note the optional fieldPath is specified here and below.
      fieldPath: spec.forProvider.vpcId
    transforms:
      - type: string
        string:
          fmt: "id:%s"
apiVersion: apiextensions.crossplane.io/v1alpha1
type: Reference
metadata:
  name: subnet-to-vpc
spec:
  type: PatchesWithCombine
  patchesWithCombine:
    variables:
      from:
        - apiVersion: s3.aws.crossplane.io/v1beta1
          kind: Bucket
          name: my-bucket
          fieldPath: metadata.annotations[crossplane.io/external-name]
        - apiVersion: s3.aws.crossplane.io/v1beta1
          kind: Bucket
          name: my-bucket
          fieldPath: metadata.annotations[crossplane.io/external-name]
    strategy: string
    to:
      apiVersion: iam.aws.crossplane.io/v1beta1
      kind: Policy
      name: my-policy
      fieldPath: spec.forProvider.document
    string:
        fmt: |
          {
            "Version": "2012-10-17",
            "Statement": [
              {
                "Effect": "Allow",
                "Action": [ "s3:*" ],
                "Resource": [
                  "arn:aws:s3:::%s",
                  "arn:aws:s3:::%s/*"
                ]
              }
            ]
          }

I don't know the exact details of the reasoning behind the design of the API for Combine* type patches and but if it were purely for backward compatibility, we could still come up with a more straightforward API here without requiring to introduce the type field.

However, at this point, I am starting to believe the patching case (generic cross-resource referencing) would require more consideration and grow in a different direction, not necessarily related to the deletion-ordering use case not only API-wise but also implementation-wise.

So, I am questioning whether we would really want to solve the two use cases with one API:

  • Are there enough actual use cases that would benefit from combining both with a single API? e.g. I want the resource that I patched from, not deleted before the resource I patched to.
  • Would the API/implementation be cleaner and more explicit once we have separate APIs?

@jbw976 jbw976 added design and removed roadmap Issues that have priority and are included in the roadmap, or are candidates to add to the roadmap labels Jun 15, 2023
@turkenh
Copy link
Member Author
turkenh commented Jun 21, 2023

We are mostly aligned that using separate API types for deletion ordering and generic cross-resource referencing is the way to go. With the design PR for deletion ordering opened, proposing a Usage type dedicated to deletion ordering, I am closing this issue.

@turkenh turkenh closed this as completed Jun 21, 2023
@yhrn
Copy link
yhrn commented Jun 21, 2023

@turkenh I was just going to post here when the issue was closed. Is there now another issue to track for "generic cross-resource referencing"?

Generic cross-resource referencing is really the most important part for us. We're building the infrastructure management part of our IDP around Crossplane. Our main cloud vendor i Azure, which means that almost every XRD needs to a reference to a resource group claim. Plus other cases like a PostgreSQL Database referencing the PostgreSQL Server it's in.

My gut feeling was that it would make sense to have a single API for them since having a reference from one resource to another typically means that you want the former to be deleted before the latter. This holds true for for the examples I gave above, i.e. Azure resource groups should be deleted after the resources in them, a PostgreSQL Database should be deleted before the PostgreSQL Server it's in. It might be though that this will be handled by Azure, i.e Azure might just refuse to delete a non-empty resource groups, but it doesn't feel great to have to depend on the downstream behavior, which probably varies a lot.

@turkenh
Copy link
Member Author
turkenh commented Jun 21, 2023

Is there now another issue to track for "generic cross-resource referencing"?

Yes. This was a feature being tracked already here and here.

My gut feeling was that it would make sense to have a single API for them since having a reference from one resource to another typically means that you want the former to be deleted before the latter.

While this sounds valid in theory, we are not aware of any real use cases where you need to define a deletion ordering between two cloud resources, especially when they are part of the same cloud provider. For example, when you compose an Azure ResourceGroup together with an Azure Database and delete the composite deleting them together, you won't get a rejection with the deletion of the Database, which is handled by the cloud provider, as you stated.
We observed the deletion ordering becoming a problem primarily with the two variants mentioned here.

With that said, I acknowledge that there could be cases where you want to block deletion of the resource you patched from, but it is not common enough to limit ourselves to building a combined API and go with a more explicit approach. One can always define a Usage in addition to the API for generic referencing whenever needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design enhancement New feature or request
Projects
Status: Done
Development

No branches or pull requests

3 participants