It might be useful to refer to a real-world example that might lead to this scenario.

sure, @turkenh or @haarchri do you have some better example from the field?

Yes, a more complex example would be nice to see, not necessarily a real world example, but at least a bigger toy example would be helpful. I'm not sure how you are envisioning this new API to be used in the context of Compositions. Should users define this new resource as part of their Composition? How will that look like? So users will most probably have to define Usage resources as defined by #4215 to achieve deletion ordering and then this new object to define mappings between values? how's that going to look like?

@phisco, we are not aware of many real-world use cases where a value reference requires defining a deletion ordering, see this comment.

However, I agree that having a dedicated section for Compositions makes sense. It would especially be helpful to see the interplay between the existing patches in compositions and the one we're proposing here.

Similar issues have been discussed regarding selecting EnvironmentConfigs and selection in general - there needs to be a way to ensure deterministic selection to avoid flapping between two or more values.

• this proposal seems to make the same mistakes that the composition patch and transformation (P&T) has done, and that eventually led to the invention of functions:
  • desirable transformations are open ended and are never enough
  • patching by field paths is a very low-level mechanism, operating on low-level data-type rather than being MR kind aware
  • references don't solve the order problem: (a) no way to order without the MR author having prepared for a reference field (b) dependencies might be more than just a field in an object (think of probes).
• in general, this proposal seems to clone the P&T idea into the object system. Do we really want that? Then the new approach competes with compositions. This is kind of the goal stated in the beginning of the doc. I miss a discussion why this is an acceptable goal, i.e. a motivation section.
• the core motivation behind this proposal seems to be that compositions are too slow (e.g. not watch-driven, but run in 60s interval). Maybe we should fix the latter.

+1
This feels like it's adding one more layer of indirection to Compositions and feels overlapping with both the Usage resource introduced by #4215 and a little bit with observe only resources too.

this proposal seems to make the same mistakes that the composition patch and transformation (P&T) has done, and that eventually led to the invention of functions

FWIW, some context here, we knew we'll need functions even before the design of the Composition itself but it was always about how far we can go with the least logic so that you can go to production without having to deal with a whole software stack to write reconciliation logic. And we wanted to let the use cases pop up over time so we can leave the minimum amount of logic that you users would have to use functions for instead of having to go all in on writing composition in a programming language.

Then the new approach competes with compositions.

Agreed, we need to talk more about the cases where using bi-directional patches in composition wouldn't be suitable.

It would be good to define and agree on the problem before talking about the solution.

Let's take the VPC / Subnet example. If someone wants to create a Subnet in a VPC created/managed by XP, they need a mechanism to read the id from VPC and put it into the Subnet spec. Today, this is possible in one of the following two ways:

a. Using cross-resource references where they can either use MR name for VPC (e.g. vpcIdRef.name) or labels/controllerRef (e.g. vpcIdRef.selector).
b. Using composition patches, but with the following limitations:

• VPC and Subnet must exist in the same composition.
• id needs to be patched to XR first, (e.g. status.vpcId) and then patched to the Subnet (toFieldPath: spec.forProvider.vpcId)

Typically, in compositions we see that the option (a) is used with matchControllerRef where the id of the VPC with the same controller ref (owned by the same XR) is extracted and used in Subnet:

            vpcIdSelector:
              matchControllerRef: true

Then, what is wrong? There are two problems invalidating both of the available options as such:

option (a): Not every MR field directly maps to a single type, so existing XRR falls short. There are a lot of issue references in the original ticket, but one example is the destinationArn on a SubscriptionFilter resource where destination could be one of the multiple types.
option (b): Resources may not be part of the same composite.

So, we need a solution here. If someone creates a SubscriptionFilter and a DeliveryStream (using the same example) with Crossplane but not necessarily within the same composition, they need a way to fill spec.forProvider.destinationArn on SubscriptionFilter with the arn of the DeliveryStream they intend to.

this proposal seems to make the same mistakes that the composition patch and transformation (P&T) has done, and that eventually led to the invention of functions:

I agree that the problem we're trying to solve here is similar to what (P&T) has done, but there is that fundamental difference that makes it impossible to solve with compositions or functions, which is resources may not be part of the same composite. I can see two ways to overcome this, both with some implications:

• If you want to patch from a resource that is not part of the same composition, define an ObserveOnly resource in the same composition and patch from it.
• Functions could access to k8s API and read anything it wants without being limited to observed state.

desirable transformations are open ended and are never enough

Yes. While we lack very basics today, like take that ARN and put it here, it would eventually need more and more, just like we see in composition patches. Leveraging and relying on some external,tool might help here.

references don't solve the order problem: (a) no way to order without the MR author having prepared for a reference field (b) dependencies might be more than just a field in an object (think of probes).

I am not aware of a real (creation) ordering problem today where we are good enough with eventual consistency.
We had the deletion ordering problem and tackling it with the Usage type.
I am not claiming it will not complicate anything but we can still stay on the "declarative" world by saying, this value needs to be constructed this way and eventually patched there.

in general, this proposal seems to clone the P&T idea into the object system. Do we really want that?

Not sure if the solution should be the one proposed here, but we need a solution to the problem defined above.

the core motivation behind this proposal seems to be that compositions are too slow (e.g. not watch-driven, but run in 60s interval). Maybe we should fix the latter.

Not sure how you came up with this assumption, but I am not aware of such a motivation for Generic XRR. IIRC, compositions not being watch-driven came up recently in the composition function (re)design; however, Generic XRR has been an open issue for more than 2 years. One can claim that, if we could live without it for 2 years, then maybe it is not that an important problem to solve :) I would love to hear from the community on that, and I believe it is not a blocker mostly because there are workaround/hacks there which could unblock use cases but ultimately complicate things further.

I'd like to add a slightly different perspective based on my experience using EnvironmentConfigs. I think there is a fair amount of overlap in the usage scenarios, and for me personally the EnvironemtConfig model is simpler and easier to understand and use than any of the other proposals for separate resource references that I have seen.

One of the common use cases we have found for EnvironmentConfig is making information available from one Composite to another. If Composite A creates a set of resources and wants to make some information regarding those resources available to any other Composite/Composition that may need them, it can create an EnvironmentConfig object with the appropriate labels and the information to be "published". Other Composites can retrieve these EnvironmentConfigs as needed in their Compositions using the appropriate label selectors.

The process of finding the appropriate EnvironmentConfigs and loading them into the Composition environment creates what is essentially a runtime context for that Composite. The user (composition author) can leverage this context to pass information between resources and can reference external data using EnvironmentConfigs. The structure of the local context is defined by the author and information is placed in specific locations by name. This model works extremely well and provides necessary functionality.

So why limit the environment external data references to EnvironmentConfigs? Why not allow the environment specification to include references to any existing Kubernetes object, which can be retrieved by the same mechanism that resolves EnvironmentConfigs today? If an author wants to retrieve data from an existing XR or MR or ConfigMap or Secret they could specify the G/V/K information required to find the object along with a name or label selector, and a path within the object to the data they want, and that information can be populated in the environment in the specified location.

From the author's perspective this would let me put all of my external data references in one place, and they can be retrieved dynamically at runtime as needed if they exist, or they can eventually reconcile as they become available.

I think one of the overarching goals needs to be keeping the user experience as simple as possible, and requiring a specific resource definition for every reference feels like it is adding complexity for the user and making the composition more difficult to develop and maintain.

Per #1770 (comment) I share the concern that generic references grows to be something like P&T Composition.

Also, per #1770 (comment) I'm pretty confident that today's hardcoded style of references isn't going to scale. It's too manual to support, too inflexible for users, and creates a coupling between providers when one provider references a type defined by another.

I could be convinced that the answer is to just not have MR-level references at all if we can find a clean way to address this at the Composition level. I would expect such an approach to be controversial though. I suspect there's a not-insignificant amount of folks who don't use Composition today who would be forced to if Composition was the only way to derive one

I could be convinced that the answer is to just not have MR-level references at all if we can find a clean way to address this at the Composition level. I would expect such an approach to be controversial though. I suspect there's a not-insignificant amount of folks who don't use Composition today who would be forced to if Composition was the only way to derive one

This proposal talks about that path as well. No matter what strategy/convention we would like to apply to MR-level reference, at the end of the day, it needs to be implemented by providers. On the other hand, solving the problem via compositions requires no changes at provider side.

Is it worth a community poll of some sort to try to determine how widespread the use of MR-only deployments is? I'm reluctant to push more composition-like functionality into the MR since that's what we have Compositions for. I agree that having MRs is great, but I think the major benefit of Crossplane is the modeling and abstraction provided by Composites/Compositions and some (most?) types of cross-resource functionality should be based on that.

Compositions today mix specification, relationships and orchestration:

1. specification: specify how objects look like
2. relationships: Patch&Transformations (P&T) establish field level dependencies
3. orchestration: P&T is used to define order, both by (a) implicit field dependencies of required fields and (b) by hacks to intentionally break objects or pause objects before some condition is true.

In that light, I would consider the goal of this proposal to promote (2) to the object system, and primarily leave compositions for (1) and (3). This bigger "philosophical" context must be clearer in this doc, maybe in some motivation section.

There is also a connection between (2) and (3a). An asynchronous controller cannot solve the required fields problem, i.e. late initialization. This means that we encourage late initialization more and more. Late initialization has the problem of more states to reason about. Resources will not be completely or correctly specified fromt he beginning, i.e. the creation of the object. This obviously has consequences: less invariants/consistency, less type-safety.

Is that ^^ a higher level goal? Isn't being more dynamic, less consistent, less type-safety a step into the wrong direction? Thinking about DevEx and "Crossplane is complex".

I think It is hard to take (2) out of compositions. At a minimum, we will always need to patch from XR to Composed resources. I would rather see this proposal as a solution to the limitations we have in defining relationships:

a. XR to Composed
b. Composed to XR
c. Composed to Composed
d. External MR/XR to Composed

(a) and (b) are fundamental for compositions, and I don't think aiming to take it out makes sense. (c) is only possible over (b+a), and (d) is not possible today (without hacks). So, the impact of this proposal on compositions is to enable (d) and (c). Of course, the ability to refer generically would have benefits with compositions not in the play.

I agree that clarifying these in a motivation section would be great.

This means that we encourage late initialization more and more. Late initialization has the problem of more states to reason about. Resources will not be completely or correctly specified fromt he beginning, i.e. the creation of the object. This obviously has consequences: less invariants/consistency, less type-safety.

I would rather state it as we rely more on eventual consistency for more flexibility. In an ideal world, I agree that more consistency and full type-safety would be great, but this ends up with limitations, i.e. we already have a referencing solution with type-safety, but it has limitations mentioned in the doc.

Yes, a more complex example would be nice to see, not necessarily a real world example, but at least a bigger toy example would be helpful. I'm not sure how you are envisioning this new API to be used in the context of Compositions. Should users define this new resource as part of their Composition? How will that look like? So users will most probably have to define Usage resources as defined by #4215 to achieve deletion ordering and then this new object to define mappings between values? how's that going to look like?

I think it's worth having a paragraph here to elaborate on why we paused the work back then and whether the same reasons still hold or not. The proposed changes were to add complexity at a foundational level that is managed resources and I had wanted to make sure we add it for good reason. On the other hand, the proposed approach here is less intrusive with its own CRD, which still requires some justification but maybe not as much. And Crossplane has come a long way since then.

To shed some light as the author of the that draft, the following is a list of the motivations and how they compare to today:

• The main reason we needed generic references was that we did not have a good way to generate references in Terrajet (Upjet today) and writing code for every reference field was just too much - imagine custom code for every reference field of thousands of CRDs. But then we did figure out a good way to generate the code, hence the only thing left was whether we tell the generator which fields reference what in development time or user tells every time the reference something. We decided to take on the burden once so users don't have to every time they need to reference.
• Another reason was people asked for retrieving information from resources that they don't want to manage via that specific Crossplane instance.
  • Some of this was common MRs that were managed by something else or another Crossplane instance, like VPC that should be used by resources in different Crossplane instances. This is largely solved by observe-only resources today.
  • The rest was arbitrary metadata information, like cost center tags etc, that they wanted to have on MRs. If you use Composition, we have EnvironmentConfig today for that use case. However, outside of composition, generic references would be needed at MR level.
• The Route53 thing is still not solved unless you use Composition but it may not be very common.

I'll recap my understanding of why we want generic cross-resource references here, since it's relevant to what @muvaf wrote above. I touched on some of this at #1770 (comment).

The current spec.forProvider.fooRef implementation:

1. Requires all possible references to be manually identified and declared at build time.
2. Is inflexible - users can only reference types the provider developers declared, in the ways they declared.
3. Requires the referring provider to import any types they refer to.

3 is of particular interest for provider families, because it creates a potential version dependency between two providers in a family. For example if provider-aws-eks v0.42.0 is compiled against the types from provider-aws-ec2 v0.42.0, there's a chance that installing provider-aws-ec2 v0.50.0 will break references. It would be ideal if there was no version coupling between providers within a family.

this proposal seems to make the same mistakes that the composition patch and transformation (P&T) has done, and that eventually led to the invention of functions

FWIW, some context here, we knew we'll need functions even before the design of the Composition itself but it was always about how far we can go with the least logic so that you can go to production without having to deal with a whole software stack to write reconciliation logic. And we wanted to let the use cases pop up over time so we can leave the minimum amount of logic that you users would have to use functions for instead of having to go all in on writing composition in a programming language.

Then the new approach competes with compositions.

Agreed, we need to talk more about the cases where using bi-directional patches in composition wouldn't be suitable.

I think It is hard to take (2) out of compositions. At a minimum, we will always need to patch from XR to Composed resources. I would rather see this proposal as a solution to the limitations we have in defining relationships:

a. XR to Composed
b. Composed to XR
c. Composed to Composed
d. External MR/XR to Composed

(a) and (b) are fundamental for compositions, and I don't think aiming to take it out makes sense. (c) is only possible over (b+a), and (d) is not possible today (without hacks). So, the impact of this proposal on compositions is to enable (d) and (c). Of course, the ability to refer generically would have benefits with compositions not in the play.

I agree that clarifying these in a motivation section would be great.

This means that we encourage late initialization more and more. Late initialization has the problem of more states to reason about. Resources will not be completely or correctly specified fromt he beginning, i.e. the creation of the object. This obviously has consequences: less invariants/consistency, less type-safety.

I would rather state it as we rely more on eventual consistency for more flexibility. In an ideal world, I agree that more consistency and full type-safety would be great, but this ends up with limitations, i.e. we already have a referencing solution with type-safety, but it has limitations mentioned in the doc.

nit: this goal is implicitly included in the above one.

It would be good to define and agree on the problem before talking about the solution.

Let's take the VPC / Subnet example. If someone wants to create a Subnet in a VPC created/managed by XP, they need a mechanism to read the id from VPC and put it into the Subnet spec. Today, this is possible in one of the following two ways:

a. Using cross-resource references where they can either use MR name for VPC (e.g. vpcIdRef.name) or labels/controllerRef (e.g. vpcIdRef.selector).
b. Using composition patches, but with the following limitations:

• VPC and Subnet must exist in the same composition.
• id needs to be patched to XR first, (e.g. status.vpcId) and then patched to the Subnet (toFieldPath: spec.forProvider.vpcId)

Typically, in compositions we see that the option (a) is used with matchControllerRef where the id of the VPC with the same controller ref (owned by the same XR) is extracted and used in Subnet:

            vpcIdSelector:
              matchControllerRef: true

Then, what is wrong? There are two problems invalidating both of the available options as such:

option (a): Not every MR field directly maps to a single type, so existing XRR falls short. There are a lot of issue references in the original ticket, but one example is the destinationArn on a SubscriptionFilter resource where destination could be one of the multiple types.
option (b): Resources may not be part of the same composite.

So, we need a solution here. If someone creates a SubscriptionFilter and a DeliveryStream (using the same example) with Crossplane but not necessarily within the same composition, they need a way to fill spec.forProvider.destinationArn on SubscriptionFilter with the arn of the DeliveryStream they intend to.

this proposal seems to make the same mistakes that the composition patch and transformation (P&T) has done, and that eventually led to the invention of functions:

I agree that the problem we're trying to solve here is similar to what (P&T) has done, but there is that fundamental difference that makes it impossible to solve with compositions or functions, which is resources may not be part of the same composite. I can see two ways to overcome this, both with some implications:

• If you want to patch from a resource that is not part of the same composition, define an ObserveOnly resource in the same composition and patch from it.
• Functions could access to k8s API and read anything it wants without being limited to observed state.

desirable transformations are open ended and are never enough

Yes. While we lack very basics today, like take that ARN and put it here, it would eventually need more and more, just like we see in composition patches. Leveraging and relying on some external,tool might help here.

references don't solve the order problem: (a) no way to order without the MR author having prepared for a reference field (b) dependencies might be more than just a field in an object (think of probes).

I am not aware of a real (creation) ordering problem today where we are good enough with eventual consistency.
We had the deletion ordering problem and tackling it with the Usage type.
I am not claiming it will not complicate anything but we can still stay on the "declarative" world by saying, this value needs to be constructed this way and eventually patched there.

in general, this proposal seems to clone the P&T idea into the object system. Do we really want that?

Not sure if the solution should be the one proposed here, but we need a solution to the problem defined above.

the core motivation behind this proposal seems to be that compositions are too slow (e.g. not watch-driven, but run in 60s interval). Maybe we should fix the latter.

Not sure how you came up with this assumption, but I am not aware of such a motivation for Generic XRR. IIRC, compositions not being watch-driven came up recently in the composition function (re)design; however, Generic XRR has been an open issue for more than 2 years. One can claim that, if we could live without it for 2 years, then maybe it is not that an important problem to solve :) I would love to hear from the community on that, and I believe it is not a blocker mostly because there are workaround/hacks there which could unblock use cases but ultimately complicate things further.

I agree - any time we fallback from one thing to another we risk the possibility that on the next reconciliation the thing we fell-back-from will be there and will change the behavior in unexpected ways.

Also overlaps in names between things we are referring to and the name of the Referrable could cause strange behaviors.

Similar issues have been discussed regarding selecting EnvironmentConfigs and selection in general - there needs to be a way to ensure deterministic selection to avoid flapping between two or more values.

This may cause confusion when users are unable to delete resources that they are not aware are being referenced by other resources, Traceability and verbose error messages will be important for user experience.

I'd like to add a slightly different perspective based on my experience using EnvironmentConfigs. I think there is a fair amount of overlap in the usage scenarios, and for me personally the EnvironemtConfig model is simpler and easier to understand and use than any of the other proposals for separate resource references that I have seen.

One of the common use cases we have found for EnvironmentConfig is making information available from one Composite to another. If Composite A creates a set of resources and wants to make some information regarding those resources available to any other Composite/Composition that may need them, it can create an EnvironmentConfig object with the appropriate labels and the information to be "published". Other Composites can retrieve these EnvironmentConfigs as needed in their Compositions using the appropriate label selectors.

The process of finding the appropriate EnvironmentConfigs and loading them into the Composition environment creates what is essentially a runtime context for that Composite. The user (composition author) can leverage this context to pass information between resources and can reference external data using EnvironmentConfigs. The structure of the local context is defined by the author and information is placed in specific locations by name. This model works extremely well and provides necessary functionality.

So why limit the environment external data references to EnvironmentConfigs? Why not allow the environment specification to include references to any existing Kubernetes object, which can be retrieved by the same mechanism that resolves EnvironmentConfigs today? If an author wants to retrieve data from an existing XR or MR or ConfigMap or Secret they could specify the G/V/K information required to find the object along with a name or label selector, and a path within the object to the data they want, and that information can be populated in the environment in the specified location.

From the author's perspective this would let me put all of my external data references in one place, and they can be retrieved dynamically at runtime as needed if they exist, or they can eventually reconcile as they become available.

I think one of the overarching goals needs to be keeping the user experience as simple as possible, and requiring a specific resource definition for every reference feels like it is adding complexity for the user and making the composition more difficult to develop and maintain.

what happens if multiple objects are selected with labels? Reminds me of #3981 when we thought about similar quite late in the implementation

Let's nominate two specific Crossplane maintainers to make it clear who we expect to approve this design.

@crossplane/crossplane-maintainers please raise your hand :)

Count me in

Thanks @bobh66

Making sure I understand: by "the proposed solution" do you mean the pattern you describe above that requires using a Composition?

I would rephrase it as "this solution" so it's clear what it is referring to.

I'll recap my understanding of why we want generic cross-resource references here, since it's relevant to what @muvaf wrote above. I touched on some of this at #1770 (comment).

The current spec.forProvider.fooRef implementation:

1. Requires all possible references to be manually identified and declared at build time.
2. Is inflexible - users can only reference types the provider developers declared, in the ways they declared.
3. Requires the referring provider to import any types they refer to.

3 is of particular interest for provider families, because it creates a potential version dependency between two providers in a family. For example if provider-aws-eks v0.42.0 is compiled against the types from provider-aws-ec2 v0.42.0, there's a chance that installing provider-aws-ec2 v0.50.0 will break references. It would be ideal if there was no version coupling between providers within a family.

Out of scope for this design, but FYI the presence of this field is a bit of a problem - see crossplane/crossplane-runtime#440.

Does this mean that provider developers still need to explicitly identify and declare what fields can be set by a reference at build time? For example here the developer would need to identify that sourceLocationArn could reference another type, and thus make sure sourceLocationArnRef existed.

Does this mean that provider developers still need to explicitly identify and declare what fields can be set by a reference at build time? For example here the developer would need to identify that sourceLocationArn could reference another type, and thus make sure sourceLocationArnRef existed.

There is no change from the current user experience. The only change is that some ref fields can take more than just name to refer the external objects.

My understanding so far (and chatting with @muvaf) is that we have automated that with Upjet, but I could be ofcourse wrong here.

My understanding so far (and chatting with @muvaf) is that we have automated that with Upjet, but I could be ofcourse wrong here.

I thought we'd automated generating the code for references, but not automated the process of identifying which fields can be set using a reference. It's possible I'm wrong too though - CC @ulucinar and @muvaf.

Either way though, I suspect only being able to use references for certain fields will be too limiting. If a field can reference an arbitrary type, how would we know at development which fields should be able to be set by a reference and which can't? Do we think that we'll be able to predict all the fields users might want to derive using a reference at provider development time?

Either way though, I suspect only being able to use references for certain fields will be too limiting. If a field can reference an arbitrary type, how would we know at development which fields should be able to be set by a reference and which can't? Do we think that we'll be able to predict all the fields users might want to derive using a reference at provider development time?

My understanding is that providers generated with the help of Upjet have mapped all fields that can reference external values, correct @ulucinar? These fields are all visible in MR schemas, hence easy discoverable for consumers. Other, non-upjet based providers needs to references in their MR schema in some way as well.

@haarchri, how is your experience? Did you stumble across some fields that should be referable, but they are not according to MR schema?

Alper can correct me if I am wrong, but AFAIK, upjet infers some of those references by parsing their documentation and the examples there but not all. For unidentified ones, it is still a manual process to configure them.

Yes, correct:

My understanding is that providers generated with the help of Upjet have mapped all fields that can reference external values, correct @ulucinar?

We attempt to deduce the reference parameters from the examples provided in the Terraform docs and this does not result in the generation of all reference parameters. For instance, none of the provided example Terraform configurations could be utilizing a possible reference. We do not attempt to parse the argument documentation for this purpose, only the provided HCL configurations are used.

Thus, we have some automation for determining the reference parameters themselves on top of the reference implementation generator in angryjet. But, it's not 100% reliable, optional (an upjet-based provider's maintainer can opt out not to generate the reference parameters) and to the best of my knowledge, is only applicable for upjet-based providers.

@negz, are you suggesting a referencing scheme where, at runtime, we can declare a reference whose source & target are arbitrary (no a priori knowledge about who can reference what), so that at build time, we don't need to know anything about the references?

My understanding is that the proposed approach here makes the reference source (where the data will be copied from) dynamic (GVK+name+fieldpath) but the reference target (where the data will be copied into) is still static, i.e., you need to know the target at build time.

Per #1770 (comment) I share the concern that generic references grows to be something like P&T Composition.

Also, per #1770 (comment) I'm pretty confident that today's hardcoded style of references isn't going to scale. It's too manual to support, too inflexible for users, and creates a coupling between providers when one provider references a type defined by another.

I could be convinced that the answer is to just not have MR-level references at all if we can find a clean way to address this at the Composition level. I would expect such an approach to be controversial though. I suspect there's a not-insignificant amount of folks who don't use Composition today who would be forced to if Composition was the only way to derive one

I wonder if enterprise users may ask for fine-granular access policies in the future. Currently, access is mutually granted to the (service accounts of the) providers A & B, if A & B are in the same family, even if their MRs are not supposed to refer to each other. This's been a natural evolution from the monoliths.

If I understand this correctly:

• There exists a cross-provider resource reference A->B (a resource in provider A references a resource in provider B)
• Because it's cross-provider, the Crossplane RBAC manager cannot install the rules that will allow A's service account to read B's resources
• The user is not willing to add the RBAC rules herself
• Instead provisions a Referable object
• The Referable controller is running as a core Crossplane controller and has read access to B's resources: Do we assume here source GVK will be a managed resource?
• Referable controller exposes the extracted value in status or reports an error if the extraction fails for some reason
• All installed provider's service accounts have read access to all the Referable objects in the cluster
• Provider A copies the data from the Referable object via a GenericReference

So this will allow data transfer across all MRs in the cluster. Do I understand this correctly?

I assume the suggestion is to have this controller running as part of the core Crossplane because in the implementation section, it's mentioned under the core Crossplane. Could the source provider (provider B in the above discussion) be publishing the data for improved security? But still, I wonder if some enterprise users may not like this scheme if I understand it correctly if we implicitly enable such transfers. What do you folks think?

Maybe I'm missing this: How can custom transformation/extraction logic on the data path from source to target be handled? We are preserving the reference.ResolutionRequest.Extract, right? This proposal is mainly for making the reference target dynamic and rest of the data path is (mostly) untouched?

I would rephrase it as "this solution" so it's clear what it is referring to.