There have been similar discussions from the provider-terraform community - see upbound/provider-terraform#189

There is definitely a difference between horizontal scaling of the provider/controller, which would require sharding of the resource UIDs and changes in controller-runtime, and moving to a job-dispatch model as discussed here and in the other issue.

provider-terraform is using the terraform-native locking to prevent multiple commands from running at the same time on the same workspace, so theoretically a dispatch model would not need to change that.

The workaround in provider-terraform is basically to throw more CPUs at the provider, and increase the --max-reconcile-rate accordingly. That has other issues with shared provider caches and cache-locking problems.

I wonder if this is a case where a different provider deployment model would be appropriate, for example something that dispatches Jobs instead of using the standard provider Deployment. That's probably a stretch but maybe something to think about. Maybe @negz has some ideas as well.

Thanks for the link to the other discussion! The idea proposed in that issue was another thing that I considered, and discussed with provider-ansible maintainers, but that seemed like a much larger change to the architecture than the locking model.

In our usage of provider-ansible, we've done pretty much the same thing - increase --max-reconcile-rate and throw more CPU at it. Our use case has reached a point where this isn't sustainable, and some ansible runs take >2hrs. Hence the motivation for this idea - this would allow multiple provider pods to do the work and spread the load.

Can you elaborate a bit more on this?

I wonder if this is a case where a different provider deployment model would be appropriate, for example something that dispatches Jobs instead of using the standard provider Deployment.