You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The overrides ability gives us flexibility to change the container args and the task count on the fly when executing a job but if the number of tasks in that execution is less than the configured parallelism then the job configuration must be updated with a lower parallelism and it defeats the purpose of having one time executions with different amounts of arguments/tasks. We need the parallelism configured because an API we use has a limit on the amount of concurrent requests and we can't just set parallelism to 0.
Describe the solution you'd like
A simple solution would be to allow us to also override the parallelism while overriding containerArgs and taskCount using the execute job with overrides feature.
Describe alternatives you've considered
It could also be possible to implement a dynamic parallelism based on the number of tasks. If the task count is less than the configured parallelism then set the parallelism as the task count.
Additional context
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
The overrides ability gives us flexibility to change the container args and the task count on the fly when executing a job but if the number of tasks in that execution is less than the configured parallelism then the job configuration must be updated with a lower parallelism and it defeats the purpose of having one time executions with different amounts of arguments/tasks. We need the parallelism configured because an API we use has a limit on the amount of concurrent requests and we can't just set parallelism to 0.
Describe the solution you'd like
A simple solution would be to allow us to also override the parallelism while overriding containerArgs and taskCount using the execute job with overrides feature.
Describe alternatives you've considered
It could also be possible to implement a dynamic parallelism based on the number of tasks. If the task count is less than the configured parallelism then set the parallelism as the task count.
Additional context
The text was updated successfully, but these errors were encountered: