[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support conversion from String to File in WDL #4554

Closed
adamnovak opened this issue Jul 28, 2023 · 1 comment · Fixed by #4589
Closed

Support conversion from String to File in WDL #4554

adamnovak opened this issue Jul 28, 2023 · 1 comment · Fixed by #4589

Comments

@adamnovak
Copy link
Member
adamnovak commented Jul 28, 2023 •
edited
Loading

Some WDL workflows rely on being able to convert between a String and a File:

https://github.com/gatk-workflows/seq-format-conversion/blob/6b69dde1877de997effedd3388aeaca929db04da/paired-fastq-to-unmapped-bam.wdl#L65-L66

This is apparently obvious and unambiguous, but the spec doesn't really say in what filesystem this is supposed to happen, and MiniWDL doesn't seem to magically call the Toil file import.

We should make sure that string to file conversions actually get around to putting the file into the file store so that this workflow can work.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1381
@adamnovak
Copy link
Member Author

I think we need to do one of:

  • Hook the string to file coercion code to do an import, and store the right kind of file path (URL or in-container path, depending on context)
  • Wherever files are used or allowed to move off a node, go through them and look for ones that are un-imported files and import them.

Either way, we need to make sure that wherever a string could be coerced to a file, it is happening on the leader node, except if it is happening in the output section of a task, in which case it has to happen on the task's worker node and deal in in-container paths. I think right now our semantics is that leader jobs may run on the leader; we might have to change that so that they must run on the leader.

@adamnovak adamnovak mentioned this issue Jul 28, 2023
Open
@adamnovak adamnovak mentioned this issue Aug 24, 2023
Open
@adamnovak adamnovak mentioned this issue Sep 19, 2023
Merged
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add String to File functionality into toil-wdl-runner DataBiosphere/toil
1 participant
@adamnovak