[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add a test for metadata of interval #14869

Draft
wants to merge 3 commits into
base: dev
Choose a base branch
from
Draft

add a test for metadata of interval #14869

wants to merge 3 commits into from

Conversation

lldelisle
Copy link
Contributor

Hi,
I am not solving an issue but raising an issue... I think this test is missing and it seems that it is failing locally.
I tested it with planemo with the following test.txt:

read1	chr1	0	2

I did not understand where the functional tests were supposed to find their inputs...

I hope it is still helpful...

For info, when I run it with planemo the json for upload is:

[{"file_type": "interval", "ext": "interval", "name": "test.txt", "dataset_id": 1, "dbkey": "?", "type": "file", "is_binary": false, "link_data_only": "copy_files", "uuid": null, "to_posix_lines": true, "auto_decompress": true, "purge_source": true, "space_to_tab": false, "run_as_real_user": false, "check_content": false, "path": "/tmp/upload_file_data_v24yjg05"}]

No metadata.

On the contrary if I use planemo to test the metadata.xml I get:

[{"file_type": "velvet", "dataset_id": 1, "dbkey": "?", "type": "composite", "metadata": {"base_name": "Example Metadata", "paired_end_reads": "False", "long_reads": "False", "short2_reads": "False"}, "primary_file": "/tmp/upload_auto_primary_file3_c7l7m7", "composite_file_paths": {"Sequences": {"type": "file", "path": "/tmp/upload_file_data_ezc902zl", "name": "files_0|file_data", "purge_source": true, "to_posix_lines": true, "auto_decompress": true, "space_to_tab": false, "uuid": null, "file_type": "", "dbkey": ""}, "Roadmaps": {"type": "file", "path": "/tmp/upload_file_data_t15pez7e", "name": "files_1|file_data", "purge_source": true, "to_posix_lines": true, "auto_decompress": true, "space_to_tab": false, "uuid": null, "file_type": "", "dbkey": ""}, "Log": {"type": "file", "path": "/tmp/upload_file_data_48oof00e", "name": "files_2|file_data", "purge_source": true, "to_posix_lines": true, "auto_decompress": true, "space_to_tab": false, "uuid": null, "file_type": "", "dbkey": ""}}, "composite_files": {"Sequences": {"name": "Sequences", "optional": false, "mimetype": "text/html", "description": "Sequences", "substitute_name_with_metadata": null, "is_binary": false, "to_posix_lines": true, "space_to_tab": false}, "Roadmaps": {"name": "Roadmaps", "optional": false, "mimetype": "text/html", "description": "Roadmaps", "substitute_name_with_metadata": null, "is_binary": false, "to_posix_lines": true, "space_to_tab": false}, "Log": {"name": "Log", "optional": "True", "mimetype": "text/html", "description": "Log", "substitute_name_with_metadata": null, "is_binary": false, "to_posix_lines": true, "space_to_tab": false}}}]

The metadata are set.

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@github-actions github-actions bot added this to the 23.1 milestone Oct 27, 2022
@lldelisle lldelisle mentioned this pull request Oct 28, 2022
Merged
@dannon dannon marked this pull request as draft December 8, 2022 15:28
@lldelisle lldelisle mentioned this pull request Dec 12, 2022
Merged
6 tasks
@bernt-matthias
Copy link
Contributor

I did not understand where the functional tests were supposed to find their inputs...

The test data is in test-data/ and the data in https://github.com/galaxyproject/galaxy-test-data is added to it.

@lldelisle
Copy link
Contributor Author

Yes I discovered it when I developped the converter tool...

@bernt-matthias
Copy link
Contributor

If you want to get this running in the functional tool tests you need to add the tool to https://github.com/galaxyproject/galaxy/blob/776df2a8e9d8872878d1b04f223f8a92057e3f5a/test/functional/tools/samples_tool_conf.xml

@lldelisle
Copy link
Contributor Author

Thanks, I did not know.
I need first to have galaxyproject/galaxy-test-data#23 merged to have a test dataset...

@bernt-matthias
Copy link
Contributor

Funnily for metadata.xml the metadata the json that you provide is not even the metadata of the input, or?

@lldelisle
Copy link
Contributor Author

I meant, when I test: https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/metadata.xml the metadata is correctly formed it contains: "metadata": {"base_name": "Example Metadata" which was what was set as input...

@bernt-matthias
Copy link
Contributor

I need first to have galaxyproject/galaxy-test-data#23 merged to have a test dataset...

I think for the test you can use any tabular file. The column assignment does not need to make sense.

@lldelisle
Copy link
Contributor Author

I think for the test you can use any tabular file. The column assignment does not need to make sense.

You are right. I just used a gtf and could show it is failing:

FAILED test/functional/test_toolbox_pytest.py::TestFrameworkTools::test_tool[metadata_interval/1.0.0-0] - galaxy.tool_util.verify.interactor.JobOutputsError: Output output_of_input_metadata:  different than expected
Expected line '1 4 5' in output ('1 2 3
')

@lldelisle lldelisle mentioned this pull request Dec 13, 2022
Open
@bernt-matthias
Copy link
Contributor

I tried to trace this back: what I can say for now is that this tests fails for all releases since 19.09 .. don't know if it worked for any release before this (seems difficult to run a framework test / planemo test for older releases)

@lldelisle
Copy link
Contributor Author

Thanks for checking

@bernt-matthias
Copy link
Contributor

xref #5029

@lldelisle
Copy link
Contributor Author

Ouah this is a very old issue then

@bernt-matthias
Copy link
Contributor

Just tried to set set_in_upload=True for the MetadataElement .. without success

Found some code here that is responsible for writing metadata to upload_params.json in the job working dir which is only implemented for composite data:

galaxy/lib/galaxy/tools/parameters/grouping.py

Line 646 in 35bf24a

for meta_name, meta_spec in d_type.metadata_spec.items():

Adding an analogous statement to the non-composite branch adds metadata to the file .. but the metadata is still not set.
On the bright side we see that set_in_upload seems to matter.

But I'm not sure yet if upload_params.json is relevant for non-composite datasets, because maybe metadata (in particular base_name might just be a dirty hack:

galaxy/lib/galaxy/tools/parameters/grouping.py

Line 269 in 35bf24a

dataset_name = context.get("files_metadata|base_name", None)

But I also checked to set another metadata of the velvet datatype in the metadata.xml test .. which worked regardless of set_in_upload being False or True. Which indicates that upload_params.json and the code in grouping.py might not be relevant for this problem.

Also noted that I already opened an analogous PR #14306 .. LOL

@lldelisle
Copy link
Contributor Author

Thank you so much for these investigations.

@dannon dannon modified the milestones: 23.0, 23.1 Jan 10, 2023
@mvdbeek mvdbeek removed this from the 23.1 milestone Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/testing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@lldelisle @bernt-matthias @dannon @mvdbeek