[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated Python and PyMC, removed TensorFlow, and added PyTorch in conda environment. #8561

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Updated Python and PyMC, removed TensorFlow, and added PyTorch in conda environment. #8561

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
aff4ea2
Updated Python and PyMC, removed TensorFlow, and added PyTorch in Doc…
samuelklee Dec 14, 2023
df2d2a4
Updated gCNV code to account for changes from PyMC3/Theano to PyMC/Py…
samuelklee Dec 14, 2023
cb8b9f7
Updated gCNV integration tests.
samuelklee Dec 14, 2023
af4f9f1
Updated gCNV WDL tests.
samuelklee Dec 14, 2023
84de6b7
Updated other tests and tools affected by environment changes.
samuelklee Dec 14, 2023
e32cb02
Reverted posterior sampling to online implementation.
samuelklee Apr 3, 2024
4dc5caa
Updated localDevCondaEnv task in build.gradle.
samuelklee Apr 25, 2024
5a66eac
Addressed review comments and cleaned up TODOs.
samuelklee Jul 1, 2024
260f8fc
Released gatkbase-3.3.0 and updated Dockerfile.
samuelklee Jul 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated other tests and tools affected by environment changes.
  • Loading branch information
@samuelklee
samuelklee committed Jul 1, 2024
commit 84de6b77a012d6cfa87c817201131200523e2ee0
2 changes: 1 addition & 1 deletion .github/workflows/gatk-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
wdlTest: [ 'RUN_CNV_GERMLINE_COHORT_WDL', 'RUN_CNV_GERMLINE_CASE_WDL', 'RUN_CNV_SOMATIC_WDL', 'RUN_M2_WDL', 'RUN_CNN_WDL', 'RUN_VCF_SITE_LEVEL_FILTERING_WDL' ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There may already be a WDL for the NVIDIA CNN in the Terra workspace, but this will be part of the CNNScoreVariants Deprecation Plan

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@droazen leaving this as a reminder that we'll need to deal with the CNN tests/code/etc. disabled here in a subsequent PR.

wdlTest: [ 'RUN_CNV_GERMLINE_COHORT_WDL', 'RUN_CNV_GERMLINE_CASE_WDL', 'RUN_CNV_SOMATIC_WDL', 'RUN_M2_WDL', 'RUN_VCF_SITE_LEVEL_FILTERING_WDL' ]
continue-on-error: true
name: WDL test ${{ matrix.wdlTest }} on cromwell
steps:
Expand Down
3 changes: 1 addition & 2 deletions .../java/org/broadinstitute/hellbender/tools/copynumber/plotting/PlotDenoisedCopyRatios.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,8 +204,7 @@ private void writeDenoisingPlots(final String sampleName,
//this runs the R statement "source("CNVPlottingLibrary.R")" before the main script runs
executor.addScript(new Resource(PlottingUtils.CNV_PLOTTING_R_LIBRARY, PlotDenoisedCopyRatios.class));
executor.addScript(new Resource(PLOT_DENOISED_COPY_RATIOS_R_SCRIPT, PlotDenoisedCopyRatios.class));
//--args is needed for Rscript to recognize other arguments properly
executor.addArgs("--args",
executor.addArgs(
"--sample_name=" + sampleName,
"--standardized_copy_ratios_file=" + CopyNumberArgumentValidationUtils.getCanonicalPath(inputStandardizedCopyRatiosFile),
"--denoised_copy_ratios_file=" + CopyNumberArgumentValidationUtils.getCanonicalPath(inputDenoisedCopyRatiosFile),
Expand Down
3 changes: 1 addition & 2 deletions ...ain/java/org/broadinstitute/hellbender/tools/copynumber/plotting/PlotModeledSegments.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,8 +285,7 @@ private void writeModeledSegmentsPlot(final String sampleName,
//this runs the R statement "source("CNVPlottingLibrary.R")" before the main script runs
executor.addScript(new Resource(PlottingUtils.CNV_PLOTTING_R_LIBRARY, PlotModeledSegments.class));
executor.addScript(new Resource(PLOT_MODELED_SEGMENTS_R_SCRIPT, PlotModeledSegments.class));
//--args is needed for Rscript to recognize other arguments properly
executor.addArgs("--args",
executor.addArgs(
"--sample_name=" + sampleName,
"--denoised_copy_ratios_file=" + (inputDenoisedCopyRatiosFile == null ? null : CopyNumberArgumentValidationUtils.getCanonicalPath(inputDenoisedCopyRatiosFile)),
"--allelic_counts_file=" + (inputAllelicCountsFile == null ? null : CopyNumberArgumentValidationUtils.getCanonicalPath(inputAllelicCountsFile)),
Expand Down
27 changes: 12 additions & 15 deletions ...ava/org/broadinstitute/hellbender/tools/walkers/vqsr/CNNScoreVariantsIntegrationTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ public void testRequirePythonEnvironment() throws IOException {
/**
* Run the tool on a small test VCF.
*/
@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testAllDefaultArgs() {
final ArgumentsBuilder argsBuilder = new ArgumentsBuilder();
final File tempVcf = createTempFile("tester", ".vcf");
Expand All @@ -66,7 +66,7 @@ public void testAllDefaultArgs() {
assertInfoFieldsAreClose(tempVcf, expectedVcf, GATKVCFConstants.CNN_1D_KEY);
}

@Test(groups = {"python"}, expectedExceptions = PythonScriptExecutorException.class)
@Test(groups = {"python"}, expectedExceptions = PythonScriptExecutorException.class, enabled = false)
public void testExceptionDuringAsyncBatch() {
final ArgumentsBuilder argsBuilder = new ArgumentsBuilder();
final File tempVcf = createTempFile("tester", ".vcf");
Expand All @@ -82,7 +82,7 @@ public void testExceptionDuringAsyncBatch() {
runCommandLine(argsBuilder);
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceArchitecture() {
final boolean newExpectations = false;
final String expectedVCFName = largeFileTestDir + "VQSR/expected/cnn_1d_chr20_subset_expected.vcf";
Expand All @@ -104,7 +104,7 @@ public void testInferenceArchitecture() {
}
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceWeights() {
final File tempVcf = createTempFile("tester", ".vcf");
final File expectedVcf = new File(largeFileTestDir + "VQSR/expected/cnn_1d_chr20_subset_expected.vcf");
Expand All @@ -119,7 +119,7 @@ public void testInferenceWeights() {
assertInfoFieldsAreClose(tempVcf, expectedVcf, GATKVCFConstants.CNN_1D_KEY);
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceArchitectureAndWeights() {
final File tempVcf = createTempFile("tester", ".vcf");
final File expectedVcf = new File(largeFileTestDir + "VQSR/expected/cnn_1d_chr20_subset_expected.vcf");
Expand All @@ -135,7 +135,7 @@ public void testInferenceArchitectureAndWeights() {
assertInfoFieldsAreClose(tempVcf, expectedVcf, GATKVCFConstants.CNN_1D_KEY);
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceWithIntervals() {
final boolean newExpectations = false;
final String expectedVCFName = largeFileTestDir + "VQSR/expected/cnn_1d_contig20_1m_10m_expected.vcf";
Expand All @@ -157,7 +157,7 @@ public void testInferenceWithIntervals() {
}
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testSmallBatchInference() {
final File tempVcf = createTempFile("tester", ".vcf");
final File expectedVcf = new File(largeFileTestDir + "VQSR/expected/cnn_1d_chr20_subset_expected.vcf");
Expand All @@ -172,7 +172,7 @@ public void testSmallBatchInference() {
assertInfoFieldsAreClose(tempVcf, expectedVcf, GATKVCFConstants.CNN_1D_KEY);
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testOnContigEdge() {
final String edgeVcf = toolsTestDir + "walkers/VQSR/variantNearContigEdge.vcf";
final File tempVcf = createTempFile("tester", ".vcf");
Expand All @@ -190,7 +190,7 @@ public void testOnContigEdge() {
/**
* Run the 2D Model on a small test VCF with the resource loaded weights and architecture.
*/
@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInference2dResourceModel() {
// We reset the random number generator at the beginning of each test so that the random down-sampling of reads
// by the reservoir down-sampler does not cause slightly different scores.
Expand All @@ -215,7 +215,7 @@ public void testInference2dResourceModel() {
/**
* Run the 2D Model on a small test VCF.
*/
@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceArchitecture2d() {
Utils.resetRandomGenerator();
final boolean newExpectations = false;
Expand Down Expand Up @@ -243,7 +243,7 @@ public void testInferenceArchitecture2d() {
}
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceWeights2d() {
Utils.resetRandomGenerator();
TensorType tt = TensorType.read_tensor;
Expand All @@ -264,7 +264,7 @@ public void testInferenceWeights2d() {
assertInfoFieldsAreClose(tempVcf, expectedVcf, GATKVCFConstants.CNN_2D_KEY);
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testInferenceArchitectureAndWeights2d() {
Utils.resetRandomGenerator();
TensorType tt = TensorType.read_tensor;
Expand Down Expand Up @@ -300,7 +300,4 @@ private void assertInfoFieldsAreClose(File actualVcf, File expectedVcf, String i
}
Assert.assertTrue(!expectedVi.hasNext() && !actualVi.hasNext());
}



}
12 changes: 6 additions & 6 deletions src/test/java/org/broadinstitute/hellbender/tools/walkers/vqsr/CNNVariantPipelineTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ public class CNNVariantPipelineTest extends GATKBaseTest {
private static File readTensorDir;
private static File referenceTensorDir;

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public static void makeTempDirectories() {
readTensorDir = createTempDir("readTensorDir");
referenceTensorDir = createTempDir("referenceTensorDir");
}

@Test(groups = {"python"}, dependsOnMethods = {"makeTempDirectories"})
@Test(groups = {"python"}, dependsOnMethods = {"makeTempDirectories"}, enabled = false)
public void testGenerateReferenceTensors() {
final ArgumentsBuilder args = new ArgumentsBuilder();
args.addRaw("CNNVariantWriteTensors")
Expand All @@ -37,7 +37,7 @@ public void testGenerateReferenceTensors() {
new Main().instanceMain(args.getArgsArray());
}

@Test(groups = {"python"}, dependsOnMethods = {"makeTempDirectories"})
@Test(groups = {"python"}, dependsOnMethods = {"makeTempDirectories"}, enabled = false)
public void testGenerateReadTensors() {
final String bamFile = largeFileTestDir + "VQSR/g94982_b37_chr20_1m_8m_bamout.bam";
final ArgumentsBuilder args = new ArgumentsBuilder();
Expand All @@ -55,7 +55,7 @@ public void testGenerateReadTensors() {
new Main().instanceMain(args.getArgsArray());
}

@Test(groups = {"python"}, dependsOnMethods = {"testGenerateReferenceTensors"})
@Test(groups = {"python"}, dependsOnMethods = {"testGenerateReferenceTensors"}, enabled = false)
public void testTrainingReferenceModel() {
final ArgumentsBuilder args = new ArgumentsBuilder();
args.addRaw("CNNVariantTrain")
Expand All @@ -69,7 +69,7 @@ public void testTrainingReferenceModel() {
new Main().instanceMain(args.getArgsArray());
}

@Test(groups = {"python"}, dependsOnMethods = {"testGenerateReadTensors"})
@Test(groups = {"python"}, dependsOnMethods = {"testGenerateReadTensors"}, enabled = false)
public void testTrainingReadModel() {
final ArgumentsBuilder args = new ArgumentsBuilder();
args.addRaw("CNNVariantTrain")
Expand All @@ -85,7 +85,7 @@ public void testTrainingReadModel() {
new Main().instanceMain(args.getArgsArray());
}

@Test(groups = {"python"})
@Test(groups = {"python"}, enabled = false)
public void testTranches() {
final String trancheVCF = largeFileTestDir + "VQSR/g94982_b37_chr20_1m_10m.vcf.gz";
final String snpTruthVCF = largeFileTestDir + "VQSR/giab_chr20_1m_10m.vcf.gz";
Expand Down
6 changes: 3 additions & 3 deletions ...rg/broadinstitute/hellbender/tools/walkers/vqsr/FilterVariantTranchesIntegrationTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ public Object[][] getGoodResourceCombos() {
/**
* Run the tool on a small test VCF.
*/
@Test(dataProvider = "getFilteringArgs")
@Test(dataProvider = "getFilteringArgs", enabled = false)
public void runTrancheFiltering(final List<String> resources, final List<String> extraArgs,
final String expectedOutput) throws IOException {
final ArgumentsBuilder argsBuilder = new ArgumentsBuilder();
Expand All @@ -113,7 +113,7 @@ public void runTrancheFiltering(final List<String> resources, final List<String>
spec.executeTest("testTrancheFiltering", this);
}

@Test(dataProvider = "getBadResourceCombos", expectedExceptions = UserException.BadInput.class)
@Test(dataProvider = "getBadResourceCombos", expectedExceptions = UserException.BadInput.class, enabled = false)
public void runTrancheFilteringOnBadInputs(final String inputVcf, final String resourceVcf) {
final ArgumentsBuilder argsBuilder = new ArgumentsBuilder();
argsBuilder.add(StandardArgumentDefinitions.VARIANT_LONG_NAME, inputVcf)
Expand All @@ -125,7 +125,7 @@ public void runTrancheFilteringOnBadInputs(final String inputVcf, final String r
runCommandLine(argsBuilder);
}

@Test(dataProvider = "getGoodResourceCombos")
@Test(dataProvider = "getGoodResourceCombos", enabled = false)
public void runTrancheFilteringOnGoodInputs(final String inputVcf, final String resourceVcf) {
final ArgumentsBuilder argsBuilder = new ArgumentsBuilder();
argsBuilder.add(StandardArgumentDefinitions.VARIANT_LONG_NAME, inputVcf)
Expand Down
2 changes: 1 addition & 1 deletion ...e/score/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snp.annot.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion .../score/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snp.scores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...able/score/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snp.vcf.idx
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...re/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snpIndel.annot.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...e/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snpIndel.scores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...score/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snpIndel.vcf.idx
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...core/expected/extract.nonAS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snp.annot.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...ore/expected/extract.nonAS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snp.scores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...e/score/expected/extract.nonAS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snp.vcf.idx
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...expected/extract.nonAS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snpIndel.annot.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...xpected/extract.nonAS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snpIndel.scores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...re/expected/extract.nonAS.snpIndel.posUn.train.snpIndel.posOnly.IF.score.snpIndel.vcf.idx
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion .../train/expected/extract.AS.snpIndel.posUn.train.snp.posOnly.IF.snp.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions ...qsr/scalable/train/expected/extract.AS.snpIndel.posUn.train.snp.posOnly.IF.snp.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...ble/train/expected/extract.AS.snpIndel.posUn.train.snp.posOnly.IF.snp.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...ed/extract.AS.snpIndel.posUn.train.snp.posOnly.IFDifferentSeed.snp.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions ...train/expected/extract.AS.snpIndel.posUn.train.snp.posOnly.IFDifferentSeed.snp.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...ected/extract.AS.snpIndel.posUn.train.snp.posOnly.IFDifferentSeed.snp.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.indel.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions ...lable/train/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.indel.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...in/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.indel.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...n/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.snp.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions ...calable/train/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.snp.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...rain/expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IF.snp.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...act.AS.snpIndel.posUn.train.snpIndel.posOnly.IFDifferentSeed.indel.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions ...xpected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IFDifferentSeed.indel.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...xtract.AS.snpIndel.posUn.train.snpIndel.posOnly.IFDifferentSeed.indel.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...tract.AS.snpIndel.posUn.train.snpIndel.posOnly.IFDifferentSeed.snp.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions .../expected/extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IFDifferentSeed.snp.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion .../extract.AS.snpIndel.posUn.train.snpIndel.posOnly.IFDifferentSeed.snp.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion ...ain/expected/extract.nonAS.snpIndel.posUn.train.snp.posOnly.IF.snp.calibrationScores.hdf5
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions .../scalable/train/expected/extract.nonAS.snpIndel.posUn.train.snp.posOnly.IF.snp.scorer.pkl
View file
Open in desktop
Git LFS file not shown
2 changes: 1 addition & 1 deletion .../train/expected/extract.nonAS.snpIndel.posUn.train.snp.posOnly.IF.snp.trainingScores.hdf5
View file
Open in desktop
Git LFS file not shown
Loading