Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making memory an optional parameter for ValidateVcf task #1144

Merged
merged 7 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions pipelines/broad/arrays/single_sample/Arrays.changelog.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 2.6.21
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 2.6.20
2023-11-29 (Date of Last Commit)

Expand Down
2 changes: 1 addition & 1 deletion pipelines/broad/arrays/single_sample/Arrays.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import "../../../../tasks/broad/Utilities.wdl" as utils

workflow Arrays {

String pipeline_version = "2.6.20"
String pipeline_version = "2.6.21"

input {
String chip_well_barcode
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 2.1.9
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter

# 2.1.8
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import "../../../../../../tasks/broad/Qc.wdl" as QC

workflow ReblockGVCF {

String pipeline_version = "2.1.8"
String pipeline_version = "2.1.9"


input {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.1.16
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 3.1.15
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ import "../../../../../../structs/dna_seq/DNASeqStructs.wdl"
# WORKFLOW DEFINITION
workflow ExomeGermlineSingleSample {

String pipeline_version = "3.1.15"
String pipeline_version = "3.1.16"


input {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.0.13
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.0.12
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ workflow UltimaGenomicsWholeGenomeGermline {
filtering_model_no_gt_name: "String describing the optional filtering model; default set to rf_model_ignore_gt_incl_hpol_runs"
}

String pipeline_version = "1.0.12"
String pipeline_version = "1.0.13"


References references = alignment_references.references
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.1.17
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 3.1.16
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ import "../../../../../../structs/dna_seq/DNASeqStructs.wdl"
workflow WholeGenomeGermlineSingleSample {


String pipeline_version = "3.1.16"
String pipeline_version = "3.1.17"


input {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 2.1.15
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 2.1.14
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import "../../../../../tasks/broad/DragenTasks.wdl" as DragenTasks
workflow VariantCalling {


String pipeline_version = "2.1.14"
String pipeline_version = "2.1.15"


input {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.0.13
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.0.12
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ workflow UltimaGenomicsWholeGenomeCramOnly {
save_bam_file: "If true, then save intermeidate ouputs used by germline pipeline (such as the output BAM) otherwise they won't be kept as outputs."
}

String pipeline_version = "1.0.12"
String pipeline_version = "1.0.13"

References references = alignment_references.references

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.12.15
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.12.14
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import "../../../../tasks/broad/Qc.wdl" as Qc

workflow IlluminaGenotypingArray {

String pipeline_version = "1.12.14"
String pipeline_version = "1.12.15"

input {
String sample_alias
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.1.5
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.1.4
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ workflow BroadInternalArrays {
description: "Push outputs of Arrays.wdl to TDR dataset table ArraysOutputsTable."
}

String pipeline_version = "1.1.4"
String pipeline_version = "1.1.5"

input {
# inputs to wrapper task
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.0.14
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.0.13
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import "../../../../../../../pipelines/broad/qc/CheckFingerprint.wdl" as FP

workflow BroadInternalUltimaGenomics {

String pipeline_version = "1.0.13"
String pipeline_version = "1.0.14"

input {

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.0.27
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.0.26
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import "../../../../tasks/broad/Utilities.wdl" as utils

workflow BroadInternalRNAWithUMIs {

String pipeline_version = "1.0.26"
String pipeline_version = "1.0.27"

input {
# input needs to be either "hg19" or "hg38"
Expand Down
5 changes: 5 additions & 0 deletions pipelines/broad/qc/CheckFingerprint.changelog.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 1.0.14
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 1.0.13
2023-11-29 (Date of Last Commit)

Expand Down
2 changes: 1 addition & 1 deletion pipelines/broad/qc/CheckFingerprint.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import "../../../tasks/broad/Qc.wdl" as Qc

workflow CheckFingerprint {

String pipeline_version = "1.0.13"
String pipeline_version = "1.0.14"

input {
File? input_vcf
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.1.16
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 3.1.15
2023-11-29 (Date of Last Commit)

Expand Down
2 changes: 1 addition & 1 deletion pipelines/broad/reprocessing/exome/ExomeReprocessing.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import "../../../../structs/dna_seq/DNASeqStructs.wdl"
workflow ExomeReprocessing {


String pipeline_version = "3.1.15"
String pipeline_version = "3.1.16"

input {
File? input_cram
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.1.18
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 3.1.17
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import "../../../../../tasks/broad/CopyFilesFromCloudToCloud.wdl" as Copy

workflow ExternalExomeReprocessing {

String pipeline_version = "3.1.17"
String pipeline_version = "3.1.18"


input {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 2.1.18
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 2.1.17
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import "../../../../../tasks/broad/CopyFilesFromCloudToCloud.wdl" as Copy
workflow ExternalWholeGenomeReprocessing {


String pipeline_version = "2.1.17"
String pipeline_version = "2.1.18"

input {
File? input_cram
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.1.17
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 3.1.16
2023-11-29 (Date of Last Commit)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import "../../../../structs/dna_seq/DNASeqStructs.wdl"

workflow WholeGenomeReprocessing {

String pipeline_version = "3.1.16"
String pipeline_version = "3.1.17"

input {
File? input_cram
Expand Down
5 changes: 5 additions & 0 deletions pipelines/skylab/multiome/Multiome.changelog.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# 3.0.1
2023-12-08 (Date of Last Commit)

* ValidateVcfs now has optional memory parameter; this does not affect this pipeline

# 3.0.0
2023-11-22 (Date of Last Commit)

Expand Down
2 changes: 1 addition & 1 deletion pipelines/skylab/multiome/Multiome.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import "../../../tasks/skylab/H5adUtils.wdl" as H5adUtils
import "https://raw.githubusercontent.com/broadinstitute/CellBender/v0.3.0/wdl/cellbender_remove_background.wdl" as CellBender

workflow Multiome {
String pipeline_version = "3.0.0"
String pipeline_version = "3.0.1"

input {
String input_id
Expand Down
6 changes: 4 additions & 2 deletions tasks/broad/Qc.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -622,12 +622,14 @@ task ValidateVCF {
Boolean is_gvcf = true
String? extra_args
String gatk_docker = "us.gcr.io/broad-gatk/gatk:4.3.0.0"
Int machine_mem_mb = 7000
}

Boolean calling_intervals_is_vcf = defined(calling_interval_list_index)
String calling_interval_list_basename = basename(calling_interval_list)
String calling_interval_list_index_basename = if calling_intervals_is_vcf then basename(select_first([calling_interval_list_index])) else ""

Int command_mem_mb = machine_mem_mb - 1000
Float ref_size = size(ref_fasta, "GiB") + size(ref_fasta_index, "GiB") + size(ref_dict, "GiB")
Int disk_size = ceil(size(input_vcf, "GiB") + size(dbsnp_vcf, "GiB") + ref_size) + 20

Expand All @@ -641,7 +643,7 @@ task ValidateVCF {
fi

# Note that WGS needs a lot of memory to do the -L *.vcf if an interval file is not supplied
gatk --java-options "-Xms6000m -Xmx6500m" \
gatk --java-options "-Xms~{command_mem_mb}m -Xmx~{command_mem_mb}m" \
ValidateVariants \
-V ~{input_vcf} \
-R ~{ref_fasta} \
Expand All @@ -654,7 +656,7 @@ task ValidateVCF {
runtime {
docker: gatk_docker
preemptible: preemptible_tries
memory: "7000 MiB"
memory: machine_mem_mb + " MiB"
bootDiskSizeGb: 15
disks: "local-disk " + disk_size + " HDD"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ slug: /Pipelines/Exome_Germline_Single_Sample_Pipeline/README

| Pipeline Version | Date Updated | Documentation Author | Questions or Feedback |
| :----: | :---: | :----: | :--------------: |
| [ExomeGermlineSingleSample_v3.1.15](https://github.com/broadinstitute/warp/releases?q=ExomeGermlineSingleSample_v3.0.0&expanded=true) | December, 2023 | [Elizabeth Kiernan](mailto:[email protected]) | Please file GitHub issues in WARP or contact [the WARP team](mailto:[email protected]) |
| [ExomeGermlineSingleSample_v3.1.16](https://github.com/broadinstitute/warp/releases?q=ExomeGermlineSingleSample_v3.0.0&expanded=true) | December, 2023 | [Elizabeth Kiernan](mailto:[email protected]) | Please file GitHub issues in WARP or contact [the WARP team](mailto:[email protected]) |


The Exome Germline Single Sample pipeline implements data pre-processing and initial variant calling according to the GATK Best Practices for germline SNP and Indel discovery in human exome sequencing data.
Expand Down
2 changes: 1 addition & 1 deletion website/docs/Pipelines/Multiome_Pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ slug: /Pipelines/Multiome_Pipeline/README

| Pipeline Version | Date Updated | Documentation Author | Questions or Feedback |
| :----: | :---: | :----: | :--------------: |
| [Multiome v3.0.0](https://github.com/broadinstitute/warp/releases) | December, 2023 | Kaylee Mathews | Please file GitHub issues in warp or contact the [WARP Pipeline Development team](mailto:[email protected]) |
| [Multiome v3.0.1](https://github.com/broadinstitute/warp/releases) | December, 2023 | Kaylee Mathews | Please file GitHub issues in warp or contact the [WARP Pipeline Development team](mailto:[email protected]) |

![Multiome_diagram](./multiome_diagram.png)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ slug: /Pipelines/Ultima_Genomics_Whole_Genome_Germline_Pipeline/README

| Pipeline Version | Date Updated | Documentation Authors | Questions or Feedback |
| :----: | :---: | :----: | :--------------: |
| [UltimaGenomicsWholeGenomeGermline_v1.0.0](https://github.com/broadinstitute/warp/releases) | June, 2022 | [Elizabeth Kiernan](mailto:[email protected]) & [Kaylee Mathews](mailto:[email protected])| Please file GitHub issues in warp or contact [the wARP team](mailto:[email protected]) |
| [UltimaGenomicsWholeGenomeGermline_v1.0.13](https://github.com/broadinstitute/warp/releases) | December, 2023 | [Elizabeth Kiernan](mailto:[email protected]) & [Kaylee Mathews](mailto:[email protected])| Please file GitHub issues in warp or contact [the wARP team](mailto:[email protected]) |

![UG_diagram](ug_diagram.png)

Expand All @@ -19,7 +19,7 @@ The [Ultima Genomics Whole Genome Germline (UG_WGS) workflow](https://github.com

Ultima Genomics sequencing is a novel technology that produces single-read, flow-based data ([Almogy et al., 2022](https://www.biorxiv.org/content/10.1101/2022.05.29.493900v1)). The sequencing platform works by flowing one nucleotide at a time in order, iteratively. This is in contrast to traditional technologies that do all four nucleotides at once. This iterative approach ensures that only one dNTP is responsible for the signal and it does not require the blocking of dNTPs.

### What does the workflow do?
### What does the workflow do?

The workflow requires either an aligned CRAM output of the sequencing platform or an unmapped BAM as input. Overall, it aligns reads to a reference genome, marks duplicate reads, calls variants, post-processes variants in the output VCF in preparation for joint calling, and calculates quality control metrics. The workflow outputs a (re)aligned CRAM, an annotated GVCF with index, and quality metrics.

Expand Down Expand Up @@ -201,7 +201,7 @@ The workflow implements initial variant calling with a version of HaplotypeCalle
| [Utilities.ScatterIntervalList](https://github.com/broadinstitute/warp/blob/develop/tasks/broad/Utilities.wdl) | IntervalListTools | Picard | Splits the calling interval list into sub-intervals in order to perform variant calling on the sub-intervals. |
| [Tasks.HaplotypeCaller](https://github.com/broadinstitute/warp/blob/develop/tasks/broad/UltimaGenomicsWholeGenomeGermlineTasks.wdl) | HaplotypeCaller | GATK | Performs initial variant calling on the aligned BAM file and outputs sub-interval GVCFs and a bamout file. |

### 6. Merge VCFs and BAMs and convert GVCF to VCF
### 6. Merge VCFs and BAMs and convert GVCF to VCF
The workflow performs multiple post-processing steps to prepare the VCF for downstream joint calling. The HaplotypeCaller GVCF outputs are merged into a single GVCF and then converted to VCF in preparation for this post-processing.

| Task name and WDL link | Tool | Software | Description |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ slug: /Pipelines/Whole_Genome_Germline_Single_Sample_Pipeline/README

| Pipeline Version | Date Updated | Documentation Author | Questions or Feedback |
| :----: | :---: | :----: | :--------------: |
| WholeGenomeGermlineSingleSample_v3.1.14 (see [releases page](https://github.com/broadinstitute/warp/releases)) | August, 2022 | [Elizabeth Kiernan](mailto:[email protected]) | Please file GitHub issues in WARP or contact [the WARP team](mailto:[email protected]) |
| WholeGenomeGermlineSingleSample_v3.1.17 (see [releases page](https://github.com/broadinstitute/warp/releases)) | December, 2023 | [Elizabeth Kiernan](mailto:[email protected]) | Please file GitHub issues in WARP or contact [the WARP team](mailto:[email protected]) |

## Introduction to the Whole Genome Germline Single Sample Pipeline
The Whole Genome Germline Single Sample (WGS) pipeline implements data pre-processing and initial variant calling according to the GATK Best Practices for germline SNP and Indel discovery in human whole-genome sequencing data. It includes the DRAGEN-GATK mode, which makes the pipeline functionally equivalent to DRAGEN’s analysis pipeline (read more in this [DRAGEN-GATK blog](https://gatk.broadinstitute.org/hc/en-us/articles/360039984151)).
Expand Down
Loading
Loading