Skip to content

Commit

Permalink
3.1.0
Browse files Browse the repository at this point in the history
  • Loading branch information
tdayris committed Feb 16, 2024
1 parent 0c880bd commit 3a458ee
Show file tree
Hide file tree
Showing 23 changed files with 310 additions and 77,670 deletions.
31 changes: 31 additions & 0 deletions .test/config/config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,17 @@ params:
# Keep all cdna sequences
cdna: ""
# Optional parameters for agat (filter/correct GTF format)
agat:
# Optional parameters for agat_convert_sp_gff2gtf.pl
gff2gtf: ""
# Optional paraasamples: config/samples.csv
# Optional parameters for pyfaidx (filter/correct fasta format)
pyfaidx:
# Filter-out non canonical chromosomes
dna: '--regex "^[[0-9]+|X|Y|MT]"'
# Keep all cdna sequences
cdna: ""
# Optional parameters for agat (filter/correct GTF format)
agat:
# Optional parameters for agat_convert_sp_gff2gtf.pl
gff2gtf: ""
Expand All @@ -24,6 +35,8 @@ params:
bedtools:
# Optional parameters for filtering non-canonical chromosomes over dbSNP
filter_non_canonical_chrom: ""
# Optional parameters for bedtools merge, when merging blacklisted regions
merge: ""
# Optional parameters for tabix index VCF
tabix: "-p vcf"
# Optional parameters for fastp
Expand All @@ -32,6 +45,14 @@ params:
adapters: ""
# Optional command line arguments for fastp
extra: ""
# Optional parameters for FastQ Screen
fastq_screen:
# Number of reads processed
subset: 100000
# Aligner used to index dna sequences
aligner: bowtie2
# Path to configuration file
fastq_screen_config: ""
# Optional parameters for fastqc
fastqc: ""
# Optional parameters for bowtie2
Expand All @@ -50,6 +71,8 @@ params:
metrics: ""
# Optional parameters for picard create sequence dictionary
createsequencedictionary: ""
# Optional parameters for collect multiple metrics
collectmultiplemetrics: ""
# Optional parameters for samtools stats
samtools:
# Optional parameters for samtools fasta index
Expand All @@ -58,6 +81,14 @@ params:
stats: ""
# Optional parameters for multiqc
multiqc: "--module picard --module fastqc --module fastp --module samtools --module bowtie2 --module sambamba --zip-data-dir --verbose --no-megaqc-upload --no-ansi --force"
# Optional parameters for wget
wget: "--verbose"
# Optional parameters for rsync
rsync: "--verbose --checksum --force --human-readable --progress"
# Optional parameters for pyroe
pyroe:
# Optional parameters for ID 2 name
idtoname: ""

# Optional path to a `genome.csv` file
genomes: config/genomes.csv
Expand Down
4 changes: 2 additions & 2 deletions .test/config/genomes.csv
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
species,build,release
saccharomyces_cerevisiae,R64-1-1,111
species,build,release,dna_fasta
saccharomyces_cerevisiae,R64-1-1,111,test
4 changes: 2 additions & 2 deletions .test/makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ report.txt: pipeline.txt
--report report.html 2>&1 | tee report.txt


pipeline.txt: linter.txt black.txt
pipeline.txt: linter.txt
mamba activate test-snakemake-wrappers && \
snakemake -s ../workflow/Snakefile \
--cores 2 \
Expand All @@ -53,7 +53,7 @@ format.txt:
snakefmt $(snakefiles) 2>&1 | tee format.txt


linter.txt: format.txt
linter.txt: format.txt black.txt
mamba activate test-snakemake-wrappers && \
snakemake -s ../workflow/Snakefile \
--lint 2>&1 | tee linter.txt
Expand Down
77,272 changes: 0 additions & 77,272 deletions .test/report.html
View file

This file was deleted.

15 changes: 15 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
# 3.1.0

Update to Snakemake v8+

## Features:

* Usage updated
* DAG as ascii art
* tempfiles, logs and benchmarks paths reorganized:
* `tmp/fair_fastqc_multiqc/{rule_name}/{wildcards}.{extension}`
* `log/fair_fastqc_multiqc/{rule_name}/{wildcards}.log`
* `benchmark/fair_fastqc_multiqc/{rule_name}/{wildcards}.tsv`
* use of `lookup` and `collect` rather than hand made functions
* All keys in configuration must be present in configuration file

# 3.0.1

## Features:
Expand Down
44 changes: 43 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,49 @@ A complete description of the results can be found here in [workflow reports](ht

The tools used in this pipeline are described [here](https://github.com/tdayris/fair_bowtie2_mapping/blob/main/workflow/report/material_methods.rst) textually. Web-links are available below:

![workflow_rulegraph](dag.png)
```
┌─────────────────────────────────┐ ┌───────────────────────────────┐
│ fair_genome_indexer_pipeline │ │ fair_fastqc_multiqc_pipeline │
└───────────┬─────────────────────┘ └───────────────────────────┬───┘
│ │
│ │
│ │
│ │
│ │
┌─────────▼───────────┐ ┌────────────────────────┐ │
│ Bowtie2 │ │ Fastp │ │
│ Index DNA sequence │ │ Trimm and check reads │ │
└──────────────────┬──┘ │ qualittiy ├─────────────────┐ │
│ └────────────────────────┘ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
┌───────▼──────────────┐ │ │
│ Bowtie2 │ ┌─▼────▼──────────┐
│ Align trimmed reads ├────────────────────────────────────────► │
└───────┬──────────────┘ │ MultiQC │
│ │ Quality report │
│ │ │
┌───────▼──────────────────────────┐ └──▲──▲───────────┘
│ Sambamba │ │ │
│ Quality filters + sorting + │ │ │
│ duplicated reads identification │ │ │
└────────────────┬───────────┬─────┘ │ │
│ │ │ │
│ ┌─────▼─────────────┐ │ │
│ │ Samtools ├───────────────────────┘ │
│ │ Quality controls │ │
│ └───────────────────┘ │
│ │
│ │
│ ┌──────────────────┐ │
│ │ Picard ├────────────────────────┘
└────────► Quality controls │
└──────────────────┘
```

### Index and genome sequences with [`fair_genome_indexer`](https://github.com/tdayris/fair_genome_indexer/)

Expand Down
22 changes: 21 additions & 1 deletion config/config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,24 @@ params:
bedtools:
# Optional parameters for filtering non-canonical chromosomes over dbSNP
filter_non_canonical_chrom: ""
# Optional parameters for bedtools merge, when merging blacklisted regions
merge: ""
# Optional parameters for tabix index VCF
tabix: "-p vcf"
# Optional parameters for FastQ Screen
fastq_screen:
# Number of reads processed
subset: 100000
# Aligner used to index dna sequences
aligner: bowtie2
# Path to configuration file
fastq_screen_config: "/mnt/beegfs/database/bioinfo/Index_DB/Fastq_Screen/0.14.0/fastq_screen.conf"
# Optional parameters for fastp
fastp:
# Optional adapters to remove
adapters: ""
# Optional command line arguments for fastp
extra: ""
extra: "--verbose --overrepresentation_analysis"
# Optional parameters for fastqc
fastqc: ""
# Optional parameters for bowtie2
Expand All @@ -50,6 +60,8 @@ params:
metrics: ""
# Optional parameters for picard create sequence dictionary
createsequencedictionary: ""
# Optional parameters for collect multiple metrics
collectmultiplemetrics: ""
# Optional parameters for samtools stats
samtools:
# Optional parameters for samtools fasta index
Expand All @@ -58,6 +70,14 @@ params:
stats: ""
# Optional parameters for multiqc
multiqc: "--module picard --module fastqc --module fastp --module samtools --module bowtie2 --module sambamba --zip-data-dir --verbose --no-megaqc-upload --no-ansi --force"
# Optional parameters for wget
wget: "--verbose"
# Optional parameters for rsync
rsync: "--verbose --checksum --force --human-readable --progress"
# Optional parameters for pyroe
pyroe:
# Optional parameters for ID 2 name
idtoname: ""

# Optional path to a `genome.csv` file
genomes: config/genomes.csv
Expand Down
Binary file removed dag.png
View file
Binary file not shown.
3 changes: 2 additions & 1 deletion workflow/envs/python.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ channels:
- conda-forge
- nodefaults
dependencies:
- pandas=2.2.0
- pandas=2.2.0
- pyarrow=15.0.0
3 changes: 2 additions & 1 deletion workflow/envs/snakemake.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ dependencies:
- snakefmt
- pandas
- mamba
- pytest
- pytest
- apptainer
2 changes: 1 addition & 1 deletion workflow/report/fastp.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ This HTML_ file contains Fastp_ quality reports for sample {{ snakemake.wildcard
It is a stand-alone file, and can be opened in your favorite web-browser.

.. _HTML: https://en.wikipedia.org/wiki/HTML
.. _Fastp: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/fastp.html
.. _Fastp: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/fastp.html
14 changes: 7 additions & 7 deletions workflow/report/material_methods.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,17 @@ installation usage, and resutls can be found on the
.. [#multiqcpaper] Ewels, Philip, et al. "MultiQC: summarize analysis results for multiple tools and samples in a single report." Bioinformatics 32.19 (2016): 3047-3048.
.. [#snakemakepaper] Köster, Johannes, and Sven Rahmann. "Snakemake—a scalable bioinformatics workflow engine." Bioinformatics 28.19 (2012): 2520-2522.
.. _Sambamba: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/sambamba.html
.. _Bowtie2: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/bowtie2.html
.. _Fastp: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/fastp.html
.. _Picard: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/picard/collectmultiplemetrics.html
.. _MultiQC: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/multiqc.html
.. _Sambamba: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/sambamba.html
.. _Bowtie2: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/bowtie2.html
.. _Fastp: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/fastp.html
.. _Picard: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/picard/collectmultiplemetrics.html
.. _MultiQC: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/multiqc.html
.. _Snakemake: https://snakemake.readthedocs.io
.. _Github: https://github.com/tdayris/fair_bowtie2_mapping
.. _`Snakemake workflow`: https://snakemake.github.io/snakemake-workflow-catalog?usage=tdayris/fair_bowtie2_mapping
.. _Agat: https://agat.readthedocs.io/en/latest/index.html
.. _Samtools: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/samtools/faidx.html
.. _FastQC: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/fastqc.html
.. _Samtools: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/samtools/faidx.html
.. _FastQC: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/fastqc.html
.. _Pyfaidx: https://github.com/mdshw5/pyfaidx


Expand Down
12 changes: 6 additions & 6 deletions workflow/report/multiqc.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ Samtools_, FastQC_, and Bowtie2_. It is a stand-alone file,
and can be opened in your favorite web-browser.

.. _HTML: https://en.wikipedia.org/wiki/HTML
.. _Fastp: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/fastp.html
.. _Bowtie2: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/bowtie2/align.html
.. _FastQC: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/fastqc.html
.. _Stats: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/samtools/stats.html
.. _Picard: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/picard/collectmultiplemetrics.html
.. _Samtools: https://snakemake-wrappers.readthedocs.io/en/v3.3.3/wrappers/samtools/stats.html
.. _Fastp: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/fastp.html
.. _Bowtie2: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/bowtie2/align.html
.. _FastQC: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/fastqc.html
.. _Stats: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/samtools/stats.html
.. _Picard: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/picard/collectmultiplemetrics.html
.. _Samtools: https://snakemake-wrappers.readthedocs.io/en/v3.3.6/wrappers/samtools/stats.html
4 changes: 2 additions & 2 deletions workflow/report/usage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ and ignore the rest of this documentation.
::

# Activate conda environment
conda activate /mnt/beegfs/pipelines/unofficial-snakemake-wrappers/shared_install/bigr_epicure_pipeline/
conda activate /mnt/beegfs/pipelines/unofficial-snakemake-wrappers/shared_install/snakemake_v8.4.8

# Deploy workflow with the version of your choice
snakedeploy deploy-workflow \
https://github.com/tdayris/fair_bowtie2_mapping . \
--tag <version>

# Run snakemake command
snakemake --profile '/mnt/beegfs/pipelines/unofficial-snakemake-wrappers/profiles/slurm-web/'
snakemake --profile '/mnt/beegfs/pipelines/unofficial-snakemake-wrappers/profiles/slurm-web-8/'

With `version` being the latest available version of this pipeline. Select your `version here`_

Expand Down
Loading

0 comments on commit 3a458ee

Please sign in to comment.