Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ParkinsonLab/quackers
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.0.2
Choose a base ref
...
head repository: ParkinsonLab/quackers
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: develop
Choose a head ref
  • 8 commits
  • 5 files changed
  • 1 contributor

Commits on Sep 18, 2024

  1. WIP: installing gapseq

    billytaj committed Sep 18, 2024
    Copy the full SHA
    16ecfce View commit details
  2. Copy the full SHA
    4656d05 View commit details

Commits on Sep 20, 2024

  1. Added gapseq but missing some deps

    Need to finish the deps.  Missing sybil, httr, and glpk solver
    billytaj committed Sep 20, 2024
    Copy the full SHA
    454bd3b View commit details
  2. WIP change R version

    Issues with dep'd packages.  Must upgrade to R 4.4.0
    billytaj committed Sep 20, 2024
    Copy the full SHA
    aaafddb View commit details

Commits on Oct 1, 2024

  1. gapseq + deps installed.

    GapSeq passes all tests.
    dockerfile needs trimming.
    reduce RUN statement count.
    billytaj committed Oct 1, 2024
    Copy the full SHA
    5e203ab View commit details
  2. cleaned up dockerfiles

    billytaj committed Oct 1, 2024
    Copy the full SHA
    94b7787 View commit details

Commits on Nov 4, 2024

  1. add gapseq pack

    billytaj committed Nov 4, 2024
    Copy the full SHA
    db2489c View commit details

Commits on Nov 11, 2024

  1. updated dockerfile code to match current version

    was stuck on 1.0.2.  now moved to 1.0.4
    billytaj committed Nov 11, 2024
    Copy the full SHA
    ecf7015 View commit details
Showing with 155 additions and 93 deletions.
  1. +71 −79 Dockerfile
  2. +20 −0 dev_notes.txt
  3. +1 −1 load_test_image.sh
  4. +49 −13 test_dockerfile
  5. +14 −0 tests/test_quackers_config.py
150 changes: 71 additions & 79 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#FROM continuumio/anaconda3
#version 1.0.0:
#version 1.0.3:

FROM ubuntu:22.04
MAINTAINER Billy Law
#MAINTAINER Billy Law

ENV TZ=America/Canada
ENV DEBIAN_FRONTEND=noninteractive
@@ -46,7 +46,7 @@ RUN python3 setup.py install
WORKDIR /quackers_tools
RUN wget http://eddylab.org/software/hmmer/hmmer.tar.gz \
&& tar -xzvf hmmer.tar.gz
WORKDIR hmmer-3.4
WORKDIR /quackers_tools/hmmer-3.4
RUN sh configure && make

WORKDIR /quackers_tools
@@ -74,7 +74,7 @@ RUN wget https://github.com/samtools/samtools/releases/download/1.20/samtools-1.
&& tar -xvf samtools.tar.bz2


WORKDIR samtools-1.20
WORKDIR /quackers_tools/samtools-1.20
RUN sh configure \
&& make \
&& make install
@@ -125,17 +125,6 @@ RUN chmod -R 777 /quackers_tools

RUN apt-get install -y python-profiler

WORKDIR /quackers

RUN wget https://raw.githubusercontent.com/billytaj/quackers/develop/MetaPro_utilities.py
RUN wget https://raw.githubusercontent.com/billytaj/quackers/develop/quackers_commands.py
RUN wget https://raw.githubusercontent.com/billytaj/quackers/develop/quackers_paths.py
RUN wget https://raw.githubusercontent.com/billytaj/quackers/develop/quackers_pipe.py
RUN wget https://raw.githubusercontent.com/billytaj/quackers/develop/quackers_stages.py

#WORKDIR /quackers/scripts
#RUN wget https://raw.githubusercontent.com/billytaj/quackers/develop/scripts/0a_Run_bbduk_trimming_filtering.sh

RUN pip install --force-reinstall -v "scikit-learn==1.1.0"

RUN pip install numpy \
@@ -153,7 +142,7 @@ RUN wget https://github.com/matsen/pplacer/releases/download/v1.1.alpha17/pplace

#for linux-only
WORKDIR /quackers_tools/checkm_data
RUN wget https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz \
RUN wget https://compsysbio.org/quackers_deps/checkm_data_2015_01_16.tar.gz \
&& tar -xzvf checkm_data_2015_01_16.tar.gz \
&& rm *.tar.gz

@@ -171,7 +160,8 @@ RUN wget https://github.com/bxlab/metaWRAP/archive/refs/tags/v1.3.tar.gz \

ENV PATH="${PATH}:/quackers_tools/metaWRAP-1.3/bin"

RUN apt-get install -y git-all \
RUN apt-get update \
&& apt-get install -y git-all \
&& git clone https://github.com/lh3/bwa.git \
&& cd bwa \
&& make
@@ -185,10 +175,6 @@ RUN wget https://bitbucket.org/berkeleylab/metabat/get/37db58fe3fda88f118dfdf188
&& mv berkeleylab-metabat-37db58fe3fda metabat \
&& cd metabat/src

#RUN git clone https://bitbucket.org/berkeleylab/metabat.git #\
#&& git checkout v2.12.1


ENV CONDA_DIR="/opt/conda"
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda
@@ -211,9 +197,6 @@ ENV PATH="${PATH}:/quackers_tools/bowtie2"
WORKDIR /quackers_tools/gtdbtk
RUN python3 setup.py install

#WORKDIR /quackers_tools/gtdbtk_data
#RUN wget https://data.ace.uq.edu.au/public/gtdb/data/releases/latest/auxillary_files/gtdbtk_data.tar.gz \
#&& tar -xzvf gtdbtk_data.tar.gz
ENV GTDBTK_DATA_PATH="/quackers_tools/gtdbtk_data/release214"

WORKDIR /quackers_tools/fastANI
@@ -242,11 +225,6 @@ RUN wget https://github.com/bwa-mem2/bwa-mem2/releases/download/v2.2.1/bwa-mem2-

WORKDIR /quackers_tools

#RUN wget https://github.com/ablab/spades/releases/download/v4.0.0/SPAdes-4.0.0-Linux.tar.gz \
#&& tar -xzf SPAdes-4.0.0-Linux.tar.gz \
#&& mv SPAdes-4.0.0-Linux SPAdes \
#&& rm *.tar.gz \
#&& rm *.tar.bz2
RUN apt-get update \
&& apt-get install -y cmake \
&& apt-get install -y zlib1g-dev \
@@ -256,22 +234,13 @@ RUN wget https://github.com/ablab/spades/archive/refs/tags/v4.0.0.zip \
&& unzip v4.0.0.zip \
&& mv spades-4.0.0 SPAdes

WORKDIR SPAdes
WORKDIR /quackers_tools/SPAdes


RUN sh spades_compile.sh


#WORKDIR SPAdes

#RUN sh spades_compile.sh
#RUN sh "PREFIX=/quackers_tools/SPAdes" spades_compile.sh

#RUN conda install -y spades
#RUN wget https://github.com/ablab/spades/releases/download/v4.0.0/SPAdes-4.0.0-Linux.tar.gz \
#&& tar -xzf SPAdes-4.0.0-Linux.tar.gz \
#&& mv SPAdes-4.0.0-Linux SPAdes \
#&& rm *.tar.gz
ENV PATH="${PATH}:/quackers_tools/SPAdes/bin"
WORKDIR /quackers_tools
RUN chmod -R 777 /quackers_tools/SPAdes
@@ -281,30 +250,6 @@ RUN chmod -R 777 /quackers_tools/SPAdes
RUN apt-get update \
&& apt-get install -y -qq build-essential libgsl0-dev bedtools mummer samtools

#RUN rm *.tar.gz
#RUN pip uninstall pandas
#RUN pip install --force-reinstall -v pandas==1.0.0
#RUN pip install --force-reinstall -v 'pandas==0.13.0'

#RUN conda install -y bioconda::maxbin2
#RUN wget http://compsysbio.org/quackers_deps/MaxBin-2.2.7.tar.gz \
#&& tar -xzvf MaxBin-2.2.7.tar.gz \
#&& rm *.gz \
#&& mv MaxBin-2.2.7 maxbin

#RUN wget https://github.com/edgraham/BinSanity/archive/refs/tags/v0.5.3.zip \
#&& unzip v0.5.3.zip \
#&& mv BinSanity-0.5.3 binsanity
#RUN pip install scikit-learn
#ENV PATH="${PATH}:/quackers_tools/binsanity/bin"


#RUN wget http://compsysbio.org/quackers_deps/subread-2.0.6-Linux-x86_64.tar.gz -O subread.tar.gz \
#&& tar -xzvf subread.tar.gz \
#&& mv subread-2.0.6-Linux-x86_64 subread \
#&& rm *.gz

#ENV PATH="${PATH}:/quackers_tools/subread/bin"

RUN apt-get update \
&& apt-get install perl
@@ -330,9 +275,6 @@ RUN pip2 install matplotlib
RUN apt-get install -y python-tk


#RUN cd biopython-1.76 \
#&& python2 setup.py install

RUN wget https://github.com/bluenote-1577/skani/releases/download/latest/skani \
&& chmod 777 skani

@@ -355,34 +297,84 @@ ENV PATH="${PATH}:/quackers_tools/fasttree"
ENV PATH="${PATH}:/quackers_tools/mash"
ENV PATH="${PATH}:/quackers_tools/adapterremoval"
ENV PATH="${PATH}:/quackers_tools/cdhit_dup"


WORKDIR /quackers_tools/gapseq

RUN apt-get update \
&& apt-get install -y ncbi-blast+ git libglpk-dev r-base-core exonerate bedtools barrnap bc parallel curl libcurl4-openssl-dev libssl-dev libsbml5-dev bc \
&& apt-get update \
&& apt-get install -y r-base \
&& apt-get update \
&& apt-get install curl \
&& apt-get install -y libcurl4-openssl-dev \
&& apt-get update \
&& wget https://compsysbio.org/quackers_deps/working_packages.R \
&& wget https://cran.r-project.org/src/contrib/Archive/sybil/sybil_2.2.0.tar.gz \
&& wget https://cran.r-project.org/src/contrib/Archive/sybilSBML/sybilSBML_3.1.2.tar.gz \
&& Rscript working_packages.R \
&& git clone https://github.com/jotech/gapseq

WORKDIR /quackers_tools/gapseq
RUN wget https://github.com/curl/curl/releases/download/curl-7_55_0/curl-7.55.0.tar.gz \
&& tar -xzvf curl-7.55.0.tar.gz \
&& cd curl-7.55.0 \
&& ./configure \
&& make \
&& make install

#RUN conda install --solver=classic conda-forge::conda-libmamba-solver conda-forge::libmamba conda-forge::libmambapy conda-forge::libarchive
RUN conda install -n base libarchive -c main --force-reinstall --solver classic

RUN apt-get update \
&& conda install -y conda-forge::glpk \
&& conda install -y r::r-httr \
&& conda install -y bioconda::libsbml



RUN apt-get install -y build-essential \
&& wget https://compsysbio.org/quackers_deps/gapseq_r_install.R \
&& wget https://cran.r-project.org/src/contrib/Archive/sybil/sybil_2.2.0.tar.gz \
&& wget https://cran.r-project.org/src/contrib/Archive/sybilSBML/sybilSBML_3.1.2.tar.gz \
&& wget https://github.com/euba/BacArena/archive/refs/tags/v1.8.1.zip



RUN Rscript gapseq_r_install.R

ENV PATH="${PATH}:/quackers_tools/gapseq/gapseq"
ENV PATH="${PATH}:/quackers_tools/megahit/bin"




WORKDIR /quackers_pipe

RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/quackers_pipe.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/quackers_commands.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/MetaPro_utilities.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/quackers_stages.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/quackers_paths.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/quackers_pipe.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/quackers_commands.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/MetaPro_utilities_v2.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/quackers_stages.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/quackers_paths.py

RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/Config.ini
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/Config.ini


WORKDIR /quackers_pipe/scripts


RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/scripts/contig_reconcile.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/scripts/sam_sift.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/scripts/clean_reads_reconcile.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/scripts/contig_reconcile.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/scripts/sam_sift.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/scripts/clean_reads_reconcile.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/tests/test_quackers_config.py

WORKDIR /quackers_pipe/modded_scripts

RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/modded_scripts/concoct_coverage_table.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/modded_scripts/extract_fasta_bins.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/modded_scripts/merge_cutup_clustering.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.2/modded_scripts/print_comment.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/modded_scripts/concoct_coverage_table.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/modded_scripts/extract_fasta_bins.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/modded_scripts/merge_cutup_clustering.py
RUN wget https://raw.githubusercontent.com/ParkinsonLab/quackers/v1.0.4/modded_scripts/print_comment.py



CMD ["bash"]
20 changes: 20 additions & 0 deletions dev_notes.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,23 @@
Oct 01, 2024:
---------------------------------
Most of the native packages are broken for GapSeq install.
Swapped to using conda and it's ok



Sept 19, 2024:
---------------------------------
Input to gapseq is MAGs.
Output of quackers are taxa bins. They're the same thing.
However, there probably needs to be a merge step to force it into 1 file? Maybe? Read more.

OK, MAGs are taxa bins. It's ok. Each bin is a MAG.

Sept 18, 2024:
--------------------------------------------------------------------
Contig resolution has been nerf'd using assumptions. Now installing GapSeq <R tool> and integrating to Quackers


Sept 12, 2024:
---------------------------------------------

2 changes: 1 addition & 1 deletion load_test_image.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
docker container run -it -v /home/billy:/home/billy parkinsonlab/quackers:v1.0.0


docker container run -it -v /home/billy:/home/billy parkinsonlab/quackers:v1.0.3
#for windows
#docker container run -it -v D:/quackers:/quackers_pipe parkinsonlab/quackers:v1.0.0
#docker container run -it -v D:/quackers:/quackers_pipe parkinsonlab/quackers:v0.0.1
Loading