diff --git a/.nojekyll b/.nojekyll index 312ee3d..3ed180b 100644 --- a/.nojekyll +++ b/.nojekyll @@ -1 +1 @@ -bebaa4a0 \ No newline at end of file +d79de239 \ No newline at end of file diff --git a/04_launch.html b/04_launch.html index a6bf9c5..ea49ffa 100644 --- a/04_launch.html +++ b/04_launch.html @@ -280,8 +280,8 @@

Local execution

N E X T F L O W  ~  version 24.10.0
 WARN: It appears you have never run this project before -- Option `-resume` is ignored
-Launching `nf-nest/examples/hello.nf` [nostalgic_carson] DSL2 - revision: 9d1a692a7e
-[96/869cf0] Submitted process > hello
+Launching `nf-nest/examples/hello.nf` [grave_allen] DSL2 - revision: 9d1a692a7e
+[b5/8ac5f1] Submitted process > hello
 Hello world!
@@ -294,8 +294,8 @@

Cluster execution

./nextflow run nf-nest/examples/hello.nf -profile cluster 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/hello.nf` [evil_ardinghelli] DSL2 - revision: 9d1a692a7e
-[96/869cf0] Cached process > hello
+Launching `nf-nest/examples/hello.nf` [romantic_solvay] DSL2 - revision: 9d1a692a7e
+[b5/8ac5f1] Cached process > hello
 Hello world!
diff --git a/05_pkg.html b/05_pkg.html index e5b31ca..d602601 100644 --- a/05_pkg.html +++ b/05_pkg.html @@ -299,9 +299,9 @@

Precompilation on HP ./nextflow run nf-nest/pkg.nf -profile cluster 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/pkg.nf` [stupefied_goldstine] DSL2 - revision: fc0374e695
-[d2/2d9cfa] Submitted process > instantiate_process
-[08/cabe06] Submitted process > precompile
+Launching `nf-nest/pkg.nf` [golden_shaw] DSL2 - revision: fc0374e695 +[9b/6b6098] Submitted process > instantiate_process +[bf/bbd094] Submitted process > precompile

Optionally, you can append an argument to specify the number of threads to request and use during pre-compilation: e.g. add --nPrecompileThreads 20 to request 20 threads instead of the default of 10.

diff --git a/06_cross_product.html b/06_cross_product.html index 9371381..fb69bb9 100644 --- a/06_cross_product.html +++ b/06_cross_product.html @@ -345,43 +345,43 @@

Running the script

N E X T F L O W  ~  version 24.10.0
 WARN: It appears you have never run this project before -- Option `-resume` is ignored
-Launching `nf-nest/examples/many_jobs.nf` [nauseous_volta] DSL2 - revision: aa082b1978
-[cf/316661] Submitted process > run_julia (7)
-[98/3f89f3] Submitted process > run_julia (16)
-[f5/9200b5] Submitted process > run_julia (8)
-[60/79d084] Submitted process > run_julia (10)
-[25/304afb] Submitted process > run_julia (12)
-[dc/f8a7cb] Submitted process > run_julia (1)
-[23/a921d6] Submitted process > run_julia (6)
-[17/60efee] Submitted process > run_julia (3)
-[fe/d3775a] Submitted process > run_julia (2)
-[23/02e5a5] Submitted process > run_julia (11)
-[61/92282e] Submitted process > run_julia (9)
-[d2/4f9929] Submitted process > run_julia (13)
-[c5/278d1e] Submitted process > run_julia (5)
-[ec/ca8573] Submitted process > run_julia (15)
-[b9/99bc51] Submitted process > run_julia (4)
-[a7/6c5a96] Submitted process > run_julia (14)
-[ff/2ee016] Submitted process > run_julia (17)
-[ba/85000e] Submitted process > run_julia (18)
+Launching `nf-nest/examples/many_jobs.nf` [pensive_cuvier] DSL2 - revision: aa082b1978
+[9a/f95c38] Submitted process > run_julia (16)
+[31/c84da1] Submitted process > run_julia (8)
+[70/882e38] Submitted process > run_julia (7)
+[f8/827180] Submitted process > run_julia (9)
+[40/502eb0] Submitted process > run_julia (5)
+[24/235871] Submitted process > run_julia (13)
+[08/c3250a] Submitted process > run_julia (3)
+[2a/d590b5] Submitted process > run_julia (11)
+[1a/18653d] Submitted process > run_julia (6)
+[ba/2ffe85] Submitted process > run_julia (2)
+[4f/48bab2] Submitted process > run_julia (4)
+[55/fb4a96] Submitted process > run_julia (14)
+[6b/ba0d04] Submitted process > run_julia (15)
+[33/3c91f1] Submitted process > run_julia (1)
+[67/d912ab] Submitted process > run_julia (12)
+[42/edc64d] Submitted process > run_julia (10)
+[d0/25bfed] Submitted process > run_julia (18)
+[39/75210b] Submitted process > run_julia (17)
+3 * 2 = 6
+2 + 2 = 4
 2 + 1 = 3
 2 * 1 = 2
-3 * 2 = 6
-2 * 2 = 4
-2 * 3 = 6
-1 + 1 = 2
-1 * 3 = 3
+1 + 3 = 4
+3 + 1 = 4
 1 + 2 = 3
-1 * 1 = 1
 2 + 3 = 5
-2 + 2 = 4
-3 + 1 = 4
-1 + 3 = 4
-3 + 2 = 5
+1 * 3 = 3
+1 * 1 = 1
 1 * 2 = 2
 3 * 1 = 3
-3 + 3 = 6
-3 * 3 = 9
+3 + 2 = 5 +1 + 1 = 2 +2 * 3 = 6 +2 * 2 = 4 +3 * 3 = 9 +3 + 3 = 6
diff --git a/07_combine.html b/07_combine.html index 1968e32..6db14b7 100644 --- a/07_combine.html +++ b/07_combine.html @@ -341,33 +341,33 @@

Running the ne ./nextflow run nf-nest/examples/full.nf -profile cluster 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/full.nf` [agitated_knuth] DSL2 - revision: a68c131baa
-[6d/3357a5] Submitted process > combine_workflow:instantiate_process
-[25/d08519] Submitted process > instantiate_process
-[bb/03d46d] Submitted process > combine_workflow:precompile
-[bb/605315] Submitted process > precompile
-[fa/aa7751] Submitted process > run_julia (9)
-[7b/150dd2] Submitted process > run_julia (8)
-[cd/7a1c39] Submitted process > run_julia (2)
-[df/ffce39] Submitted process > run_julia (13)
-[c8/23f9a2] Submitted process > run_julia (12)
-[b6/ed34c6] Submitted process > run_julia (4)
-[7f/32c560] Submitted process > run_julia (5)
-[23/dca8ec] Submitted process > run_julia (15)
-[3e/bd5153] Submitted process > run_julia (14)
-[8f/1b5deb] Submitted process > run_julia (10)
-[57/f522b3] Submitted process > run_julia (6)
-[60/eb9c28] Submitted process > run_julia (11)
-[9c/ce18ab] Submitted process > run_julia (1)
-[13/599d3d] Submitted process > run_julia (7)
-[cb/d8f2f4] Submitted process > run_julia (3)
-[e0/fad42d] Submitted process > run_julia (16)
-[34/e101c3] Submitted process > run_julia (17)
-[3b/a775ca] Submitted process > run_julia (18)
-[90/4f63ff] Submitted process > run_julia (20)
-[65/b68f71] Submitted process > run_julia (19)
-[4a/d5b369] Submitted process > combine_workflow:combine_process
-[21/1bfa99] Submitted process > plot
+Launching `nf-nest/examples/full.nf` [intergalactic_jepsen] DSL2 - revision: a68c131baa +[9c/0659ed] Submitted process > combine_workflow:instantiate_process +[71/d2fadf] Submitted process > instantiate_process +[ce/f1a696] Submitted process > combine_workflow:precompile +[4d/72b110] Submitted process > precompile +[42/0732cc] Submitted process > run_julia (9) +[5e/81171f] Submitted process > run_julia (1) +[c5/1ccbfe] Submitted process > run_julia (3) +[d4/228929] Submitted process > run_julia (4) +[2b/2baf59] Submitted process > run_julia (14) +[8c/2bb063] Submitted process > run_julia (15) +[d5/5001bb] Submitted process > run_julia (13) +[56/de7ec7] Submitted process > run_julia (7) +[ab/d18893] Submitted process > run_julia (11) +[76/fc81a9] Submitted process > run_julia (5) +[69/6f13e4] Submitted process > run_julia (10) +[28/fa731e] Submitted process > run_julia (2) +[61/f7b16b] Submitted process > run_julia (12) +[8e/3df7d5] Submitted process > run_julia (16) +[d3/fe1273] Submitted process > run_julia (6) +[56/f5fc6a] Submitted process > run_julia (8) +[ee/683820] Submitted process > run_julia (17) +[58/7fa150] Submitted process > run_julia (18) +[28/2a8382] Submitted process > run_julia (19) +[c2/f5b56b] Submitted process > run_julia (20) +[52/031bf9] Submitted process > combine_workflow:combine_process +[c0/9912ef] Submitted process > plot
@@ -388,7 +388,7 @@

Or 
tree experiment_repo/deliverables
experiment_repo/deliverables
-└── scriptName=full.nf___runName=agitated_knuth
+└── scriptName=full.nf___runName=intergalactic_jepsen
     ├── output
     │   ├── summary.csv
     │   └── swap_prs.csv
@@ -402,11 +402,11 @@ 
Or
 
cd experiment_repo
 ./nextflow log

 

-
TIMESTAMP           DURATION    RUN NAME            STATUS  REVISION ID SESSION ID                              COMMAND                                                
-2024-11-12 07:05:30 3.2s        nostalgic_carson    OK      9d1a692a7e  317a6835-37ee-4f54-9d9f-b0df69f57bbd    nextflow run nf-nest/examples/hello.nf                 
-2024-11-12 07:05:36 7.6s        evil_ardinghelli    OK      9d1a692a7e  317a6835-37ee-4f54-9d9f-b0df69f57bbd    nextflow run nf-nest/examples/hello.nf -profile cluster
-2024-11-12 07:06:32 1m 22s      stupefied_goldstine OK      fc0374e695  317a6835-37ee-4f54-9d9f-b0df69f57bbd    nextflow run nf-nest/pkg.nf -profile cluster           
-2024-11-12 07:08:17 4m 33s      agitated_knuth      OK      a68c131baa  317a6835-37ee-4f54-9d9f-b0df69f57bbd    nextflow run nf-nest/examples/full.nf -profile cluster 

+
TIMESTAMP           DURATION    RUN NAME                STATUS  REVISION ID SESSION ID                              COMMAND                                                
+2024-11-12 09:04:09 3.5s        grave_allen             OK      9d1a692a7e  bf02b0fb-fa7d-4171-88a3-e89239ee4248    nextflow run nf-nest/examples/hello.nf                 
+2024-11-12 09:04:16 6.9s        romantic_solvay         OK      9d1a692a7e  bf02b0fb-fa7d-4171-88a3-e89239ee4248    nextflow run nf-nest/examples/hello.nf -profile cluster
+2024-11-12 09:05:11 1m 13s      golden_shaw             OK      fc0374e695  bf02b0fb-fa7d-4171-88a3-e89239ee4248    nextflow run nf-nest/pkg.nf -profile cluster           
+2024-11-12 09:06:48 3m 8s       intergalactic_jepsen    OK      a68c131baa  bf02b0fb-fa7d-4171-88a3-e89239ee4248    nextflow run nf-nest/examples/full.nf -profile cluster
diff --git a/08_containers.html b/08_containers.html index 19cec01..8d76c2b 100644 --- a/08_containers.html +++ b/08_containers.html @@ -339,11 +339,11 @@

Using an exist ./nextflow run nf-nest/examples/stan_example.nf -profile cluster 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/stan_example.nf` [kickass_northcutt] DSL2 - revision: 8cef9f29d6
+Launching `nf-nest/examples/stan_example.nf` [festering_khorana] DSL2 - revision: 8cef9f29d6
 WARN: Apptainer cache directory has not been defined -- Remote image will be stored in the path: /arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work/singularity -- Use the environment variable NXF_APPTAINER_CACHEDIR to specify a different location
 Pulling Apptainer image docker://alexandrebouchardcote/default:0.1.6 [cache /arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work/singularity/alexandrebouchardcote-default-0.1.6.img]
-[0b/5344f6] Submitted process > compile_stan
-[9b/4fd335] Submitted process > run_stan
+[0d/7c12aa] Submitted process > compile_stan +[04/d00cdc] Submitted process > run_stan
diff --git a/09_gpu.html b/09_gpu.html index 3330197..8f20b9d 100644 --- a/09_gpu.html +++ b/09_gpu.html @@ -254,9 +254,9 @@

Installing CUDA.jl

./nextflow run nf-nest/pkg_gpu.nf 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/pkg_gpu.nf` [insane_monod] DSL2 - revision: 713b74ac4a
-[0a/d32c39] Submitted process > instantiate_process
-[9a/f339af] Submitted process > precompile_gpu
+Launching `nf-nest/pkg_gpu.nf` [hopeful_turing] DSL2 - revision: 713b74ac4a +[b6/0ba7e0] Submitted process > instantiate_process +[70/2ce6ba] Submitted process > precompile_gpu
@@ -303,16 +303,16 @@

R ./nextflow run nf-nest/examples/gpu.nf -profile cluster 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/gpu.nf` [sick_legentil] DSL2 - revision: 9be41cea49
-[70/537eb2] Submitted process > instantiate_process
-[d4/f4b844] Submitted process > precompile_gpu
-[05/4b36e9] Submitted process > run_julia
+Launching `nf-nest/examples/gpu.nf` [thirsty_mahavira] DSL2 - revision: 9be41cea49
+[b6/49dd3e] Submitted process > instantiate_process
+[80/583246] Submitted process > precompile_gpu
+[fc/7402e7] Submitted process > run_julia
 CPU
- 15.295592 seconds (4.79 M allocations: 517.701 MiB, 4.03% gc time, 18.96% compilation time)
- 12.138829 seconds (2 allocations: 190.735 MiB, 0.06% gc time)
+ 13.046295 seconds (4.79 M allocations: 517.701 MiB, 3.56% gc time, 17.49% compilation time)
+ 10.582634 seconds (2 allocations: 190.735 MiB, 0.10% gc time)
 GPU
-  1.972740 seconds (1.67 M allocations: 107.375 MiB, 95.51% compilation time)
-  0.000784 seconds (50 allocations: 1.172 KiB)
+  1.535394 seconds (1.67 M allocations: 107.375 MiB, 96.04% compilation time)
+  0.000591 seconds (50 allocations: 1.172 KiB)
 CUDA runtime 12.5, artifact installation
 CUDA driver 12.6
 NVIDIA driver 550.90.12
@@ -339,7 +339,7 @@ 
R
 - CUDA_Runtime_jll.version: 12.5
 
 1 device:
-  0: Tesla V100-SXM2-16GB (sm_70, 15.171 GiB / 16.000 GiB available)
+ 0: Tesla V100-SXM2-32GB (sm_70, 31.137 GiB / 32.000 GiB available)
diff --git a/10_misc.html b/10_misc.html index 8af5ba2..74ef796 100644 --- a/10_misc.html +++ b/10_misc.html @@ -317,33 +317,33 @@

Process parameters

./nextflow run nf-nest/examples/params.nf -profile cluster --n_rounds 6 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/params.nf` [trusting_golick] DSL2 - revision: 093b673fc8
-[25/d08519] Cached process > instantiate_process
-[6d/3357a5] Cached process > combine_workflow:instantiate_process
-[bb/605315] Cached process > precompile
-[bb/03d46d] Cached process > combine_workflow:precompile
-[4a/da0561] Submitted process > run_julia (15)
-[21/dfa364] Submitted process > run_julia (1)
-[b3/84c4eb] Submitted process > run_julia (10)
-[56/ec6f27] Submitted process > run_julia (8)
-[08/9b81df] Submitted process > run_julia (3)
-[91/4196b1] Submitted process > run_julia (4)
-[28/134b49] Submitted process > run_julia (11)
-[11/dfdb52] Submitted process > run_julia (6)
-[a4/0f7727] Submitted process > run_julia (13)
-[34/dcdf1e] Submitted process > run_julia (14)
-[e6/7f95d3] Submitted process > run_julia (7)
-[30/b45ad4] Submitted process > run_julia (2)
-[a3/cbce26] Submitted process > run_julia (5)
-[37/5052ca] Submitted process > run_julia (12)
-[e8/013bb9] Submitted process > run_julia (16)
-[9c/870d41] Submitted process > run_julia (9)
-[7b/56406a] Submitted process > run_julia (17)
-[dd/b2182f] Submitted process > run_julia (19)
-[8c/d13992] Submitted process > run_julia (18)
-[08/f4c162] Submitted process > run_julia (20)
-[47/a869eb] Submitted process > combine_workflow:combine_process
-[84/61c7ce] Submitted process > plot
+Launching `nf-nest/examples/params.nf` [happy_joliot] DSL2 - revision: 093b673fc8 +[71/d2fadf] Cached process > instantiate_process +[9c/0659ed] Cached process > combine_workflow:instantiate_process +[4d/72b110] Cached process > precompile +[ce/f1a696] Cached process > combine_workflow:precompile +[4b/3e423b] Submitted process > run_julia (16) +[d7/3af141] Submitted process > run_julia (4) +[b8/d7e4dc] Submitted process > run_julia (7) +[bb/0a7cfd] Submitted process > run_julia (3) +[e9/ae5559] Submitted process > run_julia (9) +[e7/fcbb74] Submitted process > run_julia (12) +[da/f228ef] Submitted process > run_julia (1) +[79/f258e4] Submitted process > run_julia (13) +[3b/21af06] Submitted process > run_julia (10) +[b8/dcf706] Submitted process > run_julia (11) +[d1/4a8583] Submitted process > run_julia (2) +[d8/d86e53] Submitted process > run_julia (5) +[0e/b0306c] Submitted process > run_julia (8) +[22/a8b690] Submitted process > run_julia (6) +[87/075a3e] Submitted process > run_julia (15) +[34/dd1cd7] Submitted process > run_julia (14) +[cb/757346] Submitted process > run_julia (17) +[85/7bb0a7] Submitted process > run_julia (18) +[8e/66a00a] Submitted process > run_julia (20) +[fe/7e6fa6] Submitted process > run_julia (19) +[09/f1b535] Submitted process > combine_workflow:combine_process +[64/776598] Submitted process > plot

Notice that the function deliverables(workflow, params) takes it into account so that the deliverables directory is organized correctly:

@@ -351,12 +351,12 @@

Process parameters

tree experiment_repo/deliverables
experiment_repo/deliverables
-├── scriptName=full.nf___runName=agitated_knuth
+├── scriptName=full.nf___runName=intergalactic_jepsen
 │   ├── output
 │   │   ├── summary.csv
 │   │   └── swap_prs.csv
 │   └── plot.png
-└── scriptName=params.nf___runName=trusting_golick___n_rounds=6
+└── scriptName=params.nf___runName=happy_joliot___n_rounds=6
     ├── output
     │   ├── summary.csv
     │   └── swap_prs.csv
@@ -414,10 +414,10 @@ 
Dry runs

 ./nextflow run nf-nest/examples/dry_run.nf -profile cluster --dryRun
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/dry_run.nf` [condescending_curie] DSL2 - revision: c7304bca14
-[25/d08519] Cached process > instantiate_process
-[bb/605315] Cached process > precompile
-[7d/a7a92f] Submitted process > run_julia (1)
+Launching `nf-nest/examples/dry_run.nf` [lonely_meninsky] DSL2 - revision: c7304bca14 +[71/d2fadf] Cached process > instantiate_process +[4d/72b110] Cached process > precompile +[29/6b28ad] Submitted process > run_julia (1)
@@ -461,8 +461,8 @@

Lightweight code

./nextflow run nf-nest/examples/includes.nf 
N E X T F L O W  ~  version 24.10.0
-Launching `nf-nest/examples/includes.nf` [shrivelled_engelbart] DSL2 - revision: f263a35a46
-[9a/1a9b97] Cached process > run_julia
+Launching `nf-nest/examples/includes.nf` [mad_solvay] DSL2 - revision: f263a35a46
+[61/96fce3] Submitted process > run_julia
 hello
 world
diff --git a/experiment_repo/dag.html b/experiment_repo/dag.html new file mode 100644 index 0000000..6f45c22 --- /dev/null +++ b/experiment_repo/dag.html @@ -0,0 +1,35 @@ + + + + + + +
+flowchart TB
+    subgraph " "
+    v0["julia_files"]
+    end
+    v1([run_julia])
+    v0 --> v1
+
+
+ + + diff --git a/experiment_repo/report.html b/experiment_repo/report.html new file mode 100644 index 0000000..f818cad --- /dev/null +++ b/experiment_repo/report.html @@ -0,0 +1,1046 @@ + + + + + + + + + + + [mad_solvay] Nextflow Workflow Report + + + + + + + +
+
+ +

Nextflow workflow report

+

[mad_solvay] (resumed run)

+ + +
+ Workflow execution completed successfully! +
+ + +
+
Run times
+
+ 12-Nov-2024 09:17:20 - 12-Nov-2024 09:17:23 + (duration: 2.4s) +
+ +
+
+
  1 succeeded  
+
  0 cached  
+
  0 ignored  
+
  0 failed  
+
+
+ +
Nextflow command
+ 
nextflow run nf-nest/examples/includes.nf
+
+ +
+
CPU-Hours
+
(a few seconds)
+ +
Launch directory
+
/arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo
+ +
Work directory
+
/arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work
+ +
Project directory
+
/arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/nf-nest/examples
+ + +
Script name
+
includes.nf
+ + + +
Script ID
+
f263a35a4664374d4927f0f3364e618d
+ + +
Workflow session
+
bf02b0fb-fa7d-4171-88a3-e89239ee4248
+ + + +
Workflow profile
+
standard
+ + +
Workflow container
+
[withLabel:containerized:alexandrebouchardcote/default:0.1.6]
+ +
Container engine
+
docker
+ + + + +
Nextflow version
+
version 24.10.0, build 5928 (27-10-2024 18:36 UTC)
+
+
+
+ +
+

Resource Usage

+

These plots give an overview of the distribution of resource usage for each process.

+ +

CPU

+ +
+
+
+
+
+
+
+ +
+ +

Memory

+ +
+
+
+
+
+
+
+
+
+
+
+ +

Job Duration

+ +
+
+
+
+
+
+
+
+ +

I/O

+ +
+
+
+
+
+
+
+
+
+ +
+
+

Tasks

+

This table shows information about each task in the workflow. Use the search box on the right + to filter rows for specific values. Clicking headers will sort the table by that value and + scrolling side to side will reveal more columns.

+
+ + +
+
+
+
+
+ +
+ (tasks table omitted because the dataset is too big) +
+
+ + + + + + + diff --git a/experiment_repo/timeline.html b/experiment_repo/timeline.html new file mode 100644 index 0000000..60d1949 --- /dev/null +++ b/experiment_repo/timeline.html @@ -0,0 +1,223 @@ + + + + + + + + + + + + + +
+

Processes execution timeline

+

+ Launch time:
+ Elapsed time:
+ Legend: job wall time / memory usage (RAM) +

+
+
+ + + + + + + diff --git a/search.json b/search.json index 7236641..e4ae95d 100644 --- a/search.json +++ b/search.json @@ -4,7 +4,7 @@ "href": "10_misc.html", "title": "Misc", "section": "", - "text": "To declare a command line argument, say n_rounds to your nextflow script, with default argument 5, use:\nparams.n_rounds = 5\nHere is an example:\n\ninclude { crossProduct; filed; deliverables } from '../cross.nf'\ninclude { instantiate; precompile; activate } from '../pkg.nf'\ninclude { combine_csvs; } from '../combine.nf'\n\ndef julia_env = file(moduleDir/'julia_env')\ndef plot_script = file(moduleDir/'plot.jl')\n\nparams.n_rounds = 5\n\ndef variables = [\n seed: 1..10,\n n_chains: [10, 20], \n]\n\nworkflow {\n compiled_env = instantiate(julia_env) | precompile\n configs = crossProduct(variables)\n combined = run_julia(compiled_env, configs) | combine_csvs\n plot(compiled_env, plot_script, combined)\n}\n\nprocess run_julia {\n input:\n path julia_env \n val config \n output:\n path \"${filed(config)}\"\n \"\"\"\n ${activate(julia_env)}\n\n # run your code\n using Pigeons \n using CSV \n pt = pigeons(\n target = toy_mvn_target(1000), \n n_chains = ${config.n_chains}, \n seed = ${config.seed},\n n_rounds = ${params.n_rounds})\n\n mkdir(\"${filed(config)}\")\n CSV.write(\"${filed(config)}/summary.csv\", pt.shared.reports.summary)\n CSV.write(\"${filed(config)}/swap_prs.csv\", pt.shared.reports.swap_prs)\n \"\"\"\n}\n\nprocess plot {\n input:\n path julia_env \n path plot_script\n path combined_csvs_folder \n output:\n path '*.png'\n path combined_csvs_folder\n publishDir \"${deliverables(workflow, params)}\", mode: 'copy', overwrite: true\n \"\"\"\n ${activate(julia_env)}\n\n include(\"$plot_script\")\n create_plots(\"$combined_csvs_folder\")\n \"\"\"\n}\n\nTo run it, notice that the arguments specified by params.my_arg should be specified using --my_arg value (in contrast, nextflow’s argument use a single dash, as in -profile cluster):\n\ncd experiment_repo\n./nextflow run nf-nest/examples/params.nf -profile cluster --n_rounds 6\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/params.nf` [trusting_golick] DSL2 - revision: 093b673fc8\n[25/d08519] Cached process > instantiate_process\n[6d/3357a5] Cached process > combine_workflow:instantiate_process\n[bb/605315] Cached process > precompile\n[bb/03d46d] Cached process > combine_workflow:precompile\n[4a/da0561] Submitted process > run_julia (15)\n[21/dfa364] Submitted process > run_julia (1)\n[b3/84c4eb] Submitted process > run_julia (10)\n[56/ec6f27] Submitted process > run_julia (8)\n[08/9b81df] Submitted process > run_julia (3)\n[91/4196b1] Submitted process > run_julia (4)\n[28/134b49] Submitted process > run_julia (11)\n[11/dfdb52] Submitted process > run_julia (6)\n[a4/0f7727] Submitted process > run_julia (13)\n[34/dcdf1e] Submitted process > run_julia (14)\n[e6/7f95d3] Submitted process > run_julia (7)\n[30/b45ad4] Submitted process > run_julia (2)\n[a3/cbce26] Submitted process > run_julia (5)\n[37/5052ca] Submitted process > run_julia (12)\n[e8/013bb9] Submitted process > run_julia (16)\n[9c/870d41] Submitted process > run_julia (9)\n[7b/56406a] Submitted process > run_julia (17)\n[dd/b2182f] Submitted process > run_julia (19)\n[8c/d13992] Submitted process > run_julia (18)\n[08/f4c162] Submitted process > run_julia (20)\n[47/a869eb] Submitted process > combine_workflow:combine_process\n[84/61c7ce] Submitted process > plot\n\n\nNotice that the function deliverables(workflow, params) takes it into account so that the deliverables directory is organized correctly:\n\ntree experiment_repo/deliverables\n\nexperiment_repo/deliverables\n├── scriptName=full.nf___runName=agitated_knuth\n│   ├── output\n│   │   ├── summary.csv\n│   │   └── swap_prs.csv\n│   └── plot.png\n└── scriptName=params.nf___runName=trusting_golick___n_rounds=6\n ├── output\n │   ├── summary.csv\n │   └── swap_prs.csv\n └── plot.png\n\n4 directories, 6 files", + "text": "To declare a command line argument, say n_rounds to your nextflow script, with default argument 5, use:\nparams.n_rounds = 5\nHere is an example:\n\ninclude { crossProduct; filed; deliverables } from '../cross.nf'\ninclude { instantiate; precompile; activate } from '../pkg.nf'\ninclude { combine_csvs; } from '../combine.nf'\n\ndef julia_env = file(moduleDir/'julia_env')\ndef plot_script = file(moduleDir/'plot.jl')\n\nparams.n_rounds = 5\n\ndef variables = [\n seed: 1..10,\n n_chains: [10, 20], \n]\n\nworkflow {\n compiled_env = instantiate(julia_env) | precompile\n configs = crossProduct(variables)\n combined = run_julia(compiled_env, configs) | combine_csvs\n plot(compiled_env, plot_script, combined)\n}\n\nprocess run_julia {\n input:\n path julia_env \n val config \n output:\n path \"${filed(config)}\"\n \"\"\"\n ${activate(julia_env)}\n\n # run your code\n using Pigeons \n using CSV \n pt = pigeons(\n target = toy_mvn_target(1000), \n n_chains = ${config.n_chains}, \n seed = ${config.seed},\n n_rounds = ${params.n_rounds})\n\n mkdir(\"${filed(config)}\")\n CSV.write(\"${filed(config)}/summary.csv\", pt.shared.reports.summary)\n CSV.write(\"${filed(config)}/swap_prs.csv\", pt.shared.reports.swap_prs)\n \"\"\"\n}\n\nprocess plot {\n input:\n path julia_env \n path plot_script\n path combined_csvs_folder \n output:\n path '*.png'\n path combined_csvs_folder\n publishDir \"${deliverables(workflow, params)}\", mode: 'copy', overwrite: true\n \"\"\"\n ${activate(julia_env)}\n\n include(\"$plot_script\")\n create_plots(\"$combined_csvs_folder\")\n \"\"\"\n}\n\nTo run it, notice that the arguments specified by params.my_arg should be specified using --my_arg value (in contrast, nextflow’s argument use a single dash, as in -profile cluster):\n\ncd experiment_repo\n./nextflow run nf-nest/examples/params.nf -profile cluster --n_rounds 6\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/params.nf` [happy_joliot] DSL2 - revision: 093b673fc8\n[71/d2fadf] Cached process > instantiate_process\n[9c/0659ed] Cached process > combine_workflow:instantiate_process\n[4d/72b110] Cached process > precompile\n[ce/f1a696] Cached process > combine_workflow:precompile\n[4b/3e423b] Submitted process > run_julia (16)\n[d7/3af141] Submitted process > run_julia (4)\n[b8/d7e4dc] Submitted process > run_julia (7)\n[bb/0a7cfd] Submitted process > run_julia (3)\n[e9/ae5559] Submitted process > run_julia (9)\n[e7/fcbb74] Submitted process > run_julia (12)\n[da/f228ef] Submitted process > run_julia (1)\n[79/f258e4] Submitted process > run_julia (13)\n[3b/21af06] Submitted process > run_julia (10)\n[b8/dcf706] Submitted process > run_julia (11)\n[d1/4a8583] Submitted process > run_julia (2)\n[d8/d86e53] Submitted process > run_julia (5)\n[0e/b0306c] Submitted process > run_julia (8)\n[22/a8b690] Submitted process > run_julia (6)\n[87/075a3e] Submitted process > run_julia (15)\n[34/dd1cd7] Submitted process > run_julia (14)\n[cb/757346] Submitted process > run_julia (17)\n[85/7bb0a7] Submitted process > run_julia (18)\n[8e/66a00a] Submitted process > run_julia (20)\n[fe/7e6fa6] Submitted process > run_julia (19)\n[09/f1b535] Submitted process > combine_workflow:combine_process\n[64/776598] Submitted process > plot\n\n\nNotice that the function deliverables(workflow, params) takes it into account so that the deliverables directory is organized correctly:\n\ntree experiment_repo/deliverables\n\nexperiment_repo/deliverables\n├── scriptName=full.nf___runName=intergalactic_jepsen\n│   ├── output\n│   │   ├── summary.csv\n│   │   └── swap_prs.csv\n│   └── plot.png\n└── scriptName=params.nf___runName=happy_joliot___n_rounds=6\n ├── output\n │   ├── summary.csv\n │   └── swap_prs.csv\n └── plot.png\n\n4 directories, 6 files", "crumbs": [ "Misc" ] @@ -14,7 +14,7 @@ "href": "10_misc.html#process-parameters", "title": "Misc", "section": "", - "text": "To declare a command line argument, say n_rounds to your nextflow script, with default argument 5, use:\nparams.n_rounds = 5\nHere is an example:\n\ninclude { crossProduct; filed; deliverables } from '../cross.nf'\ninclude { instantiate; precompile; activate } from '../pkg.nf'\ninclude { combine_csvs; } from '../combine.nf'\n\ndef julia_env = file(moduleDir/'julia_env')\ndef plot_script = file(moduleDir/'plot.jl')\n\nparams.n_rounds = 5\n\ndef variables = [\n seed: 1..10,\n n_chains: [10, 20], \n]\n\nworkflow {\n compiled_env = instantiate(julia_env) | precompile\n configs = crossProduct(variables)\n combined = run_julia(compiled_env, configs) | combine_csvs\n plot(compiled_env, plot_script, combined)\n}\n\nprocess run_julia {\n input:\n path julia_env \n val config \n output:\n path \"${filed(config)}\"\n \"\"\"\n ${activate(julia_env)}\n\n # run your code\n using Pigeons \n using CSV \n pt = pigeons(\n target = toy_mvn_target(1000), \n n_chains = ${config.n_chains}, \n seed = ${config.seed},\n n_rounds = ${params.n_rounds})\n\n mkdir(\"${filed(config)}\")\n CSV.write(\"${filed(config)}/summary.csv\", pt.shared.reports.summary)\n CSV.write(\"${filed(config)}/swap_prs.csv\", pt.shared.reports.swap_prs)\n \"\"\"\n}\n\nprocess plot {\n input:\n path julia_env \n path plot_script\n path combined_csvs_folder \n output:\n path '*.png'\n path combined_csvs_folder\n publishDir \"${deliverables(workflow, params)}\", mode: 'copy', overwrite: true\n \"\"\"\n ${activate(julia_env)}\n\n include(\"$plot_script\")\n create_plots(\"$combined_csvs_folder\")\n \"\"\"\n}\n\nTo run it, notice that the arguments specified by params.my_arg should be specified using --my_arg value (in contrast, nextflow’s argument use a single dash, as in -profile cluster):\n\ncd experiment_repo\n./nextflow run nf-nest/examples/params.nf -profile cluster --n_rounds 6\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/params.nf` [trusting_golick] DSL2 - revision: 093b673fc8\n[25/d08519] Cached process > instantiate_process\n[6d/3357a5] Cached process > combine_workflow:instantiate_process\n[bb/605315] Cached process > precompile\n[bb/03d46d] Cached process > combine_workflow:precompile\n[4a/da0561] Submitted process > run_julia (15)\n[21/dfa364] Submitted process > run_julia (1)\n[b3/84c4eb] Submitted process > run_julia (10)\n[56/ec6f27] Submitted process > run_julia (8)\n[08/9b81df] Submitted process > run_julia (3)\n[91/4196b1] Submitted process > run_julia (4)\n[28/134b49] Submitted process > run_julia (11)\n[11/dfdb52] Submitted process > run_julia (6)\n[a4/0f7727] Submitted process > run_julia (13)\n[34/dcdf1e] Submitted process > run_julia (14)\n[e6/7f95d3] Submitted process > run_julia (7)\n[30/b45ad4] Submitted process > run_julia (2)\n[a3/cbce26] Submitted process > run_julia (5)\n[37/5052ca] Submitted process > run_julia (12)\n[e8/013bb9] Submitted process > run_julia (16)\n[9c/870d41] Submitted process > run_julia (9)\n[7b/56406a] Submitted process > run_julia (17)\n[dd/b2182f] Submitted process > run_julia (19)\n[8c/d13992] Submitted process > run_julia (18)\n[08/f4c162] Submitted process > run_julia (20)\n[47/a869eb] Submitted process > combine_workflow:combine_process\n[84/61c7ce] Submitted process > plot\n\n\nNotice that the function deliverables(workflow, params) takes it into account so that the deliverables directory is organized correctly:\n\ntree experiment_repo/deliverables\n\nexperiment_repo/deliverables\n├── scriptName=full.nf___runName=agitated_knuth\n│   ├── output\n│   │   ├── summary.csv\n│   │   └── swap_prs.csv\n│   └── plot.png\n└── scriptName=params.nf___runName=trusting_golick___n_rounds=6\n ├── output\n │   ├── summary.csv\n │   └── swap_prs.csv\n └── plot.png\n\n4 directories, 6 files", + "text": "To declare a command line argument, say n_rounds to your nextflow script, with default argument 5, use:\nparams.n_rounds = 5\nHere is an example:\n\ninclude { crossProduct; filed; deliverables } from '../cross.nf'\ninclude { instantiate; precompile; activate } from '../pkg.nf'\ninclude { combine_csvs; } from '../combine.nf'\n\ndef julia_env = file(moduleDir/'julia_env')\ndef plot_script = file(moduleDir/'plot.jl')\n\nparams.n_rounds = 5\n\ndef variables = [\n seed: 1..10,\n n_chains: [10, 20], \n]\n\nworkflow {\n compiled_env = instantiate(julia_env) | precompile\n configs = crossProduct(variables)\n combined = run_julia(compiled_env, configs) | combine_csvs\n plot(compiled_env, plot_script, combined)\n}\n\nprocess run_julia {\n input:\n path julia_env \n val config \n output:\n path \"${filed(config)}\"\n \"\"\"\n ${activate(julia_env)}\n\n # run your code\n using Pigeons \n using CSV \n pt = pigeons(\n target = toy_mvn_target(1000), \n n_chains = ${config.n_chains}, \n seed = ${config.seed},\n n_rounds = ${params.n_rounds})\n\n mkdir(\"${filed(config)}\")\n CSV.write(\"${filed(config)}/summary.csv\", pt.shared.reports.summary)\n CSV.write(\"${filed(config)}/swap_prs.csv\", pt.shared.reports.swap_prs)\n \"\"\"\n}\n\nprocess plot {\n input:\n path julia_env \n path plot_script\n path combined_csvs_folder \n output:\n path '*.png'\n path combined_csvs_folder\n publishDir \"${deliverables(workflow, params)}\", mode: 'copy', overwrite: true\n \"\"\"\n ${activate(julia_env)}\n\n include(\"$plot_script\")\n create_plots(\"$combined_csvs_folder\")\n \"\"\"\n}\n\nTo run it, notice that the arguments specified by params.my_arg should be specified using --my_arg value (in contrast, nextflow’s argument use a single dash, as in -profile cluster):\n\ncd experiment_repo\n./nextflow run nf-nest/examples/params.nf -profile cluster --n_rounds 6\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/params.nf` [happy_joliot] DSL2 - revision: 093b673fc8\n[71/d2fadf] Cached process > instantiate_process\n[9c/0659ed] Cached process > combine_workflow:instantiate_process\n[4d/72b110] Cached process > precompile\n[ce/f1a696] Cached process > combine_workflow:precompile\n[4b/3e423b] Submitted process > run_julia (16)\n[d7/3af141] Submitted process > run_julia (4)\n[b8/d7e4dc] Submitted process > run_julia (7)\n[bb/0a7cfd] Submitted process > run_julia (3)\n[e9/ae5559] Submitted process > run_julia (9)\n[e7/fcbb74] Submitted process > run_julia (12)\n[da/f228ef] Submitted process > run_julia (1)\n[79/f258e4] Submitted process > run_julia (13)\n[3b/21af06] Submitted process > run_julia (10)\n[b8/dcf706] Submitted process > run_julia (11)\n[d1/4a8583] Submitted process > run_julia (2)\n[d8/d86e53] Submitted process > run_julia (5)\n[0e/b0306c] Submitted process > run_julia (8)\n[22/a8b690] Submitted process > run_julia (6)\n[87/075a3e] Submitted process > run_julia (15)\n[34/dd1cd7] Submitted process > run_julia (14)\n[cb/757346] Submitted process > run_julia (17)\n[85/7bb0a7] Submitted process > run_julia (18)\n[8e/66a00a] Submitted process > run_julia (20)\n[fe/7e6fa6] Submitted process > run_julia (19)\n[09/f1b535] Submitted process > combine_workflow:combine_process\n[64/776598] Submitted process > plot\n\n\nNotice that the function deliverables(workflow, params) takes it into account so that the deliverables directory is organized correctly:\n\ntree experiment_repo/deliverables\n\nexperiment_repo/deliverables\n├── scriptName=full.nf___runName=intergalactic_jepsen\n│   ├── output\n│   │   ├── summary.csv\n│   │   └── swap_prs.csv\n│   └── plot.png\n└── scriptName=params.nf___runName=happy_joliot___n_rounds=6\n ├── output\n │   ├── summary.csv\n │   └── swap_prs.csv\n └── plot.png\n\n4 directories, 6 files", "crumbs": [ "Misc" ] @@ -24,7 +24,7 @@ "href": "10_misc.html#dry-runs", "title": "Misc", "section": "Dry runs", - "text": "Dry runs\nA useful application of process parameter is a “dry run switch” for doing a quick version of the pipeline to help quickly debugging.\nHere is an example below. Notice that we pass the dry run option to crossProduct(); instead of emitting all values in the cross product, it will only emit one:\n\ninclude { crossProduct; filed; deliverables } from '../cross.nf'\ninclude { instantiate; precompile; activate } from '../pkg.nf'\n\ndef julia_env = file(moduleDir/'julia_env')\n\nparams.dryRun = false\nparams.n_rounds = params.dryRun ? 1 : 5\n\ndef variables = [\n seed: 1..10,\n n_chains: [10, 20], \n]\n\nworkflow {\n compiled_env = instantiate(julia_env) | precompile\n configs = crossProduct(variables, params.dryRun)\n run_julia(compiled_env, configs) \n}\n\nprocess run_julia {\n input:\n path julia_env \n val config \n \"\"\"\n ${activate(julia_env)}\n\n # run your code\n using Pigeons \n using CSV \n pt = pigeons(\n target = toy_mvn_target(1000), \n n_chains = ${config.n_chains}, \n seed = ${config.seed},\n n_rounds = ${params.n_rounds})\n \"\"\"\n}\n\nTo run it in dry run model:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/dry_run.nf -profile cluster --dryRun\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/dry_run.nf` [condescending_curie] DSL2 - revision: c7304bca14\n[25/d08519] Cached process > instantiate_process\n[bb/605315] Cached process > precompile\n[7d/a7a92f] Submitted process > run_julia (1)", + "text": "Dry runs\nA useful application of process parameter is a “dry run switch” for doing a quick version of the pipeline to help quickly debugging.\nHere is an example below. Notice that we pass the dry run option to crossProduct(); instead of emitting all values in the cross product, it will only emit one:\n\ninclude { crossProduct; filed; deliverables } from '../cross.nf'\ninclude { instantiate; precompile; activate } from '../pkg.nf'\n\ndef julia_env = file(moduleDir/'julia_env')\n\nparams.dryRun = false\nparams.n_rounds = params.dryRun ? 1 : 5\n\ndef variables = [\n seed: 1..10,\n n_chains: [10, 20], \n]\n\nworkflow {\n compiled_env = instantiate(julia_env) | precompile\n configs = crossProduct(variables, params.dryRun)\n run_julia(compiled_env, configs) \n}\n\nprocess run_julia {\n input:\n path julia_env \n val config \n \"\"\"\n ${activate(julia_env)}\n\n # run your code\n using Pigeons \n using CSV \n pt = pigeons(\n target = toy_mvn_target(1000), \n n_chains = ${config.n_chains}, \n seed = ${config.seed},\n n_rounds = ${params.n_rounds})\n \"\"\"\n}\n\nTo run it in dry run model:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/dry_run.nf -profile cluster --dryRun\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/dry_run.nf` [lonely_meninsky] DSL2 - revision: c7304bca14\n[71/d2fadf] Cached process > instantiate_process\n[4d/72b110] Cached process > precompile\n[29/6b28ad] Submitted process > run_julia (1)", "crumbs": [ "Misc" ] @@ -44,7 +44,7 @@ "href": "10_misc.html#updating-code", "title": "Misc", "section": "Updating code", - "text": "Updating code\nNumerical experiment are often based on code you are developing along the way. When the code is updated, with a bit of organization, nextflow can figure out which subset of the workflow needs to be re-run. We present two models for doing this: one for lightweight code such as plotting/analysis, and one for more substantial code, e.g. a method you are developing.\n\nLightweight code\nInclude the .jl file in the nextflow repo, feed it to the node as input, and use a Julia include() on it. We have already used that pattern for the plot node in an earlier example.\nIf you have several Julia files, put them in a directory, and use the syntax:\n\ninclude { activate; } from \"../pkg.nf\"\n\ndef julia_files = file(moduleDir/\"julia/*.jl\")\n\nworkflow {\n run_julia(julia_files)\n}\n\nprocess run_julia {\n debug true\n input:\n file julia_files\n \"\"\"\n ${activate()}\n include(\"a.jl\")\n include(\"b.jl\")\n \"\"\"\n}\n\n\ncd experiment_repo\n./nextflow run nf-nest/examples/includes.nf \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/includes.nf` [shrivelled_engelbart] DSL2 - revision: f263a35a46\n[9a/1a9b97] Cached process > run_julia\nhello\nworld\n\n\n\n\n\n\n\n\nWarning\n\n\n\nPassing a directory as input (rather than a collection of files as done above), is not ideal in this context. This is because nextflow does not currently recurse inside the directory to compute the checksum used to determine if the cache can be used. Recall that in unix, the change date of a directory only changes when a file is deleted or added under it, and not when a file under it is edited!\n\n\n\n\nLibrary\nIf the code you include is more complex, and/or might be used outside of the context of one nextflow script, it is better to package it.\nIn Julia, creating a package is very simple and it can be published right away and for free on github: see this tutorial.\nFor example, we wrote a small Julia package, CombineCSV.jl to perform the CSV combination in this earlier section.\nTo add or update, you can use a script of that form:\nENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0\nusing Pkg \nPkg.activate(\"julia_env\")\nPkg.add(url = \"https://github.com/UBC-Stat-ML/CombineCSVs\")\nwhere you would replace the URL by the git repo you are using. Note that add will also update to the head of the main branch.\nThis updates the “Manifest.toml” file which in turns signal our “pkg.jl” instantiate utility that it needs to be reran by nextflow:\n\nparams.nPrecompileThreads = 10\n\nparams.julia_env = 'julia_env'\njulia_env = file(params.julia_env)\njulia_env.mkdir()\n\n// Can be used as standalone, but typically used inside a user nf file\nworkflow {\n instantiate(julia_env) | precompile\n}\n\ndef instantiate(julia_env) { instantiate_process(julia_env, file(julia_env/\"Manifest.toml\"))}\n\nprocess instantiate_process {\n executor 'local' // we need internet access\n scratch false // we want changes in Manifest.toml to be saved\n input: \n path julia_env\n path toml // needed for correct cache behaviour under updates\n output:\n path julia_env\n\n \"\"\"\n ${activate(julia_env)}\n\n ENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0\n using Pkg\n Pkg.instantiate()\n \"\"\"\n}\n\nprocess precompile {\n input:\n path julia_env\n output:\n path julia_env\n cpus params.nPrecompileThreads \n memory 15.GB\n \"\"\"\n ${activate(julia_env, params.nPrecompileThreads)}\n\n using Pkg \n Pkg.offline(true) \n Pkg.precompile()\n \"\"\"\n}\n\n// Start Julia and with the provided environment and optionally, number of threads (1 by default) \n// Needs to be the very first line of the process script\ndef activate(julia_env, nThreads = 1) {\n return \"#!/usr/bin/env julia --threads=${nThreads} --project=$julia_env\"\n}\n\ndef activate() {\n return \"#!/usr/bin/env julia --threads=1\"\n}", + "text": "Updating code\nNumerical experiment are often based on code you are developing along the way. When the code is updated, with a bit of organization, nextflow can figure out which subset of the workflow needs to be re-run. We present two models for doing this: one for lightweight code such as plotting/analysis, and one for more substantial code, e.g. a method you are developing.\n\nLightweight code\nInclude the .jl file in the nextflow repo, feed it to the node as input, and use a Julia include() on it. We have already used that pattern for the plot node in an earlier example.\nIf you have several Julia files, put them in a directory, and use the syntax:\n\ninclude { activate; } from \"../pkg.nf\"\n\ndef julia_files = file(moduleDir/\"julia/*.jl\")\n\nworkflow {\n run_julia(julia_files)\n}\n\nprocess run_julia {\n debug true\n input:\n file julia_files\n \"\"\"\n ${activate()}\n include(\"a.jl\")\n include(\"b.jl\")\n \"\"\"\n}\n\n\ncd experiment_repo\n./nextflow run nf-nest/examples/includes.nf \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/includes.nf` [mad_solvay] DSL2 - revision: f263a35a46\n[61/96fce3] Submitted process > run_julia\nhello\nworld\n\n\n\n\n\n\n\n\nWarning\n\n\n\nPassing a directory as input (rather than a collection of files as done above), is not ideal in this context. This is because nextflow does not currently recurse inside the directory to compute the checksum used to determine if the cache can be used. Recall that in unix, the change date of a directory only changes when a file is deleted or added under it, and not when a file under it is edited!\n\n\n\n\nLibrary\nIf the code you include is more complex, and/or might be used outside of the context of one nextflow script, it is better to package it.\nIn Julia, creating a package is very simple and it can be published right away and for free on github: see this tutorial.\nFor example, we wrote a small Julia package, CombineCSV.jl to perform the CSV combination in this earlier section.\nTo add or update, you can use a script of that form:\nENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0\nusing Pkg \nPkg.activate(\"julia_env\")\nPkg.add(url = \"https://github.com/UBC-Stat-ML/CombineCSVs\")\nwhere you would replace the URL by the git repo you are using. Note that add will also update to the head of the main branch.\nThis updates the “Manifest.toml” file which in turns signal our “pkg.jl” instantiate utility that it needs to be reran by nextflow:\n\nparams.nPrecompileThreads = 10\n\nparams.julia_env = 'julia_env'\njulia_env = file(params.julia_env)\njulia_env.mkdir()\n\n// Can be used as standalone, but typically used inside a user nf file\nworkflow {\n instantiate(julia_env) | precompile\n}\n\ndef instantiate(julia_env) { instantiate_process(julia_env, file(julia_env/\"Manifest.toml\"))}\n\nprocess instantiate_process {\n executor 'local' // we need internet access\n scratch false // we want changes in Manifest.toml to be saved\n input: \n path julia_env\n path toml // needed for correct cache behaviour under updates\n output:\n path julia_env\n\n \"\"\"\n ${activate(julia_env)}\n\n ENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0\n using Pkg\n Pkg.instantiate()\n \"\"\"\n}\n\nprocess precompile {\n input:\n path julia_env\n output:\n path julia_env\n cpus params.nPrecompileThreads \n memory 15.GB\n \"\"\"\n ${activate(julia_env, params.nPrecompileThreads)}\n\n using Pkg \n Pkg.offline(true) \n Pkg.precompile()\n \"\"\"\n}\n\n// Start Julia and with the provided environment and optionally, number of threads (1 by default) \n// Needs to be the very first line of the process script\ndef activate(julia_env, nThreads = 1) {\n return \"#!/usr/bin/env julia --threads=${nThreads} --project=$julia_env\"\n}\n\ndef activate() {\n return \"#!/usr/bin/env julia --threads=1\"\n}", "crumbs": [ "Misc" ] @@ -84,7 +84,7 @@ "href": "08_containers.html#using-an-existing-container", "title": "Containers", "section": "Using an existing container", - "text": "Using an existing container\nIn the file nextflow.config provided in nf-nest we provide an example of container documentation:\n\nhead -n 28 experiment_repo/nextflow.config\n\nprofiles {\n standard { \n docker.enabled = true\n process {\n withLabel:containerized {\n container = 'alexandrebouchardcote/default:0.1.6'\n }\n }\n }\n cluster {\n apptainer.enabled = true\n process {\n scratch = true\n executor = 'slurm'\n cpus = 1\n memory = 4.GB\n time = '2h'\n clusterOptions = \"--nodes=1 --account $ALLOCATION_CODE\"\n withLabel:containerized {\n container = 'alexandrebouchardcote/default:0.1.6'\n module = 'apptainer'\n }\n withLabel:gpu {\n clusterOptions = \"--nodes=1 --account ${ALLOCATION_CODE}-gpu --gpus 1\"\n }\n }\n }\n}\n\n\nThere are two profiles, the default one using Docker, and the cluster one, using apptainer. We define a label for this container, which we call here containerized.\nTo instruct one of the process in the workflow to use that container, add the directive label 'containerized'.\nHere is an example of a process to compile and run a Stan program, where Stan is part of the above container:\n\nprocess compile_stan {\n label 'containerized' \n\n input:\n path stan_file\n output:\n path \"${stan_file.baseName}\"\n \n \"\"\"\n # name of stan file without extension\n base_name=`basename $stan_file .stan`\n\n # need to run Stan's make file from the CMDSTAN dir\n CUR=`pwd`\n cd \\$CMDSTAN\n make \\$CUR/\\$base_name\n \"\"\"\n}\n\nprocess run_stan {\n label 'containerized' \n\n input:\n path stan_exec \n path data\n\n \"\"\"\n ./$stan_exec sample \\\n data file=$data \\\n output file=samples.csv \n\n # Compute ESS from Stan's stansummary utility\n \\$CMDSTAN/bin/stansummary samples.csv --csv_filename ess.csv\n \"\"\"\n}\n\nHere is an example workflow using these containerized processes:\n\ninclude { compile_stan; run_stan; } from \"./stan.nf\"\n\ndef stanModel = file(\"https://raw.githubusercontent.com/Julia-Tempering/Pigeons.jl/refs/heads/main/examples/stan/bernoulli.stan\")\ndef data = file(\"https://raw.githubusercontent.com/Julia-Tempering/Pigeons.jl/refs/heads/main/examples/stan/bernoulli.data.json\")\n\nworkflow {\n compiled = compile_stan(stanModel)\n run_stan(compiled, data)\n}\n\nTo run it:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/stan_example.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/stan_example.nf` [kickass_northcutt] DSL2 - revision: 8cef9f29d6\nWARN: Apptainer cache directory has not been defined -- Remote image will be stored in the path: /arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work/singularity -- Use the environment variable NXF_APPTAINER_CACHEDIR to specify a different location\nPulling Apptainer image docker://alexandrebouchardcote/default:0.1.6 [cache /arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work/singularity/alexandrebouchardcote-default-0.1.6.img]\n[0b/5344f6] Submitted process > compile_stan\n[9b/4fd335] Submitted process > run_stan", + "text": "Using an existing container\nIn the file nextflow.config provided in nf-nest we provide an example of container documentation:\n\nhead -n 28 experiment_repo/nextflow.config\n\nprofiles {\n standard { \n docker.enabled = true\n process {\n withLabel:containerized {\n container = 'alexandrebouchardcote/default:0.1.6'\n }\n }\n }\n cluster {\n apptainer.enabled = true\n process {\n scratch = true\n executor = 'slurm'\n cpus = 1\n memory = 4.GB\n time = '2h'\n clusterOptions = \"--nodes=1 --account $ALLOCATION_CODE\"\n withLabel:containerized {\n container = 'alexandrebouchardcote/default:0.1.6'\n module = 'apptainer'\n }\n withLabel:gpu {\n clusterOptions = \"--nodes=1 --account ${ALLOCATION_CODE}-gpu --gpus 1\"\n }\n }\n }\n}\n\n\nThere are two profiles, the default one using Docker, and the cluster one, using apptainer. We define a label for this container, which we call here containerized.\nTo instruct one of the process in the workflow to use that container, add the directive label 'containerized'.\nHere is an example of a process to compile and run a Stan program, where Stan is part of the above container:\n\nprocess compile_stan {\n label 'containerized' \n\n input:\n path stan_file\n output:\n path \"${stan_file.baseName}\"\n \n \"\"\"\n # name of stan file without extension\n base_name=`basename $stan_file .stan`\n\n # need to run Stan's make file from the CMDSTAN dir\n CUR=`pwd`\n cd \\$CMDSTAN\n make \\$CUR/\\$base_name\n \"\"\"\n}\n\nprocess run_stan {\n label 'containerized' \n\n input:\n path stan_exec \n path data\n\n \"\"\"\n ./$stan_exec sample \\\n data file=$data \\\n output file=samples.csv \n\n # Compute ESS from Stan's stansummary utility\n \\$CMDSTAN/bin/stansummary samples.csv --csv_filename ess.csv\n \"\"\"\n}\n\nHere is an example workflow using these containerized processes:\n\ninclude { compile_stan; run_stan; } from \"./stan.nf\"\n\ndef stanModel = file(\"https://raw.githubusercontent.com/Julia-Tempering/Pigeons.jl/refs/heads/main/examples/stan/bernoulli.stan\")\ndef data = file(\"https://raw.githubusercontent.com/Julia-Tempering/Pigeons.jl/refs/heads/main/examples/stan/bernoulli.data.json\")\n\nworkflow {\n compiled = compile_stan(stanModel)\n run_stan(compiled, data)\n}\n\nTo run it:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/stan_example.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/stan_example.nf` [festering_khorana] DSL2 - revision: 8cef9f29d6\nWARN: Apptainer cache directory has not been defined -- Remote image will be stored in the path: /arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work/singularity -- Use the environment variable NXF_APPTAINER_CACHEDIR to specify a different location\nPulling Apptainer image docker://alexandrebouchardcote/default:0.1.6 [cache /arc/burst/st-alexbou-1/abc/nf-nest-doc/experiment_repo/work/singularity/alexandrebouchardcote-default-0.1.6.img]\n[0d/7c12aa] Submitted process > compile_stan\n[04/d00cdc] Submitted process > run_stan", "crumbs": [ "Containers" ] @@ -144,7 +144,7 @@ "href": "05_pkg.html#precompilation-on-hpc", "title": "Managing Julia packages on HPC", "section": "Precompilation on HPC", - "text": "Precompilation on HPC\nOnce we have downloaded the packages in the login node, we now turn to the task of performing pre-compilation.\n\ncd experiment_repo \n./nextflow run nf-nest/pkg.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/pkg.nf` [stupefied_goldstine] DSL2 - revision: fc0374e695\n[d2/2d9cfa] Submitted process > instantiate_process\n[08/cabe06] Submitted process > precompile\n\n\nOptionally, you can append an argument to specify the number of threads to request and use during pre-compilation: e.g. add --nPrecompileThreads 20 to request 20 threads instead of the default of 10.", + "text": "Precompilation on HPC\nOnce we have downloaded the packages in the login node, we now turn to the task of performing pre-compilation.\n\ncd experiment_repo \n./nextflow run nf-nest/pkg.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/pkg.nf` [golden_shaw] DSL2 - revision: fc0374e695\n[9b/6b6098] Submitted process > instantiate_process\n[bf/bbd094] Submitted process > precompile\n\n\nOptionally, you can append an argument to specify the number of threads to request and use during pre-compilation: e.g. add --nPrecompileThreads 20 to request 20 threads instead of the default of 10.", "crumbs": [ "Add Julia packages on HPC" ] @@ -254,7 +254,7 @@ "href": "06_cross_product.html#running-the-script", "title": "Job cross products", "section": "Running the script", - "text": "Running the script\nRunning it with the -profile cluster option will:\n\nbuild a cross-product from variables\nfor each one, automatically create submission scripts\nrun these Julia processes and show the standard out.\n\nFrom the command line, running the script is done as follows:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/many_jobs.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nWARN: It appears you have never run this project before -- Option `-resume` is ignored\nLaunching `nf-nest/examples/many_jobs.nf` [nauseous_volta] DSL2 - revision: aa082b1978\n[cf/316661] Submitted process > run_julia (7)\n[98/3f89f3] Submitted process > run_julia (16)\n[f5/9200b5] Submitted process > run_julia (8)\n[60/79d084] Submitted process > run_julia (10)\n[25/304afb] Submitted process > run_julia (12)\n[dc/f8a7cb] Submitted process > run_julia (1)\n[23/a921d6] Submitted process > run_julia (6)\n[17/60efee] Submitted process > run_julia (3)\n[fe/d3775a] Submitted process > run_julia (2)\n[23/02e5a5] Submitted process > run_julia (11)\n[61/92282e] Submitted process > run_julia (9)\n[d2/4f9929] Submitted process > run_julia (13)\n[c5/278d1e] Submitted process > run_julia (5)\n[ec/ca8573] Submitted process > run_julia (15)\n[b9/99bc51] Submitted process > run_julia (4)\n[a7/6c5a96] Submitted process > run_julia (14)\n[ff/2ee016] Submitted process > run_julia (17)\n[ba/85000e] Submitted process > run_julia (18)\n2 + 1 = 3\n2 * 1 = 2\n3 * 2 = 6\n2 * 2 = 4\n2 * 3 = 6\n1 + 1 = 2\n1 * 3 = 3\n1 + 2 = 3\n1 * 1 = 1\n2 + 3 = 5\n2 + 2 = 4\n3 + 1 = 4\n1 + 3 = 4\n3 + 2 = 5\n1 * 2 = 2\n3 * 1 = 3\n3 + 3 = 6\n3 * 3 = 9", + "text": "Running the script\nRunning it with the -profile cluster option will:\n\nbuild a cross-product from variables\nfor each one, automatically create submission scripts\nrun these Julia processes and show the standard out.\n\nFrom the command line, running the script is done as follows:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/many_jobs.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nWARN: It appears you have never run this project before -- Option `-resume` is ignored\nLaunching `nf-nest/examples/many_jobs.nf` [pensive_cuvier] DSL2 - revision: aa082b1978\n[9a/f95c38] Submitted process > run_julia (16)\n[31/c84da1] Submitted process > run_julia (8)\n[70/882e38] Submitted process > run_julia (7)\n[f8/827180] Submitted process > run_julia (9)\n[40/502eb0] Submitted process > run_julia (5)\n[24/235871] Submitted process > run_julia (13)\n[08/c3250a] Submitted process > run_julia (3)\n[2a/d590b5] Submitted process > run_julia (11)\n[1a/18653d] Submitted process > run_julia (6)\n[ba/2ffe85] Submitted process > run_julia (2)\n[4f/48bab2] Submitted process > run_julia (4)\n[55/fb4a96] Submitted process > run_julia (14)\n[6b/ba0d04] Submitted process > run_julia (15)\n[33/3c91f1] Submitted process > run_julia (1)\n[67/d912ab] Submitted process > run_julia (12)\n[42/edc64d] Submitted process > run_julia (10)\n[d0/25bfed] Submitted process > run_julia (18)\n[39/75210b] Submitted process > run_julia (17)\n3 * 2 = 6\n2 + 2 = 4\n2 + 1 = 3\n2 * 1 = 2\n1 + 3 = 4\n3 + 1 = 4\n1 + 2 = 3\n2 + 3 = 5\n1 * 3 = 3\n1 * 1 = 1\n1 * 2 = 2\n3 * 1 = 3\n3 + 2 = 5\n1 + 1 = 2\n2 * 3 = 6\n2 * 2 = 4\n3 * 3 = 9\n3 + 3 = 6", "crumbs": [ "Job cross products" ] @@ -494,7 +494,7 @@ "href": "04_launch.html#local-execution", "title": "Launching an experiment", "section": "Local execution", - "text": "Local execution\nUse the following command to run the nextflow script locally:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/hello.nf\n\nN E X T F L O W ~ version 24.10.0\nWARN: It appears you have never run this project before -- Option `-resume` is ignored\nLaunching `nf-nest/examples/hello.nf` [nostalgic_carson] DSL2 - revision: 9d1a692a7e\n[96/869cf0] Submitted process > hello\nHello world!", + "text": "Local execution\nUse the following command to run the nextflow script locally:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/hello.nf\n\nN E X T F L O W ~ version 24.10.0\nWARN: It appears you have never run this project before -- Option `-resume` is ignored\nLaunching `nf-nest/examples/hello.nf` [grave_allen] DSL2 - revision: 9d1a692a7e\n[b5/8ac5f1] Submitted process > hello\nHello world!", "crumbs": [ "Launch an experiment" ] @@ -504,7 +504,7 @@ "href": "04_launch.html#cluster-execution", "title": "Launching an experiment", "section": "Cluster execution", - "text": "Cluster execution\nTo run on a cluster, add the argument -profile cluster which instructs nextflow to use the configs in section cluster { ... } of the file nextflow.config created in the setup instructions.1\nHere is a minimal example:\n\n./nextflow run nf-nest/examples/hello.nf -profile cluster \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/hello.nf` [evil_ardinghelli] DSL2 - revision: 9d1a692a7e\n[96/869cf0] Cached process > hello\nHello world!", + "text": "Cluster execution\nTo run on a cluster, add the argument -profile cluster which instructs nextflow to use the configs in section cluster { ... } of the file nextflow.config created in the setup instructions.1\nHere is a minimal example:\n\n./nextflow run nf-nest/examples/hello.nf -profile cluster \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/hello.nf` [romantic_solvay] DSL2 - revision: 9d1a692a7e\n[b5/8ac5f1] Cached process > hello\nHello world!", "crumbs": [ "Launch an experiment" ] @@ -584,7 +584,7 @@ "href": "07_combine.html#running-the-nextflow-script", "title": "Combine outputs", "section": "Running the nextflow script", - "text": "Running the nextflow script\n\ncd experiment_repo\n./nextflow run nf-nest/examples/full.nf -profile cluster \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/full.nf` [agitated_knuth] DSL2 - revision: a68c131baa\n[6d/3357a5] Submitted process > combine_workflow:instantiate_process\n[25/d08519] Submitted process > instantiate_process\n[bb/03d46d] Submitted process > combine_workflow:precompile\n[bb/605315] Submitted process > precompile\n[fa/aa7751] Submitted process > run_julia (9)\n[7b/150dd2] Submitted process > run_julia (8)\n[cd/7a1c39] Submitted process > run_julia (2)\n[df/ffce39] Submitted process > run_julia (13)\n[c8/23f9a2] Submitted process > run_julia (12)\n[b6/ed34c6] Submitted process > run_julia (4)\n[7f/32c560] Submitted process > run_julia (5)\n[23/dca8ec] Submitted process > run_julia (15)\n[3e/bd5153] Submitted process > run_julia (14)\n[8f/1b5deb] Submitted process > run_julia (10)\n[57/f522b3] Submitted process > run_julia (6)\n[60/eb9c28] Submitted process > run_julia (11)\n[9c/ce18ab] Submitted process > run_julia (1)\n[13/599d3d] Submitted process > run_julia (7)\n[cb/d8f2f4] Submitted process > run_julia (3)\n[e0/fad42d] Submitted process > run_julia (16)\n[34/e101c3] Submitted process > run_julia (17)\n[3b/a775ca] Submitted process > run_julia (18)\n[90/4f63ff] Submitted process > run_julia (20)\n[65/b68f71] Submitted process > run_julia (19)\n[4a/d5b369] Submitted process > combine_workflow:combine_process\n[21/1bfa99] Submitted process > plot", + "text": "Running the nextflow script\n\ncd experiment_repo\n./nextflow run nf-nest/examples/full.nf -profile cluster \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/full.nf` [intergalactic_jepsen] DSL2 - revision: a68c131baa\n[9c/0659ed] Submitted process > combine_workflow:instantiate_process\n[71/d2fadf] Submitted process > instantiate_process\n[ce/f1a696] Submitted process > combine_workflow:precompile\n[4d/72b110] Submitted process > precompile\n[42/0732cc] Submitted process > run_julia (9)\n[5e/81171f] Submitted process > run_julia (1)\n[c5/1ccbfe] Submitted process > run_julia (3)\n[d4/228929] Submitted process > run_julia (4)\n[2b/2baf59] Submitted process > run_julia (14)\n[8c/2bb063] Submitted process > run_julia (15)\n[d5/5001bb] Submitted process > run_julia (13)\n[56/de7ec7] Submitted process > run_julia (7)\n[ab/d18893] Submitted process > run_julia (11)\n[76/fc81a9] Submitted process > run_julia (5)\n[69/6f13e4] Submitted process > run_julia (10)\n[28/fa731e] Submitted process > run_julia (2)\n[61/f7b16b] Submitted process > run_julia (12)\n[8e/3df7d5] Submitted process > run_julia (16)\n[d3/fe1273] Submitted process > run_julia (6)\n[56/f5fc6a] Submitted process > run_julia (8)\n[ee/683820] Submitted process > run_julia (17)\n[58/7fa150] Submitted process > run_julia (18)\n[28/2a8382] Submitted process > run_julia (19)\n[c2/f5b56b] Submitted process > run_julia (20)\n[52/031bf9] Submitted process > combine_workflow:combine_process\n[c0/9912ef] Submitted process > plot", "crumbs": [ "Combine outputs" ] @@ -594,7 +594,7 @@ "href": "07_combine.html#accessing-the-output", "title": "Combine outputs", "section": "Accessing the output", - "text": "Accessing the output\nEach nextflow process is associated with a unique work directory to ensure the processes do not interfere with each other. Here we cover two ways to quickly access these work directories.\n\nQuick inspection\nA quick way to find the output of a nextflow process that we just ran is to use:\ncd experiment_repo \nnf-nest/nf-open\nThis lists the work folders for the last nextflow job.\n\n\nOrganizing the output with a publishDir\nA better approach is to use the publishDir directive, combined with nf-nest’s deliverables() utility, as illustrated in the run_julia process above. This will automatically copy the output of the process associated with the directive in a sub-directory of experiment_repo/deliverables.\n\ntree experiment_repo/deliverables\n\nexperiment_repo/deliverables\n└── scriptName=full.nf___runName=agitated_knuth\n ├── output\n │   ├── summary.csv\n │   └── swap_prs.csv\n └── plot.png\n\n2 directories, 3 files\n\n\nHere the runName can be used with nextflow’s log command to obtain more information on the run.\n\ncd experiment_repo\n./nextflow log\n\nTIMESTAMP DURATION RUN NAME STATUS REVISION ID SESSION ID COMMAND \n2024-11-12 07:05:30 3.2s nostalgic_carson OK 9d1a692a7e 317a6835-37ee-4f54-9d9f-b0df69f57bbd nextflow run nf-nest/examples/hello.nf \n2024-11-12 07:05:36 7.6s evil_ardinghelli OK 9d1a692a7e 317a6835-37ee-4f54-9d9f-b0df69f57bbd nextflow run nf-nest/examples/hello.nf -profile cluster\n2024-11-12 07:06:32 1m 22s stupefied_goldstine OK fc0374e695 317a6835-37ee-4f54-9d9f-b0df69f57bbd nextflow run nf-nest/pkg.nf -profile cluster \n2024-11-12 07:08:17 4m 33s agitated_knuth OK a68c131baa 317a6835-37ee-4f54-9d9f-b0df69f57bbd nextflow run nf-nest/examples/full.nf -profile cluster", + "text": "Accessing the output\nEach nextflow process is associated with a unique work directory to ensure the processes do not interfere with each other. Here we cover two ways to quickly access these work directories.\n\nQuick inspection\nA quick way to find the output of a nextflow process that we just ran is to use:\ncd experiment_repo \nnf-nest/nf-open\nThis lists the work folders for the last nextflow job.\n\n\nOrganizing the output with a publishDir\nA better approach is to use the publishDir directive, combined with nf-nest’s deliverables() utility, as illustrated in the run_julia process above. This will automatically copy the output of the process associated with the directive in a sub-directory of experiment_repo/deliverables.\n\ntree experiment_repo/deliverables\n\nexperiment_repo/deliverables\n└── scriptName=full.nf___runName=intergalactic_jepsen\n ├── output\n │   ├── summary.csv\n │   └── swap_prs.csv\n └── plot.png\n\n2 directories, 3 files\n\n\nHere the runName can be used with nextflow’s log command to obtain more information on the run.\n\ncd experiment_repo\n./nextflow log\n\nTIMESTAMP DURATION RUN NAME STATUS REVISION ID SESSION ID COMMAND \n2024-11-12 09:04:09 3.5s grave_allen OK 9d1a692a7e bf02b0fb-fa7d-4171-88a3-e89239ee4248 nextflow run nf-nest/examples/hello.nf \n2024-11-12 09:04:16 6.9s romantic_solvay OK 9d1a692a7e bf02b0fb-fa7d-4171-88a3-e89239ee4248 nextflow run nf-nest/examples/hello.nf -profile cluster\n2024-11-12 09:05:11 1m 13s golden_shaw OK fc0374e695 bf02b0fb-fa7d-4171-88a3-e89239ee4248 nextflow run nf-nest/pkg.nf -profile cluster \n2024-11-12 09:06:48 3m 8s intergalactic_jepsen OK a68c131baa bf02b0fb-fa7d-4171-88a3-e89239ee4248 nextflow run nf-nest/examples/full.nf -profile cluster", "crumbs": [ "Combine outputs" ] @@ -604,7 +604,7 @@ "href": "09_gpu.html", "title": "GPU programming", "section": "", - "text": "In a context such as Sockeye where it is not possible to access GPU nodes with internet access, precompilation becomes more complicated than the earlier page on non-CUDA precompilation.\nWe provide a workaround, pkg_gpu.nf, which offers the same functionality as pkg.nf but is slower since all precompilation has to occur on the login node.\nFirst, add the package as before:\n\nENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0 # Hold off precompile since we are in login node\nusing Pkg \nPkg.activate(\"experiment_repo/julia_env\")\nPkg.add(\"CUDA\")\n\nNext, use the GPU precompilation script:\n\ncd experiment_repo \n./nextflow run nf-nest/pkg_gpu.nf \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/pkg_gpu.nf` [insane_monod] DSL2 - revision: 713b74ac4a\n[0a/d32c39] Submitted process > instantiate_process\n[9a/f339af] Submitted process > precompile_gpu", + "text": "In a context such as Sockeye where it is not possible to access GPU nodes with internet access, precompilation becomes more complicated than the earlier page on non-CUDA precompilation.\nWe provide a workaround, pkg_gpu.nf, which offers the same functionality as pkg.nf but is slower since all precompilation has to occur on the login node.\nFirst, add the package as before:\n\nENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0 # Hold off precompile since we are in login node\nusing Pkg \nPkg.activate(\"experiment_repo/julia_env\")\nPkg.add(\"CUDA\")\n\nNext, use the GPU precompilation script:\n\ncd experiment_repo \n./nextflow run nf-nest/pkg_gpu.nf \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/pkg_gpu.nf` [hopeful_turing] DSL2 - revision: 713b74ac4a\n[b6/0ba7e0] Submitted process > instantiate_process\n[70/2ce6ba] Submitted process > precompile_gpu", "crumbs": [ "GPU programming" ] @@ -614,7 +614,7 @@ "href": "09_gpu.html#installing-cuda.jl", "title": "GPU programming", "section": "", - "text": "In a context such as Sockeye where it is not possible to access GPU nodes with internet access, precompilation becomes more complicated than the earlier page on non-CUDA precompilation.\nWe provide a workaround, pkg_gpu.nf, which offers the same functionality as pkg.nf but is slower since all precompilation has to occur on the login node.\nFirst, add the package as before:\n\nENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0 # Hold off precompile since we are in login node\nusing Pkg \nPkg.activate(\"experiment_repo/julia_env\")\nPkg.add(\"CUDA\")\n\nNext, use the GPU precompilation script:\n\ncd experiment_repo \n./nextflow run nf-nest/pkg_gpu.nf \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/pkg_gpu.nf` [insane_monod] DSL2 - revision: 713b74ac4a\n[0a/d32c39] Submitted process > instantiate_process\n[9a/f339af] Submitted process > precompile_gpu", + "text": "In a context such as Sockeye where it is not possible to access GPU nodes with internet access, precompilation becomes more complicated than the earlier page on non-CUDA precompilation.\nWe provide a workaround, pkg_gpu.nf, which offers the same functionality as pkg.nf but is slower since all precompilation has to occur on the login node.\nFirst, add the package as before:\n\nENV[\"JULIA_PKG_PRECOMPILE_AUTO\"]=0 # Hold off precompile since we are in login node\nusing Pkg \nPkg.activate(\"experiment_repo/julia_env\")\nPkg.add(\"CUDA\")\n\nNext, use the GPU precompilation script:\n\ncd experiment_repo \n./nextflow run nf-nest/pkg_gpu.nf \n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/pkg_gpu.nf` [hopeful_turing] DSL2 - revision: 713b74ac4a\n[b6/0ba7e0] Submitted process > instantiate_process\n[70/2ce6ba] Submitted process > precompile_gpu", "crumbs": [ "GPU programming" ] @@ -624,7 +624,7 @@ "href": "09_gpu.html#running-nextflow-processes-requiring-gpu", "title": "GPU programming", "section": "Running nextflow processes requiring GPU", - "text": "Running nextflow processes requiring GPU\nAn example of a workflow using GPUs:\n\ninclude { instantiate; precompile_gpu; } from \"../pkg_gpu.nf\"\ninclude { activate; } from \"../pkg.nf\"\n\ndef julia_env = file('julia_env')\n\nworkflow {\n instantiate(julia_env) | precompile_gpu | run_julia\n}\n\nprocess run_julia {\n debug true\n label 'gpu'\n input:\n path julia_env\n \"\"\"\n ${activate(julia_env)}\n\n using CUDA \n\n println(\"CPU\")\n x = rand(5000, 5000);\n @time x * x;\n @time x * x;\n\n println(\"GPU\")\n x = CUDA.rand(5000, 5000);\n @time x * x;\n @time x * x;\n\n CUDA.versioninfo()\n \"\"\"\n}\n\nWe run it using the same command as usual:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/gpu.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/gpu.nf` [sick_legentil] DSL2 - revision: 9be41cea49\n[70/537eb2] Submitted process > instantiate_process\n[d4/f4b844] Submitted process > precompile_gpu\n[05/4b36e9] Submitted process > run_julia\nCPU\n 15.295592 seconds (4.79 M allocations: 517.701 MiB, 4.03% gc time, 18.96% compilation time)\n 12.138829 seconds (2 allocations: 190.735 MiB, 0.06% gc time)\nGPU\n 1.972740 seconds (1.67 M allocations: 107.375 MiB, 95.51% compilation time)\n 0.000784 seconds (50 allocations: 1.172 KiB)\nCUDA runtime 12.5, artifact installation\nCUDA driver 12.6\nNVIDIA driver 550.90.12\n\nCUDA libraries: \n- CUBLAS: 12.5.3\n- CURAND: 10.3.6\n- CUFFT: 11.2.3\n- CUSOLVER: 11.6.3\n- CUSPARSE: 12.5.1\n- CUPTI: 2024.2.1 (API 23.0.0)\n- NVML: 12.0.0+550.90.12\n\nJulia packages: \n- CUDA: 5.5.2\n- CUDA_Driver_jll: 0.10.3+0\n- CUDA_Runtime_jll: 0.15.3+0\n\nToolchain:\n- Julia: 1.10.2\n- LLVM: 15.0.7\n\nPreferences:\n- CUDA_Runtime_jll.version: 12.5\n\n1 device:\n 0: Tesla V100-SXM2-16GB (sm_70, 15.171 GiB / 16.000 GiB available)", + "text": "Running nextflow processes requiring GPU\nAn example of a workflow using GPUs:\n\ninclude { instantiate; precompile_gpu; } from \"../pkg_gpu.nf\"\ninclude { activate; } from \"../pkg.nf\"\n\ndef julia_env = file('julia_env')\n\nworkflow {\n instantiate(julia_env) | precompile_gpu | run_julia\n}\n\nprocess run_julia {\n debug true\n label 'gpu'\n input:\n path julia_env\n \"\"\"\n ${activate(julia_env)}\n\n using CUDA \n\n println(\"CPU\")\n x = rand(5000, 5000);\n @time x * x;\n @time x * x;\n\n println(\"GPU\")\n x = CUDA.rand(5000, 5000);\n @time x * x;\n @time x * x;\n\n CUDA.versioninfo()\n \"\"\"\n}\n\nWe run it using the same command as usual:\n\ncd experiment_repo\n./nextflow run nf-nest/examples/gpu.nf -profile cluster\n\nN E X T F L O W ~ version 24.10.0\nLaunching `nf-nest/examples/gpu.nf` [thirsty_mahavira] DSL2 - revision: 9be41cea49\n[b6/49dd3e] Submitted process > instantiate_process\n[80/583246] Submitted process > precompile_gpu\n[fc/7402e7] Submitted process > run_julia\nCPU\n 13.046295 seconds (4.79 M allocations: 517.701 MiB, 3.56% gc time, 17.49% compilation time)\n 10.582634 seconds (2 allocations: 190.735 MiB, 0.10% gc time)\nGPU\n 1.535394 seconds (1.67 M allocations: 107.375 MiB, 96.04% compilation time)\n 0.000591 seconds (50 allocations: 1.172 KiB)\nCUDA runtime 12.5, artifact installation\nCUDA driver 12.6\nNVIDIA driver 550.90.12\n\nCUDA libraries: \n- CUBLAS: 12.5.3\n- CURAND: 10.3.6\n- CUFFT: 11.2.3\n- CUSOLVER: 11.6.3\n- CUSPARSE: 12.5.1\n- CUPTI: 2024.2.1 (API 23.0.0)\n- NVML: 12.0.0+550.90.12\n\nJulia packages: \n- CUDA: 5.5.2\n- CUDA_Driver_jll: 0.10.3+0\n- CUDA_Runtime_jll: 0.15.3+0\n\nToolchain:\n- Julia: 1.10.2\n- LLVM: 15.0.7\n\nPreferences:\n- CUDA_Runtime_jll.version: 12.5\n\n1 device:\n 0: Tesla V100-SXM2-32GB (sm_70, 31.137 GiB / 32.000 GiB available)", "crumbs": [ "GPU programming" ]