Source Code Availibility for nfcore_external_java_deps.jar

[zhuchcn]

Is your feature request related to a problem? Please describe

It seems like the lib/nfcore_external_java_deps.jar compiles groovy code under lib at nextflow runtime. This is really handy. I searched the documentation but could not find the source code or any description to this jar. Would be great if some detail can be provided, for many reasons such as security.

Describe the solution you'd like

Make the source code available if not already and if possible, and/or add description to documentation.

Additional context

Please let me know if it is already documented somewhere else or if the source code is already publicly available. Appreciated!

[ewels]

Yeah good question - the only docs that I know of are this mention here: (fairly obscure place to find it)

nfcore_external_java_deps.jar - Bundled Groovy dependencies so that pipelines work offline (mostly for JSON schema validation - see imports in NfcoreSchema.groovy)

@KevinMenden compiled this file when we built the JSON schema code. I think it's basically these dependencies:

tools/nf_core/pipeline-template/lib/NfcoreSchema.groovy

Lines 5 to 12 in e0d00d3

	import org.everit.json.schema.Schema
	import org.everit.json.schema.loader.SchemaLoader
	import org.everit.json.schema.ValidationException
	import org.json.JSONObject
	import org.json.JSONTokener
	import org.json.JSONArray
	import groovy.json.JsonSlurper
	import groovy.json.JsonBuilder

tools/nf_core/pipeline-template/lib/NfcoreTemplate.groovy

Line 5 in e0d00d3

import org.yaml.snakeyaml.Yaml

I'm not sure if it's all of them (some may be already available in Nextflow, so it could possibly be just the json.schema stuff), or how the file is compiled. I agree that it would be good to have some description of this somewhere. It's probably discussed somewhere in the vast Slack history if we can find it..

[KevinMenden]

I can't give you the exact list of dependencies right now, but those sound about right. I basically just collected all the dependencies that were necessary and combined their JARs.
Agree that it would be nicer to have this reproducible and know what actually is in there.
I'll try to have a look this weekend! Might even have written something down about this in my notes, who knows :)

[sralchemab]

Hi @KevinMenden,

I believe I found a bug related to the parameters validation against the schema (org.everit.json.schema.loader.SchemaLoader usage). I reported it here but they asked me about the version I'm using, and to be honest, I don't know how to get that info, and I believe the package is compiled in this file.

Do you mind giving me a hand with this?

Thanks in advance!

[sralchemab]

Hi @KevinMenden, I wonder if the package I'm talking about is used from your file or from here:

https://github.com/nextflow-io/nf-validation/blob/37ea1b571977d8b8e26a7ff3636a115a3fda3498/plugins/nf-validation/build.gradle#L63

[KevinMenden]

Hi @sralchemab,
I will try to have a look today and come back to you this evening, so for the delay!

[apeltzer]

@KevinMenden Let me know once you found things - I might actually setup a separate directory for the code to generate the JAR file with some kind of release functionality that automates the process a bit more nicely. That would also mean we can pull in updated / bugfixed versions of upstream libraries if necessary :-)

[sralchemab]

Hi @KevinMenden, have you been able to look into this?

[KevinMenden]

Sorry - didn't geht around to check this week, but will do today :) will let you know

[KevinMenden]

Alright, so unfortunately I don't have any notes about the exact versions used for putting together the JAR file anymore - definitely a bit of a mess up on my side. This should be kept somehow ...
I'm not much of a Java-expert, so I'm not sure if it's possible to get the actual version back out of there.
I did extract the JAR file though and look in the .class files using the javap command, and all of the files I looked at were last modified on 15.12.2018, which happens to be the release date of version 1.10.0 (https://github.com/everit-org/json-schema/releases), so I'm guessing that's the version we have. Sorry for taking so long @sralchemab !

@apeltzer sounds like a good idea! I guess it makes sense to make this whole exercise again and create the JAR file from scratch - and make it reproducible, not like I did 🤦
Do you need any help with the dependencies that were used? It's possible to get them from the JAR file. But there should also be a list in the issue where I added this I would guess 🤔

[sralchemab]

Thanks, Kevin. Could you tell me in which circumstances this JAR file is used? Is it only when running nf-core tools commands? Or is it used as well during the execution of a pipeline?

[KevinMenden]

I have been out of the project for a bit so I can't tell you exactly where it used now.
But the schema validation library is used to check whether the input parameters for the pipeline match the schema definition, if I recall correctly. So yes it is used during pipeline execution, albeit only in the beginning (as far as I know).

[mirpedrol]

As the parameter validation code has been moved to the nf-validation plugin, dependencies are specified there, and we don't use the .jar file any more. Thus, I am closing this issue.