[Paper] add comparison tables and benchmark (#127)

* modified: joss/paper.md Possible alternative to comparing the 3 packages * Update paper.md add dates of package bivaratecopula * Update paper.md modifying tables, Benchmarktools.jl needs to be cited correctly * adding bibliography and code * Update paper.md Add Acknowledgement * Clean up and write a last version --------- Co-authored-by: Oskar Laverny <[email protected]>
lrnv · Feb 10, 2024 · 7cf9beb · 7cf9beb
1 parent 4e7262a
commit 7cf9beb
Show file tree

Hide file tree

Showing 2 changed files with 43 additions and 1 deletion.
diff --git a/joss/paper.bib b/joss/paper.bib
@@ -224,3 +224,17 @@ @software{BivariateCopulas
   url          = {https://doi.org/10.5281/zenodo.10412898}
 }
 
+@Article{BenchmarkTools,
+author = {{Chen}, Jiahao and {Revels}, Jarrett},
+title = "{Robust benchmarking in noisy environments}",
+journal = {arXiv e-prints},
+keywords = {Computer Science - Performance, 68N30, B.8.1, D.2.5},
+year = 2016,
+month = "Aug",
+eid = {arXiv:1608.04295},
+archivePrefix ={arXiv},
+eprint = {1608.04295},
+primaryClass = {cs.PF},
+adsurl = {https://ui.adsabs.harvard.edu/abs/2016arXiv160804295C},
+adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
diff --git a/joss/paper.md b/joss/paper.md
@@ -39,7 +39,31 @@ The Julia package `Copulas.jl` brings most standard copula-related features into
 
 The R package `copula` [@r_copula_citation1; @r_copula_citation2; @r_copula_citation3; @r_copula_citation4] is the gold standard when it comes to sampling, estimating, or simply working around dependence structures. However, in other languages, the available tools are not as developed and/or not as recognized. We bridge the gap in the Julian ecosystem with this Julia-native implementation. Due to the very flexible type system in Julia, our code expressiveness and tidiness will increase its usability and maintainability in the long-run. Type-stability allows sampling in arbitrary precision without requiring more code, and Julia's multiple dispatch yields most of the below-described applications.
 
-There are competing packages in Julia, such as [`BivariateCopulas.jl`](https://github.com/AnderGray/BivariateCopulas.jl) [@BivariateCopulas] which only deals with a few models in bivariate settings but has very nice graphs, or [`DatagenCopulaBased.jl`](https://github.com/iitis/DatagenCopulaBased.jl) [@DatagenCopulaBased_1; @DatagenCopulaBased_2; @DatagenCopulaBased_3; @DatagenCopulaBased_4], which only provides sampling and does not have exactly the same models as `Copulas.jl`. While not fully covering out both of these package's functionality (mostly because the three projects chose different copulas to implement), `Copulas.jl` is clearly the most fully featured, and brings, as a key feature, the compliance with the broader ecosystem.
+There are competing packages in Julia, such as [`BivariateCopulas.jl`](https://github.com/AnderGray/BivariateCopulas.jl) [@BivariateCopulas] which only deals with a few models in bivariate settings but has very nice graphs, or [`DatagenCopulaBased.jl`](https://github.com/iitis/DatagenCopulaBased.jl) [@DatagenCopulaBased_1; @DatagenCopulaBased_2; @DatagenCopulaBased_3; @DatagenCopulaBased_4], which only provides sampling and does not have exactly the same models as `Copulas.jl`. While not fully covering out both of these package's functionality (mostly because the three projects chose different implementation paths), `Copulas.jl` brings, as a key feature, the compliance with the broader ecosystem. The following table provides a feature comparison between the three: 
+
+| Characteristic                                | `Copulas.jl`       | `DatagenCopulaBased.jl`      | `BivariateCopulas.jl`     |
+|-----------------------------------------------|--------------------|------------------------------|---------------------------|
+| `Distributions.jl`'s API | ✔️ | ❌ | ✔️ |
+| Fitting                  | ✔️ | ✔️ | ❌ |
+| Plotting                 | ❌ | ❌ | ✔️ |
+| Available copulas        |     |     |    |
+| - Classic Bivariate      | ✔️ | ✔️ | ✔️ |
+| - Classic Multivariate   | ✔️ | ✔️ | ❌ |
+| - Archimedeans           | ✔️ (All of them) | ⚠️ Selected ones | ⚠️Selected ones |
+| - Obscure Bivariate      | ✔️ | ❌ | ❌ |
+| - Archimedean Chains     | ❌ | ✔️ | ❌ |
+
+Since our primary target is maintainability and readability of the implementation, we did not consider the efficiency and the performance of the code yet. However, a (limited in scope) benchmark on Clayton's pdf shows competitive behavior of our implementation. To perform this test we use the [`BenchmarkTools.jl`](https://github.com/JuliaCI/BenchmarkTools.jl) [@BenchmarkTools] package and generate 10^6 samples for Clayton copulas of dimensions 2, 5, 10 with parameter 0.8:
+
+| Package                           | Dimension | Execution Time (seconds) | Memory Usage (bytes) |
+|-----------------------------------|-----------|--------------------------------------|-------------------------|
+| Copulas.Clayton                   | 2         | 1.1495578e9                          | 408973296               |
+| Copulas.Clayton                   | 5         | 1.3448951e9                          | 386723344               |
+| Copulas.Clayton                   | 10        | 1.8044065e9                          | 464100752               |
+| BivariateCopulas.Clayton          | 2         | 1.331608e8                           | 56000864                |
+| DatagenCopulaBased.Clayton        | 2         | 1.9868345e9                          | 1178800464              |
+| DatagenCopulaBased.Clayton        | 5         | 2.4276321e9                          | 1314855488              |
+| DatagenCopulaBased.Clayton        | 10        | 2.8009263e9                          | 1627164656              |
 
 # Examples
 
@@ -112,4 +136,8 @@ The package is starting to get used in several other places of the ecosystem. Am
 - The package [`GlobalSensitivity.jl`](https://github.com/SciML/GlobalSensitivity.jl) exploits `Copulas.jl` to provide Shapley effects implementation, see [this documentation](https://docs.sciml.ai/GlobalSensitivity/stable/tutorials/shapley/). 
 - [`EconomicScenarioGenerators.jl`](https://github.com/JuliaActuary/EconomicScenarioGenerators.jl) uses `Copulas.jl`'s dependence structures to construct multivariate financial assets. 
 
+# Acknowledgement
+
+Santiago Jiménez Ramos thanks FACEPE for the full financing of his postgraduate studies.
+
 # References