Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add new docs structure #816

Merged
merged 57 commits into from
Jan 7, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
f458b9f
docs: add new docs structure
marikaris Nov 14, 2024
72b1e65
chore: remove unnecessary import
marikaris Nov 15, 2024
2892ad7
style: fix styling
marikaris Nov 15, 2024
f415a17
style: update styling
marikaris Nov 15, 2024
18c1691
chore: change order in menu
marikaris Nov 15, 2024
31699aa
chore: allow for usage of icons
marikaris Nov 15, 2024
756afba
style: update style
marikaris Nov 15, 2024
c7fa8cb
style: use material style to allow for flexibility and usage of icons
marikaris Nov 15, 2024
2e5b158
style: make top menu molgenis blue
marikaris Nov 15, 2024
c1ae2df
docs(quick-start): make start with initial docs
marikaris Nov 15, 2024
63cebbd
style: slightly adjust styling
marikaris Nov 18, 2024
2f03a31
update basic usage role researcher and datamanager
erikzwart Nov 18, 2024
76349b5
update mkdocs.yml added content tabs and unordered lists config
erikzwart Nov 18, 2024
f527dc5
style: fix right menu active styling
marikaris Nov 18, 2024
a6524ab
docs(quick start): write docs
marikaris Nov 18, 2024
b6ea10e
docs: make start with basic concepts
marikaris Nov 19, 2024
d14dfab
chore: add plugins for image formatting to config
marikaris Nov 19, 2024
9b7a1a8
docs(quick_start): add captions to images
marikaris Nov 19, 2024
82a9520
chore: add syntax highlighting and copy button in code blocks
marikaris Nov 19, 2024
49a0804
docs(basic concepts): add images
marikaris Nov 19, 2024
85a09a0
Update mkdocs yaml
erikzwart Nov 20, 2024
dfcb36d
Update basic usage, include Armadillo UI readme
erikzwart Nov 20, 2024
d731b4f
style: add option for simple css styling
marikaris Nov 21, 2024
5b01871
update mkdocs.yml (add keyboard keys)
erikzwart Nov 21, 2024
8c3bca2
docs(install_management.md): install/update/migrate
erikzwart Nov 21, 2024
0d13d69
docs(dev_guides): java/docker/apache
erikzwart Nov 21, 2024
08d49ae
docs(basic concepts): add docs
marikaris Nov 21, 2024
81006e4
docs: create homepage for docs
marikaris Nov 21, 2024
9f496bc
docs: fix image and margins
marikaris Nov 21, 2024
87fe236
docs(license): change link to clickable link
marikaris Nov 21, 2024
154ec72
docs(index): fix scrambly layout on small screen
marikaris Nov 22, 2024
11cf2b7
docs: slightly tweaked the language
timcadman Nov 25, 2024
7e238c9
docs: tweaked english and added links
timcadman Nov 25, 2024
65f458f
docs: added links, editted english and R scripts
timcadman Nov 25, 2024
745eca1
docs: move apache setup to install management docs
marikaris Nov 26, 2024
9e6105e
style: make font-size in tabs slightly bigger
marikaris Nov 26, 2024
77b854d
docs(dev_guides): add devguides
marikaris Nov 26, 2024
d983c56
Reorder basic usage and update
erikzwart Nov 27, 2024
f4d503c
Reorder install setup and management and update
erikzwart Nov 27, 2024
a370b80
fix docker compose url (dev_guides)
erikzwart Nov 28, 2024
5cf9f2e
fix install guide url (quick_start)
erikzwart Nov 28, 2024
89fc5bd
docs(faq): write docs
erikzwart Nov 28, 2024
d3a6d46
docs(dev_guides): add in last missing information from contributing.md
marikaris Nov 28, 2024
1fb0694
style: add in light and dark theme
marikaris Nov 28, 2024
c3a72ec
docs(index): restructure index
marikaris Nov 28, 2024
7289eea
chore: change docs command in ui to mkdocs
marikaris Nov 28, 2024
10f5e6b
update faq
erikzwart Nov 28, 2024
10a8f1a
docs(about): there's something there, needs improvement
marikaris Nov 29, 2024
4520037
docs(troubleshooting): comment out as we don't have text yet
marikaris Nov 29, 2024
e3abbe2
style: fix additional ;
marikaris Nov 29, 2024
ab3acbf
style: delete old not used style
marikaris Nov 29, 2024
ee1eec4
style: set light and dark theme using colour variables so it works on…
marikaris Nov 29, 2024
55b9f68
docs: minor edit as the last bit felt repetitive and this way it fits…
timcadman Nov 29, 2024
d8db679
docs(contributing): remove old docs and refer to new dev guides
marikaris Dec 3, 2024
5ae75a1
docs: add citation footer
marikaris Dec 4, 2024
a8678cd
docs(copyright): adjust message
marikaris Dec 4, 2024
bc709fc
docs(README): alter readme to link to new docs
marikaris Dec 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,4 @@ build/
/application.yaml
/application.properties
.Rproj.user
site/
196 changes: 2 additions & 194 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,195 +1,3 @@
# Contributing to Armadillo

## Running Armadillo from source code

You can run from source code as follows:

1. Install Java and Docker
2. Checkout the source using `git clone https://github.com/molgenis/molgenis-service-armadillo.git`
3. Optionally copy `application.template.yml` to `application.yml` to change settings (will be .gitignored)
4. Compile and execute the code using `./gradlew run`

Note: contact MOLGENIS team if you want to contribute and need a testing OIDC config that you can run against localhost.

# Developing Armadillo

We use gradle to build:

## Running locally

```bash
./gradlew run
```

As we now have the option to download the logfile from the application we need to have one to begin with:

```bash
# Leave out the -a to overwrite instead of append
./gradlew run | tee -a logs/armadillo.log
```

## Running tests

```bash
./gradlew test
```

## Upgrade plugins

Use the commands listed below.

```bash
./gradlew useLatestVersions
./gradlew useLatestVersionsCheck
```

Some plugins need manually editing the build files.

To help find the right `build.gradle` use the command below with your keyword.

```bash
find . -type f -name "build.gradle" -exec echo {} \; -exec grep YOUR_KEY_WORD {} \;
```

## Upgrading gradle

```bash
./gradlew wrapper --gradle-version 8.6
```

## Check for updates

To get a list of new dependencies run

```bash
./gradlew dependencyUpdates
```

## Tools

We use intellij to develop
* To run or debug in intellij, right click on armadillo/src/main/java/org.molgenis.armdadillo/ArmadilloServiceAppliction and choose 'Run/Debug Armadillo...'
* To run using oidc, create a copy of [application.yml](application.template.yml) in root of your project

We have a swagger-ui to quickly see and test available web services at http://localhost:8080/swagger-ui/

## Components

We have several components

- [Armadillo](./armadillo/src/) source
- [UI](./ui/README.md) readme
- [R](./r/) java integration source
- [docker](./docker/README.md)
- [ci](./docker/ci/README.md)
- [scripts](./scripts/README.md) migration
- [install](./scripts/install/README.md)
- [release](./scripts/release/README.md)
- [ops](./scripts/ops/README.md)
- [upgrade](./scripts/upgrade/README.md)

## Releasing

Releases are done whenever the version number gets bumped. For more information see the `Commit messages and versioning (Major, Minor, Patch updates)` section below.

We use mooltiverse [Nyx](https://mooltiverse.github.io/nyx/guide/user/introduction/how-nyx-works/) for changelog and publishing to github.

Run `./gradlew tasks --group Release` to see all release tasks

Use `./gradlew nyxMake` to see what is build in [build/distributions](./build/distributions/).

### Commit messages and versioning (Major, Minor, Patch updates)

Versionnumbers are updated according to [Semantic versioning](https://semver.org/), using [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/).

Please be aware that new releases will only be done when one of the above prefixes is used.

Other prefixes do not indicate user-facing changes and will therefore not result in a version bump, consequently not resulting in a new (pre) release.

Each commit with `!` just before the colon `:` is a major update, indicating a breaking change. So use it wisely. You can also add `BREAKING CHANGE:` in the long commit message format.

- Use `feat!: ...` or `fix!: ...` for a major upgrade, indicating a breaking change.
- Use `feat: ...` for a minor upgrade, indicating a new feature.
- Use `fix: ...` for a patch update, indicating a bugfix.

### Checking log messages

As [changelog template](./changelog-notes.tpl) uses the commit message it is good to check their quality.

List messages to see usage of conventional commits from the past.

```sh
# How many colon usages
git log --pretty=format:"%s" | cut -d: -f1 | sort | uniq -c | sort -n
```

### Building locally

```sh
./gradlew clean assemble
```

## Continuous integration

- We test on each PR and merges on master
- We build docker compose set for CI testing and demo purposes.
- [CI testing](./docker/ci/README.md)
- Demo zip file is a delivery you as artifact
- Master build have a armadillo-compose.zip for download

### Local CI build

```
./gradlew docker
./docker/bin/prepare.bash ci

cd build/docker/armadillo-compose
# Follow README.md to see Armadillo and R images run in container
docker compose build
docker compose up
```

then run `./release-test.R` against this.

### Local CI test of armadillo-compose

Follow [docker CI README.md](./docker/ci/README.md) to run `release-test.R` using `molgenis/r-cicd` image

## Profile xenon with resourcer whitelisted returns a host.docker.internal error
When developing locally, it might be possible to come across the container error: `Could not resolve host: host.docker.internal`,
especially when developing on a non-supported operating system when resourcer is whitelisted (such as xenon).
Sadly, the only way around this error is to edit the JAVA source code of Armadillo to include starting with an extra host.
To enable this feature, you must edit the private method `installImage` of [DockerService.java](https://github.com/molgenis/molgenis-service-armadillo/blob/master/armadillo/src/main/java/org/molgenis/armadillo/profile/DockerService.java) `CreateContainerCmd cmd` from `.withHostConfig(new HostConfig().withPortBindings(portBindings))` to `.withHostConfig(new HostConfig().withPortBindings(portBindings).withExtraHosts("host.docker.internal:host-gateway"))`.

Please note that in order for this change to work, you must use Intellij to run Armadillo or compile the new source code.
Also, if you already have a xenon container build and running, stop and remove that container.

# Developing DataSHIELD packages in Armadillo
As package developer will want to push your new packages into a DataSHIELD profile

* You can start Armadillo with defaults as described above; then use admin/admin as authentication
* to see what profile are available and has been selected:
```
curl -u admin:admin http://localhost:8080/profiles
```
* to change selected profile 'my-profile':
```
curl -X POST http://localhost:8080/select-profile \
-H 'Content-Type: application/json' \
-d 'default'
```
* to install-packages in DataSHIELD current using admin user:
```
curl -u admin:admin -v \
-H 'Content-Type: multipart/form-data' \
-F "file=@dsBase_6.3.0.tar.gz" \
-X POST http://localhost:8080/install-package
```
* to update whitelist of your current profile:
```
curl -u admin:admin -X POST http://localhost:8080/whitelist/dsBase
```
* to get whitelist of current profile:
```
curl -u admin:admin http://localhost:8080/whitelist
```
We really appreciate anyone wanting to contribute to Armadillo. For more information on doing so, visit our
[Developer guides](https://molgenis.github.io/molgenis-service-armadillo/pages/dev_guides/)
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

# Overview

Use MOLGENIS/Armadillo to make data available for privacy protecting federated analysis using [DataSHIELD](https://datashield.org) protocol. Armadillo
Use MOLGENIS Armadillo to make data available for privacy protecting federated analysis using [DataSHIELD](https://datashield.org) protocol. Armadillo
service provides the following features:
* **manage data projects**. Projects can either hold tabular data in the efficient 'parquet' format or any other file use DataSHIELD
'resources' framework.
Expand All @@ -16,11 +16,28 @@ service provides the following features:
* **configure DataSHIELD analysis profiles**. [DataSHIELD analysis profiles](https://www.datashield.org/help/standard-profiles-and-plaforms) are
Docker images that contain a collection of multiple [DataSHIELD analysis packages](https://www.datashield.org/help/community-packages).

![DataSHIELD overview](https://raw.githubusercontent.com/molgenis/molgenis-service-armadillo/master/docs/img/overview-datashield.png)
![DataSHIELD overview](docs/img/ds-complete-setup.png)

## Getting started

For installing and using Armadillo see https://molgenis.github.io/molgenis-service-armadillo/#/
For installing and using Armadillo see our
[Documentation](https://molgenis.github.io/molgenis-service-armadillo/pages/quick_start/).

For developing and contributing see [Contributing](./CONTRIBUTING.md).
For developing and contributing see [our dev guides](https://molgenis.github.io/molgenis-service-armadillo/pages/dev_guides/).

### Quick start
#### Jar
1. Download the jar from our
[releases page](https://github.com/molgenis/molgenis-service-armadillo/releases).
2. Copy paste the contents of
[application-template.yml](https://github.com/molgenis/molgenis-service-armadillo/blob/master/application.template.yml)
and paste it in a file called `application.yml`, in the same folder as the downloaded jar.
3. To start the application, run `java -jar molgenis-armadillo-x.yy.zz.jar`.
4. Go to `http://localhost:8080` to see the Armadillo UI.

#### Docker
For testing without having to installing Java you can run using docker:
1. Install [docker-compose](https://docs.docker.com/compose/install/)
2. Download this [docker-compose.yml](docker-compose.yml).
3. Execute `docker-compose up`
4. Once it says 'Started', go to http://localhost:8080 to see your Armadillo running.
Empty file removed docs/.nojekyll
Empty file.
19 changes: 0 additions & 19 deletions docs/DataSHIELD-datamanagement/0_Prerequisite.r

This file was deleted.

118 changes: 0 additions & 118 deletions docs/DataSHIELD-datamanagement/1_MolgenisArmadillo.r

This file was deleted.

Loading
Loading