Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: dCache/dcache
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 10.2.2
Choose a base ref
...
head repository: dCache/dcache
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref

Commits on Oct 7, 2024

  1. [maven-release-plugin] prepare for next development iteration

    lemora committed Oct 7, 2024
    Copy the full SHA
    046511b View commit details

Commits on Oct 9, 2024

  1. dcache: reject access to files with nearline QoS

    Motivation:
    If file has QoS policy "HSM-only" then dCache should reject the access
    to the file, even if file is still available on disk (as transition
    might take some time).
    
    Modification:
    - Update QoS policy engine to reflect policy in the namespace attributes
    - Update Transfer to request policy state (as an indicate of applied
      policy) and reject read transfers if access latency is nearline.
    
    Result:
    dCache behavior compliant with selected QoS policy.
    
    Acked-by: Lea Morschel
    Acked-by: Dmitry Litvintsev
    Target: master
    Require-book: no
    Require-notes: yes
    kofemann committed Oct 9, 2024
    Copy the full SHA
    1a3c12e View commit details
  2. pool manager: retry request on pool up

    Motivation:
    If a pool with the file is online and a tape copy available, then
    dCache will trigger stage and wait until file is restored on disk.
    However, if pool becomes available again, the stage request is not
    interrupted and client will wait for tape.
    
    Modification:
    Update request container 'onPoolUp' logic to retry the request if
    the file expected to be on that pool. Added unit test to validate the
    behavior.
    
    Result:
    pool selection succeeds then a pool with the file becomes online
    despite the on-going stage request.
    
    NOTE (1): the stage request is not interrupted
    NOTE (2): if newly enabled pool doesn't contains the expected file, then
    double stage is very likely.
    
    Target: master
    Acked-by: Lea Morschel
    Require-book: no
    Require-notes: yes
    kofemann committed Oct 9, 2024
    Copy the full SHA
    f945c3d View commit details
  3. github: mirror pull requests

    kofemann committed Oct 9, 2024
    Copy the full SHA
    cd9c90d View commit details
  4. Copy the full SHA
    e697949 View commit details
  5. github: keep branch name for regualr push events

    kofemann committed Oct 9, 2024
    Copy the full SHA
    703e5e0 View commit details
  6. github: drop branch name manipulations

    kofemann committed Oct 9, 2024
    Copy the full SHA
    43a6082 View commit details

Commits on Oct 14, 2024

  1. build(deps): bump org.eclipse.jetty:jetty-servlets

    Bumps org.eclipse.jetty:jetty-servlets from 9.4.52.v20230823 to 9.4.54.v20240208.
    
    ---
    updated-dependencies:
    - dependency-name: org.eclipse.jetty:jetty-servlets
      dependency-type: direct:production
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Oct 14, 2024
    Copy the full SHA
    c18934e View commit details

Commits on Oct 15, 2024

  1. Merge pull request #7680 from dCache/dependabot/maven/org.eclipse.jet…

    …ty-jetty-servlets-9.4.54.v20240208
    
    build(deps): bump org.eclipse.jetty:jetty-servlets from 9.4.52.v20230823 to 9.4.54.v20240208
    kofemann authored Oct 15, 2024
    Copy the full SHA
    2e4e416 View commit details
  2. build(deps): bump org.eclipse.jetty:jetty-server

    Bumps org.eclipse.jetty:jetty-server from 9.4.52.v20230823 to 9.4.55.v20240627.
    
    ---
    updated-dependencies:
    - dependency-name: org.eclipse.jetty:jetty-server
      dependency-type: direct:production
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    dependabot[bot] authored Oct 15, 2024
    Copy the full SHA
    8254b67 View commit details
  3. Merge pull request #7679 from dCache/dependabot/maven/org.eclipse.jet…

    …ty-jetty-server-9.4.55.v20240627
    
    build(deps): bump org.eclipse.jetty:jetty-server from 9.4.52.v20230823 to 9.4.55.v20240627
    kofemann authored Oct 15, 2024
    Copy the full SHA
    53e6665 View commit details
  4. http(old): modernize color palette

    Motivation:
    match dcache.org page colors
    
    Modification:
    pulled colors into css variables
    synced colors across pages
    fixed html generation
    
    Result:
    new look-and-feel
    
    Acked-by: Lea Morschel
    Target: master, 10.2
    Require-book: no
    Require-notes: yes
    kofemann committed Oct 15, 2024
    Copy the full SHA
    7e2040e View commit details

Commits on Oct 21, 2024

  1. chimera: correct a typo

    Target: master
    Requires-notes: no
    Requires-book: no
    Acked-by: Lea Morschel
    mksahakyan committed Oct 21, 2024
    Copy the full SHA
    cbff12d View commit details

Commits on Oct 23, 2024

  1. ci: use java-21 for runtime

    kofemann committed Oct 23, 2024
    Copy the full SHA
    8d83ed5 View commit details

Commits on Oct 28, 2024

  1. xroot: handle haproxy and checksum command

    Motivation:
    ----------
    When using xrootd doors behind an HAProxy w/
    `xrootd.enable.proxy-protocol=true` it has been discovered
    that
    ```
    xrdcp --cksum adler32:<value> <source> <destiation>
    ```
    hangs after upload has completed and then eventually fails
    after a timeout. This is due to xrootd door repoting actual
    door address to the client.
    
    Modification:
    -------------
    Return destination address (that is haproxy address) if
    `xrootd.enable.proxy-protocol=true` is set.
    
    Result:
    -------
    ```
    xrdcp --cksum adler32:<value> <source> <destiation>
    ```
    works as expected (and likely many other similar commands)
    
    Target: trunk
    Request: 10.*
    Request: 9.2
    Patch: https://rb.dcache.org/r/14338/
    Acked-by: Tigran
    Require-book: no
    Require-notes: yes
    Signed-off-by: Dmitry Litvintsev <litvinse@fnal.gov>
    DmitryLitvintsev committed Oct 28, 2024
    Copy the full SHA
    e98ab94 View commit details

Commits on Nov 5, 2024

  1. bulk/qos: shutdown executor services when stopped

    Motivation:
    As scheduled executors are not shutdown cleanly, when dcache stops
    running threads logged.
    
    Modification:
    shutdown ConcurrentRequestManager and adjuster-executor when dcache
    stopped.
    
    Result:
    
    less stack traces in logs.
    
    Acked-by: Lea Morschel
    Target: master, 10.2
    Require-book: no
    Require-notes: yes
    kofemann committed Nov 5, 2024
    Copy the full SHA
    5f67bdd View commit details

Commits on Nov 6, 2024

  1. tape REST api: additional fix tohandling of prefixed paths

    Motivation:
    -----------
    Users reported 2 day pin lifetime on staged files (which is a default)
    despite specifying different values. This is due to failure to match
    target key on truncated vs prefixed path in target argument map keyed on
    un-prefixed paths.
    
    Modification:
    -------------
    Use full target paths throughout the system. Make sure to strip prefix
    when exposing paths to users.
    
    Result:
    -------
    Observe correct user specified pin lifetime.
    
    Ticket: #7687
    Patch: https://rb.dcache.org/r/14339/
    Target: trunk
    Request: 10.2, 10.1, 10.0, 9.2
    Require-book: no
    Require-notes: yes
    
    Signed-off-by: Dmitry Litvintsev <litvinse@fnal.gov>
    DmitryLitvintsev committed Nov 6, 2024
    Copy the full SHA
    fff8b79 View commit details

Commits on Nov 12, 2024

  1. container: add plugin classpath if defined

    kofemann committed Nov 12, 2024
    Copy the full SHA
    404dc6d View commit details

Commits on Nov 15, 2024

  1. docs: correcting typos in chapter 2

    Motivation: found some typos
    
    Modification: removal of typos
    
    Result: no more typos
    
    Acked-by:
    Target: master
    Require-book: no
    Require-notes: no
    khys95 committed Nov 15, 2024
    Copy the full SHA
    47ef9a7 View commit details

Commits on Nov 18, 2024

  1. test releases xml file creation

    mksahakyan committed Nov 18, 2024
    Copy the full SHA
    d92fdf1 View commit details

Commits on Nov 19, 2024

  1. docs: update to the minimal dCache installation

    Motivation: the minimal dCache installation file contained some confusing information
    
    Modification: wording, information, links, general aesthetic, structure
    
    Result: A more user friendly installation ...maybe?
    
    Acked-by: Tigran Mkrtchyan
    Target: master
    Require-book: no
    Require-notes: no
    khys95 committed Nov 19, 2024
    Copy the full SHA
    6a3e899 View commit details

Commits on Nov 20, 2024

  1. cells: ignore empty core domain uris propagated by zk

    Motivation:
    When Zookeeper updates core domain infos, dCache will first kill the existing cell tunnels
    and then later try to read and parse the new value. If the new value is an empty
    string (for whatever reason), parsing will fail, but a new connection will not be
    established. The corresponding error in the log:
    
    18 Nov 2024 08:45:00 (c-dcache-head-xxx03_messageDomain-AAYmVA1LtnA-AAYmVA16phA) [dcache-head-xxx03_messageDomain,9.2.21,CORE] Error while reading from tunnel: java.net.SocketExceptio>
    18 Nov 2024 08:45:43 (c-dcache-head-xxx03_messageDomain-AAYnKxn40fA) [] Uncaught exception in thread TunnelConnector-dcache-head-xxx03_messageDomain
    java.lang.NullPointerException: null
            at java.base/java.net.Socket.<init>(Socket.java:448)
            at java.base/java.net.Socket.<init>(Socket.java:264)
            at java.base/javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
            at dmg.cells.network.LocationManagerConnector.connect(LocationManagerConnector.java:64)
            at dmg.cells.network.LocationManagerConnector.run(LocationManagerConnector.java:94)
            at dmg.cells.nucleus.CellNucleus.lambda$wrapLoggingContext$2(CellNucleus.java:725)
            at java.base/java.lang.Thread.run(Thread.java:829)
    
    Modification:
    before killing existing tunnel check that ZK didn't propagate empty
    data.
    
    Result:
    More roust cell communication
    
    NOTE: a non empty invalid data still accepted!!!
    
    Fixes: #7696
    Acked-by: Lea Morschel
    Target: master, 10.2, 10.1, 10.0, 9.2
    Require-book: no
    Require-notes: yes
    kofemann committed Nov 20, 2024
    Copy the full SHA
    30829c9 View commit details
  2. docs: diagram on successful write workflow

    Motivation: Got curious about what the flow of a client writing to a file would look like.
    
    Modification: Adding two mermaid diagrams that show the flow of a successful write for supported protocols, specifically steps 14 and 15.
    
    Result: Help someone understand some concepts better.
    
    Acked-by: Tigran Mkrtchyan
    Target: master
    Require-book: no
    Require-notes: no
    khys95 committed Nov 20, 2024
    Copy the full SHA
    4b922a1 View commit details
  3. vehicles: support door requesting multiple checksums

    Motivation:
    
    A client that is uploading a file or initiating an HTTP TPC pull
    transfer may wish to validate the transfer did not result in data
    corruption, using the new file's checksum / digest value.  When
    receiving a file, the pool will calculate a set of checksums (from
    different checksum algorithms) based on the pool's configuration.  This
    configured list of checksum algorithms might not include the client's
    desired checksum algorithm.
    
    One solution would be for the pool configuration to be updated, to
    include the client's desired checksum.  Although technically possible,
    this is impractical, as it would require the person transferring the
    data to negotiate with the dCache admins, asking them to update the pool
    configuration for all pools their transfer might use.  Such support
    operations are undesirable.
    
    As an alternative approach, the WebDAV door accepts a `Want-Digest` HTTP
    request header on PUT and COPY (HTTP-TPC) requests.  The door uses this
    header to select a single desired checksum algorithm.  This algorithm
    (if provided) is transferred to the pool, which then ensures that this
    algorithm is calculated during the file upload.
    
    A dCache instance may provide storage to multiple communities, with
    different conventions on which algorithm is use for data integrety.  A
    client initiating a transfer between these communities (using dCache as
    an intermediate) would require two checksums: one to verify data
    integrety of the transfer to dCache and another to validate transferred
    within the second community.
    
    Modification:
    
    The ProtocolInfo subclasses are updated to carry a collection of
    algorithms, taking care to maintain backwards compatibility.
    
    Result:
    
    No user- or admin observable change, but dCache now supports a door
    requesting multiple checksum algorithms when a pool receives a file.
    
    Target: master
    Request: 10.2
    Requires-notes: yes
    Requires-book: no
    Patch: https://rb.dcache.org/r/14341/
    Acked-by: Tigran Mkrtchyan
    paulmillar committed Nov 20, 2024
    Copy the full SHA
    9023cea View commit details

Commits on Nov 27, 2024

  1. poolmanager: fix typo

    weghtedPools -> weightedPools
    
    Acked-by: Paul Millar
    Target: master
    Require-book: no
    Require-notes: no
    kofemann committed Nov 27, 2024
    Copy the full SHA
    69f5c77 View commit details
  2. cells: remove backward compatibility 7.0

    Motivation:
    The initial implementation of tls support cell tunneling used
    `/dcache/lm/core-config` file to flag dcache to start inter-cell tunnels
    in plain of tls modes. This have been later replaces with urls syntax
    tcp:// and tls://. The old code left in place for backward
    compatibility, so, time to drop.
    
    Modification:
    Remove handling of `/dcache/lm/core-config` ZK node. Removed commands
    `get/set core-config`
    
    Result:
    Less dead code.
    
    Acked-by: Lea Morschel
    Target: master
    Require-book: no
    Require-notes: yes
    kofemann committed Nov 27, 2024
    Copy the full SHA
    b63677e View commit details
  3. jvm: drop UseCompressedOops JVM option

    Motivation:
    Starting java7, the UseCompressedOops is dynamically controed by heap
    size.
    
    ```
    $ java -Xmx32g -XX:+PrintFlagsFinal 2>/dev/null | grep UseCompressedOops
         bool UseCompressedOops                        = false                          {product lp64_product} {default}
    
    $ java -Xmx28g -XX:+PrintFlagsFinal 2>/dev/null | grep UseCompressedOops
         bool UseCompressedOops                        = true                           {product lp64_product} {ergonomic}
    ```
    
    The mismatch between UseCompressedOops endup with error:
    
    ```
    OpenJDK 64-Bit Server VM warning: Max heap size too large for Compressed Oops
    ***** WARNING! INCORRECT SYSTEM CONFIGURATION DETECTED! *****
    The system limit on number of memory mappings per process might be too low for the given
    [gc] max Java heap size (40960M). Please adjust /proc/sys/vm/max_map_count to allow for at
    [gc] least 73728 mappings (current limit is 65530). Continuing execution with the current
    ```
    
    Modification:
    drop UseCompressedOops JVM option from defaults.
    
    Result:
    correct behavior on JVMs with large heap
    
    Acked-by: Lea Morschel
    Target: master
    Require-book: no
    Require-notes: yes
    kofemann committed Nov 27, 2024
    Copy the full SHA
    59ea69f View commit details

Commits on Dec 3, 2024

  1. AI release notes: ignore commits by maven-release-plugin

    lemora authored Dec 3, 2024
    Copy the full SHA
    103bfeb View commit details

Commits on Dec 4, 2024

  1. pool: allow max disksize to be specified as percent of filesystem

    Motivation:
    To avoid disk dise miscalculations, it will be useful if pool is
    configured not with an absolute pool size, but with a percentage of
    available space.
    
    Modification:
    Update `set max diskspace` to accept percentage. Update space
    calculation to use percentage of filesystems total size, if configured.
    
    Result:
    simplified pool configuration
    
    ```
    set max diskspace 94%
    ...
    Limits for maximum disk space
        File system (limit)  : 547735052288 (94%)
        Statically configured: -
        Runtime configured   : -
    ```
    
    Acked-by: Dmitry Litvintsev
    Target: master
    Require-book: no
    Require-notes: yes
    kofemann committed Dec 4, 2024
    Copy the full SHA
    8c71927 View commit details
  2. ci: skip maven-release-plugin generated commit messages by release notes

    kofemann committed Dec 4, 2024
    Copy the full SHA
    289e9f1 View commit details

Commits on Dec 6, 2024

  1. cells: always try to re-establish dead tunnel, unless stopped

    Motivation:
    As long as cell tunnel is not explicitly stopped by calling
    dmg.cells.network.LocationManagerConnector#stopped, other interrupts
    should be ignored.
    
    Modification:
    Update retry logic to never give up, unless _isRunning flag set to
    false.
    
    Result:
    More robust tulles in case of network issues.
    
    Issue: #7707, #5326
    Acked-by: Dmitry Litvintsev
    Target: master, 10.2, 10.1, 10.0, 9.2
    Require-book: no
    Require-notes: yes
    kofemann committed Dec 6, 2024
    Copy the full SHA
    600ed1f View commit details
  2. qos: QOS fails with 'Attribute is not defined: QOS_POLICY'

    Motivation: Find out where in the code we throw the above error and resolve it.
    
    Modification: Removal of redundant qos policy logic.
    
    Result: Instead of checking if the qos_policy attribute is defined AND null, we will check if it is present using one of the predefined methods.
    If not, set it to null and resume previous logic.
    
    Acked-by: Tigran Mkrtchyan
    Target: master, 10.2, 10.1, 10.0, 9.2
    Require-book: no
    Require-notes: yes
    khys95 committed Dec 6, 2024
    Copy the full SHA
    8b9dfb3 View commit details
  3. bulk: handle absolute/relative paths in uniform fashion

    Motivation:
    -----------
    Recent change(s) that massaged user input target paths and
    stored absolute paths on bulk backend lead to ambiguity between
    user provided and dcache resolved paths and also resulted in inability
    to use full paths (i.e. only relative paths are supported). At
    Fermilab we need to use both - relative and absolute paths
    
    Modification:
    -------------
    Revert all recent changes that appended prefix to user
    supplied paths, stored the result and then stripped the
    prefix so that only "original" paths are exposed to the user.
    Instead, like before, store user supplied paths but carry
    over request prefix which is computed from user root and
    door root. When calling PnfsManager using paths the full
    paths of the targets are reassembled using the prefix
    
    Result:
    ------
    Restored ability to use absolute paths when using REST API.
    
    Issue: #7693
    Patch: https://rb.dcache.org/r/14355/
    Target: trunk
    Request: 10.2, 10.1, 10.0, 9.2
    Require-book: no
    Require-notes: yes
    Acked-by: Lea Morschel, Tigran Mkrtchyan
    DmitryLitvintsev committed Dec 6, 2024
    Copy the full SHA
    484e967 View commit details

Commits on Dec 13, 2024

  1. bulk: update message returned when trying to stage a file present on …

    …disk
    
    Motivation: as per ticket #10666 when a user tries to stage a file from disk it is an illegal transition.
    
    Modification: update message displayed to user.
    
    Result: user will know this is intended functionality.
    
    Acked-by:Lea Morschel
    Target: master
    Require-book: no
    Require-notes: no
    khys95 committed Dec 13, 2024
    Copy the full SHA
    abe4bd9 View commit details

Commits on Dec 19, 2024

  1. pool: separate cases for disk error and no space available

    Motivation
    
    when there are no avvailable space on a pool, the pool will go to DISABLED mode and no data could be read anymore.
    
    This is wrong we still want to be able to read data from the file.
    
    Modification
    
    this is a temp change, trying to get the info from the eroor message, sice therer is no a specific error code,
    
    and then based on that info separate two cases DISABLED AND READONLY
    
    Acked-by: Tigran Mkrtchyan
    Target: master. 10.2, 10.1, 10.0, 9.2
    Require-book: no
    Require-notes: yes
    Commited:master@862963e
    mksahakyan committed Dec 19, 2024
    Copy the full SHA
    bfbf240 View commit details

Commits on Dec 20, 2024

  1. migration: new MigrationModule option and copy completion message

    **Motivation** allow modified replication behavior to support hotspot
    mitigation. Specifically we need replication jobs to complete without
    error when the number of replicas is maximized with respect to the
    available pools, even if more replicas were requested.
    
    **Modification:**
    
    * In `Job.java`: `Error` -> `Note`.
    
      The names of all related methods are changed accordingly
      (e.g. `addError()` -> `addNote()`).
    
    * `MigrationModule` gets new option `-target-deficit-mode=(wait|limit)`
      (default `wait`) determining behavior when requested replicas exceeds
      target pool availability:
    
        * `wait`
    
            The job will wait for new pools to become available (e.g. added
            to specified pool group). This is the default.
    
        * `limit`
    
            If more replicas are requested than exist available pools, the
            job will complete successfully (with a note) when the number of
            replicas is maximized for the pools currently available.
    
    * In interface `TaskCompletionHandler`: new method
      `taskCompletedWithNote(Task, String)` with default implementation
      forwarding to `taskCompleted(Task)`.
    
    * New (manually-generated) files `Task_sm.dot`, and `Task_sm.dot.png`
      showing the state machine model encoded by `Task.sm`.
    
    * New method in `Task.java`: `moreReplicasPossible()` for use in state
      transition guards.
    
    * New finite state machine (FSM) action
      `notifyCompletedWithInsufficientReplicas()` in `Task.java`.
    
    * In `Task.sm`, `Done` entry action `notifyCompleted()` removed in favor
      of per-transition actions `notifyCompleted()`, or
      `notifyCompletedWithInsufficientReplicas()` as appropriate. It was
      determined by analysis that moving the post-state-change entry action
      to a pre-state-change transition did not disrupt the intended
      semantics of the state machine.
    
    * New guarded transitions to `Done` when more replicas are requested
      than are possible with the target pools available currently.
    
    *  `Task_sm.dot`, and `Task_sm.dot.png` updated manually to reflect the
       changes to the state machine.
    
    **Result:**
    
    * Partially implements #7675
    
    * Replication jobs can be issued with `-target-deficit-mode=limit` to
      terminate successfully when all pools available currently have been
      populated with replicas, but more replicas were requested.
    
    **Target:** master
    **Request:** -
    **Patch:** https://rb.dcache.org/r/14362/
    **Closes:**
    **Requires-notes:** yes
    **Requires-book:**
    **Acked-by:** Tigran Mkrtchyan, Dmitry Litvintsev
    greenc-FNAL committed Dec 20, 2024
    Copy the full SHA
    f41cd3e View commit details

Commits on Jan 20, 2025

  1. poolselection: fix typo

    everthing --> everything
    
    Acked-by: Marina Sahakyan
    Target: master
    Require-book: no
    Require-notes: no
    khys95 committed Jan 20, 2025
    Copy the full SHA
    693fffa View commit details

Commits on Jan 27, 2025

  1. utils: use JFR events to track door interaction with other cells

    Motivation:
    for slow transfers we want to be able to see how long cell
    communications took time. This information can be collected with java
    flight recorder.
    
    Modification:
    Update Transfer class to record JFR events when interaction with other
    cells takes place. The mover stop implementation uses an addition event
    holder, as mover kill uses notification, thus no callback.
    
    Result:
    message interaction information visible in JFR.
    
    ```
    $ jfr print --categories dCache /tmp/dCacheDomain_12455560939331420208.jfr | head -20                                                            (git)-[transfer-jfr]
    org.dcache.door.Messages {
      startTime = 09:45:35.672
      duration = 37.8 ms
      message = "PnfsGetFileAttributes"
      Source = "NFS-nairi@dCacheDomain"
      Destination = "PnfsManager"
      eventThread = "proc-9" (javaThreadId = 73)
      stackTrace = [
        org.dcache.util.Transfer.lambda$readNameSpaceEntryAsync$2(Transfer$MessageEvent, boolean, PnfsGetFileAttributes) line: 818
        org.dcache.cells.CellStub.lambda$transformAsync$3(AsyncFunction, Message) line: 559
        com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AsyncFunction, Object) line: 223
        com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(Object, Object) line: 210
        com.google.common.util.concurrent.AbstractTransformFuture.run() line: 123
      ]
    }
    
    ...
    ```
    
    Acked-by: Lea Morschel
    Target: master
    Require-book: no
    Require-notes: yes
    kofemann committed Jan 27, 2025
    Copy the full SHA
    3b12aa7 View commit details
  2. chimera: on shutdown propagate close to DB specific driver

    Motivation:
    If chimera DB driver is statefull it might require an explicit close on
    shutdown.
    
    Modification:
    Add FsSqlDriver#close method that is called by JdbcFs#close.
    
    Result:
    better handling of db resources.
    
    Acked-by: Paul Millar
    Target: master
    Require-book: no
    Require-notes: no
    kofemann committed Jan 27, 2025
    Copy the full SHA
    76fe850 View commit details
  3. Bulk: PinManagerActivity do not set state of files that are pinned in…

    …definitely to SKIPPED
    
    Motivaton:
    ----------
    
    Setting files having infinite pin to state SKIPPED seems to
    prevents them from being staged if pool goes down.
    
    Modification:
    -------------
    
    Set state to COMPLETED if pin lifetime is infinite.
    
    Result:
    ------
    
    As tested and reported by DESY, the staging of files that happen
    to be on offline pools works properly
    
    Target: trunk
    Request: 10.2
    Request: 9.2
    Patch: https://rb.dcache.org/r/14365/
    
    Require-notes: yes
    Require-book: no
    DmitryLitvintsev committed Jan 27, 2025
    Copy the full SHA
    2b5f8bc View commit details

Commits on Jan 29, 2025

  1. libs: update Liquibase to 4.29.2

    Motivation:
    due to compatibility breaking changes (in particular the type which is
    used for BLOBs) in Liquibase we ware unable to update Liquibase version.
    As a result, enhancements and bug fixes.
    
    liqubase uses id+author to identify changesets. In case of collisions
    the behavour is unpredictable, in particular, with liquibase 4.xx the
    wrong changeset is selected for validation.
    
    Modification:
    Add DB specific conditional statement to explicitly handle incompatibility.
    
    To ensure unique author+id combination.  Add a concept of changeset-pre that will
    update checksums in databasechangelog to match new calculation rules
    (whitespace included vs excluded) and match checksums to new ids.
    
    `dcache database update` command as well as auto-migrate are updates to
    apply `pre` changes before the main change set.
    
    Result:
    Up-to-date liquibase version.
    
    Acked-by: Lea Morschel
    Acked-by: Chris Green
    Target: master
    Require-book: no
    Require-notes: yes
    kofemann committed Jan 29, 2025
    Copy the full SHA
    68f7ffe View commit details

Commits on Jan 30, 2025

  1. dcache-billing-indexer: add explicit dependency on commons-io

    Motivation:
    the dcache-billing-indexer uses commons-compress to handle bz2 files,
    and has non-optional dependency on commons-io, which is missing. So we
    get:
    
    ```
    /usr/sbin/dcache-billing-indexer -index /var/lib/dcache/billing/2025/01/billing-2025.01.27.bz2
    ERROR - Uncaught exception
    java.lang.NoClassDefFoundError: org/apache/commons/io/input/CloseShieldInputStream
    at org.dcache.services.billing.text.Indexer$7.openStream(Indexer.java:526)
    at com.google.common.io.ByteSource$AsCharSource.openStream(ByteSource.java:474)
    at com.google.common.io.CharSource.readLines(CharSource.java:371)
    at org.dcache.services.billing.text.Indexer.produceIndex(Indexer.java:510)
    at org.dcache.services.billing.text.Indexer.index(Indexer.java:396)
    at org.dcache.services.billing.text.Indexer.(Indexer.java:211)
    at org.dcache.services.billing.text.Indexer.main(Indexer.java:686)
    Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.input.CloseShieldInputStream
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
    ... 7 common frames omitted
    ```
    
    Modification:
    explicitly add commons-io into dcache-billing-indexer classpath.
    
    Result:
    dcache-billing-indexer can process compressed files.
    
    Fixes: #7738
    Acked-by: Lea Morschel
    Tested-by: Ville Salmela
    Target: master, 10.2
    Require-book: no
    Require-notes: yes
    kofemann committed Jan 30, 2025
    Copy the full SHA
    153a018 View commit details

Commits on Jan 31, 2025

  1. cells: fix race condition between thread start and flag check

    Motivation:
    
    If newly started thread runs before `running` flag is set, then tunnel
    with shutdown instantly.
    
    Thread-1: Create T2 -->  | T2.start()         | --> set running flag
    Thread-2:                | check flag; exit   |
    
    Modification:
    
    set `running` before thread starts.
    
    Result:
    race is fixed
    
    Issue: #5326
    Acked-by: Paul Millar
    Target: master, 10.2
    Require-book: no
    Require-notes: yes
    kofemann committed Jan 31, 2025
    Copy the full SHA
    2440b22 View commit details

Commits on Feb 4, 2025

  1. billing: database fixes necessitated by Liquibase update

    Motivation:
    
    [Patch](https://rb.dcache.org/r/13594/) updating Liquibase to version
    4.29 omitted a necessary checksum change for a changeset for the
    optional billing system.
    
    Modification:
    
    * Create a new changeset for the billing system with highest priority to
      update the defective checksum in the DB prior to any other changesets
      being evaluated.
    
    * Add a new `bean` to ensure that the new changeset is loaded prior to
      any others.
    
    Result:
    
    Billing system will start correctly with Liquibase 4.29.
    
    Target: master
    Patch: https://rb.dcache.org/r/14369
    Requires-notes: no
    Requires-book: no
    Acked-by: Dmitry Litvintsev
    greenc-FNAL committed Feb 4, 2025
    Copy the full SHA
    705a6dd View commit details

Commits on Feb 7, 2025

  1. pool: add workaround space mismanagementd by XFS (and others?)

    Motivation:
    Some filesystems miscalculate/report free space in respect tu used
    space:
    
    ```
    $ df /dcache/pool-a
    Filesystem        1K-blocks         Used Available Use% Mounted on
    /dev/sda1      558602657792 558397210232 205447560 100% /dcache/pool-a
    
    $ du -s /dcache/pool-a
    554502450484	/dcache/pool-a
    
    $ bc
    558602657792 - 554502450484
    4100207308
    ```
    
    File system reports 200GB of the free space, however, total - real used
    gives ~4TB.
    
    Thus dCache assumes 4TB and write the file system full... ==> IO error
    
    Modification:
    
    use an `effective` free space, which is the minimum between disk
    reported and internal accounting free spaces.
    
    Result:
    Pool uses the effective free space instead of mathematically correct
    value.
    
    Acked-by: Dmitry Litvintsev
    Target: master, 10.2
    Require-book: no
    Require-notes: yes
    kofemann committed Feb 7, 2025
    Copy the full SHA
    208bfbe View commit details

Commits on Feb 11, 2025

  1. qos: Uncaught NullPointerException due to QoS being set to null

    Motivation: null is a valid value for FileAttribute.QOS_POLICY, however, it is not a valid value to be encapsulated within an Optional.
    
    Modification: Update Optional.of for Optional.ofNullable since the later will take into account null as a value and return Optional.empty() where Optional.of will not and hence throw the NPE.
    
    Result: QosPolicy can still be set to null and we will no longer show a NPE.
    
    Acked-by: Dmitry Litvintsev
    Target: master, 10.2, 10.1, 10.0 and 9.2
    Require-book: no
    Require-notes: no
    khys95 committed Feb 11, 2025
    Copy the full SHA
    466c97e View commit details
Showing with 2,211 additions and 2,077 deletions.
  1. +150 −0 .ci/generate-releases-xml.sh
  2. +17 −16 .github/workflows/mirror.yml
  3. +27 −3 .gitlab-ci.yml
  4. +1 −1 archetypes/dcache-nearline-plugin-archetype/pom.xml
  5. +1 −1 archetypes/pom.xml
  6. +1 −1 docs/TheBook/pom.xml
  7. +131 −1,063 docs/TheBook/src/main/markdown/dcache-minimal-installation.md
  8. +3 −4 docs/TheBook/src/main/markdown/install.md
  9. +1 −1 docs/UserGuide/pom.xml
  10. +1 −1 docs/pom.xml
  11. +1 −1 modules/acl-vehicles/pom.xml
  12. +1 −1 modules/acl/pom.xml
  13. +1 −1 modules/benchmarks/pom.xml
  14. +1 −1 modules/cells/pom.xml
  15. +10 −5 modules/cells/src/main/java/dmg/cells/network/LocationManagerConnector.java
  16. +26 −209 modules/cells/src/main/java/dmg/cells/services/LocationManager.java
  17. +1 −6 modules/chimera/pom.xml
  18. +9 −1 modules/chimera/src/main/java/org/dcache/chimera/FsSqlDriver.java
  19. +1 −0 modules/chimera/src/main/java/org/dcache/chimera/JdbcFs.java
  20. +71 −0 modules/chimera/src/main/resources/org/dcache/chimera/changelog/00-liquibase-fixes.xml
  21. +5 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changelog-master.xml
  22. +10 −10 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-1.8.0.xml
  23. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-1.9.12.xml
  24. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-1.9.13.xml
  25. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-1.9.2.xml
  26. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-1.9.3.xml
  27. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-10.xml
  28. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.10.xml
  29. +4 −4 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.13.xml
  30. +5 −5 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.14.xml
  31. +23 −23 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.15.xml
  32. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.3.xml
  33. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.7.xml
  34. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.8.xml
  35. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.9.xml
  36. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-3.1.xml
  37. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-5.2.xml
  38. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-6.0.xml
  39. +2 −2 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-6.2.xml
  40. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-7.1.xml
  41. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-7.2.xml
  42. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-8.2.xml
  43. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-9.1.xml
  44. +1 −1 modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-9.2.xml
  45. +6 −1 modules/common-cli/pom.xml
  46. +1 −1 modules/common-security/pom.xml
  47. +1 −1 modules/common/pom.xml
  48. +1 −1 modules/dcache-bulk/pom.xml
  49. +3 −2 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/BulkActivity.java
  50. +3 −2 ...les/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/delete/DeleteActivity.java
  51. +1 −1 ...les/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/log/LogTargetActivity.java
  52. +7 −6 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinActivity.java
  53. +4 −5 ...es/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/PinManagerActivity.java
  54. +4 −2 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/ReleaseActivity.java
  55. +10 −8 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/StageActivity.java
  56. +4 −2 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/pin/UnpinActivity.java
  57. +20 −8 ...les/dcache-bulk/src/main/java/org/dcache/services/bulk/activity/plugin/qos/UpdateQoSActivity.java
  58. +18 −58 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/job/BulkRequestContainerJob.java
  59. +1 −0 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/manager/ConcurrentRequestManager.java
  60. +8 −5 modules/dcache-bulk/src/main/java/org/dcache/services/bulk/util/BulkRequestTarget.java
  61. +1 −1 modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/bulk.xml
  62. +3 −3 modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/model/db.changelog-8.0.xml
  63. +9 −9 modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/model/db.changelog-9.0.xml
  64. +1 −1 modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/model/db.changelog-9.2.xml
  65. +1 −1 modules/dcache-bulk/src/main/resources/org/dcache/services/bulk/model/db.changelog-master.xml
  66. +1 −1 modules/dcache-chimera/pom.xml
  67. +1 −1 modules/dcache-chimera/src/main/java/org/dcache/chimera/namespace/ChimeraNameSpaceProvider.java
  68. +8 −1 modules/dcache-chimera/src/main/resources/diskCacheV111/namespace/pnfsmanager-chimera.xml
  69. +1 −1 modules/dcache-dcap/pom.xml
  70. +1 −1 modules/dcache-frontend/pom.xml
  71. +18 −4 modules/dcache-frontend/src/main/java/org/dcache/restful/resources/bulk/BulkResources.java
  72. +5 −7 modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ArchiveInfoResources.java
  73. +5 −3 modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/ReleaseResources.java
  74. +8 −2 modules/dcache-frontend/src/main/java/org/dcache/restful/resources/tape/StageResources.java
  75. +19 −4 modules/dcache-frontend/src/main/java/org/dcache/restful/util/wlcg/ArchiveInfoCollector.java
  76. +2 −1 ...les/dcache-frontend/src/test/java/org/dcache/restful/resources/bulk/BulkRequestJsonParseTest.java
  77. +1 −1 modules/dcache-ftp/pom.xml
  78. +1 −1 modules/dcache-gplazma/pom.xml
  79. +1 −1 modules/dcache-history/pom.xml
  80. +1 −1 modules/dcache-info/pom.xml
  81. +1 −1 modules/dcache-nearline-spi/pom.xml
  82. +1 −1 modules/dcache-nfs/pom.xml
  83. +8 −1 modules/dcache-nfs/src/main/resources/org/dcache/chimera/nfsv41/door/nfsv41-common.xml
  84. +1 −1 modules/dcache-qos/pom.xml
  85. +2 −1 modules/dcache-qos/src/main/java/org/dcache/qos/services/adjuster/adjusters/CopyAdjuster.java
  86. +7 −29 modules/dcache-qos/src/main/java/org/dcache/qos/services/engine/provider/PolicyBasedQoSProvider.java
  87. +1 −1 modules/dcache-qos/src/main/resources/org/dcache/qos/model/db.changelog-8.0.xml
  88. +1 −1 modules/dcache-qos/src/main/resources/org/dcache/qos/model/db.changelog-9.2.xml
  89. +1 −1 modules/dcache-qos/src/main/resources/org/dcache/qos/model/db.changelog-master.xml
  90. +1 −1 modules/dcache-qos/src/main/resources/org/dcache/qos/qos-adjuster.xml
  91. +1 −1 modules/dcache-resilience/pom.xml
  92. +2 −1 modules/dcache-resilience/src/main/java/org/dcache/resilience/handlers/FileOperationHandler.java
  93. +1 −1 modules/dcache-spacemanager/pom.xml
  94. +1 −1 ...he-spacemanager/src/main/resources/diskCacheV111/services/space/db/spacemanager.changelog-2.8.xml
  95. +7 −7 ...he-spacemanager/src/main/resources/diskCacheV111/services/space/db/spacemanager.changelog-2.9.xml
  96. +1 −1 ...spacemanager/src/main/resources/diskCacheV111/services/space/db/spacemanager.changelog-master.xml
  97. +1 −1 modules/dcache-srm/pom.xml
  98. +3 −2 modules/dcache-srm/src/main/java/diskCacheV111/srm/dcache/Storage.java
  99. +1 −1 modules/dcache-vehicles/pom.xml
  100. +14 −0 modules/dcache-vehicles/src/main/java/diskCacheV111/util/DiskErrorCacheException.java
  101. +35 −7 modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/HttpProtocolInfo.java
  102. +40 −8 modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpDataTransferProtocolInfo.java
  103. +28 −8 ...les/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpsDataTransferProtocolInfo.java
  104. +8 −1 modules/dcache-vehicles/src/main/java/org/dcache/util/FileAttributesBuilder.java
  105. +1 −1 modules/dcache-vehicles/src/main/java/org/dcache/vehicles/FileAttributes.java
  106. +1 −1 modules/dcache-webdav/pom.xml
  107. +4 −1 modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java
  108. +8 −3 modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java
  109. +1 −1 modules/dcache-xrootd/pom.xml
  110. +3 −1 modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/NettyXrootdServer.java
  111. +11 −4 modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdRedirectHandler.java
  112. +70 −0 modules/dcache/README.md
  113. +1 −5 modules/dcache/pom.xml
  114. +2 −1 modules/dcache/src/main/java/diskCacheV111/doors/CopyManager.java
  115. +16 −16 modules/dcache/src/main/java/diskCacheV111/poolManager/HttpPoolMgrEngineV3.java
  116. +1 −1 modules/dcache/src/main/java/diskCacheV111/poolManager/PoolSelectionUnitV2.java
  117. +14 −1 modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java
  118. +1 −1 modules/dcache/src/main/java/diskCacheV111/util/HTMLWriter.java
  119. +1 −1 modules/dcache/src/main/java/org/dcache/http/HttpPoolRequestHandler.java
  120. +22 −2 modules/dcache/src/main/java/org/dcache/pool/classic/MoverRequestScheduler.java
  121. +29 −17 modules/dcache/src/main/java/org/dcache/pool/migration/Job.java
  122. +9 −1 modules/dcache/src/main/java/org/dcache/pool/migration/JobDefinition.java
  123. +12 −1 modules/dcache/src/main/java/org/dcache/pool/migration/MigrationModule.java
  124. +14 −0 modules/dcache/src/main/java/org/dcache/pool/migration/Task.java
  125. +1 −0 modules/dcache/src/main/java/org/dcache/pool/migration/TaskCompletionHandler.java
  126. +7 −1 modules/dcache/src/main/java/org/dcache/pool/migration/TaskParameters.java
  127. +1 −1 modules/dcache/src/main/java/org/dcache/pool/movers/RemoteHttpDataTransferProtocol.java
  128. +11 −1 modules/dcache/src/main/java/org/dcache/pool/repository/json/RepositoryData.java
  129. +46 −6 modules/dcache/src/main/java/org/dcache/pool/repository/v5/ReplicaRepository.java
  130. +4 −4 modules/dcache/src/main/java/org/dcache/poolmanager/WRandomPartition.java
  131. +0 −110 modules/dcache/src/main/java/org/dcache/util/SpringLiquibase.java
  132. +80 −2 modules/dcache/src/main/java/org/dcache/util/Transfer.java
  133. +1 −1 modules/dcache/src/main/resources/org/dcache/alarms/db/sql/alarms.changelog-2.6.1.xml
  134. +1 −1 modules/dcache/src/main/resources/org/dcache/alarms/db/sql/alarms.changelog-master.xml
  135. +1 −1 modules/dcache/src/main/resources/org/dcache/pinmanager/model/db.changelog-2.16.xml
  136. +1 −1 modules/dcache/src/main/resources/org/dcache/pinmanager/model/db.changelog-master.xml
  137. +8 −1 modules/dcache/src/main/resources/org/dcache/services/billing/cells/billing.xml
  138. +24 −0 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/00-billing.changelog-pre.xml
  139. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-1.9.13.xml
  140. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-2.15.xml
  141. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-2.16.xml
  142. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-2.2.8.xml
  143. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-2.4.xml
  144. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-2.6.xml
  145. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-3.2.xml
  146. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-5.2.xml
  147. +1 −1 modules/dcache/src/main/resources/org/dcache/services/billing/db/sql/billing.changelog-master.xml
  148. +1 −1 modules/dcache/src/main/resources/org/dcache/services/pinmanager1/changelog/db.changelog-master.xml
  149. +32 −1 modules/dcache/src/main/smc/org/dcache/pool/migration/Task.sm
  150. +410 −0 modules/dcache/src/main/smc/org/dcache/pool/migration/Task_sm.dot
  151. BIN modules/dcache/src/main/smc/org/dcache/pool/migration/Task_sm.dot.png
  152. +72 −0 modules/dcache/src/test/java/diskCacheV111/poolManager/RequestContainerV5Test.java
  153. +1 −1 modules/ftp-client/pom.xml
  154. +1 −1 modules/gplazma2-alise/pom.xml
  155. +1 −1 modules/gplazma2-banfile/pom.xml
  156. +2 −2 modules/gplazma2-fermi/pom.xml
  157. +1 −1 modules/gplazma2-grid/pom.xml
  158. +1 −1 modules/gplazma2-htpasswd/pom.xml
  159. +1 −1 modules/gplazma2-jaas/pom.xml
  160. +1 −1 modules/gplazma2-kpwd/pom.xml
  161. +1 −1 modules/gplazma2-krb5/pom.xml
  162. +1 −1 modules/gplazma2-ldap/pom.xml
  163. +1 −1 modules/gplazma2-multimap/pom.xml
  164. +1 −1 modules/gplazma2-nis/pom.xml
  165. +1 −1 modules/gplazma2-nsswitch/pom.xml
  166. +1 −1 modules/gplazma2-oidc-te/pom.xml
  167. +1 −1 modules/gplazma2-oidc/pom.xml
  168. +1 −1 modules/gplazma2-omnisession/pom.xml
  169. +1 −1 modules/gplazma2-pyscript/pom.xml
  170. +1 −1 modules/gplazma2-roles/pom.xml
  171. +1 −1 modules/gplazma2-scitoken/pom.xml
  172. +1 −1 modules/gplazma2-voms/pom.xml
  173. +1 −1 modules/gplazma2/pom.xml
  174. +1 −1 modules/logback-console-config/pom.xml
  175. +1 −1 modules/logback-test-config/pom.xml
  176. +1 −1 modules/missingfiles-semsg/pom.xml
  177. +1 −1 modules/srm-client/pom.xml
  178. +1 −1 modules/srm-common/pom.xml
  179. +1 −1 modules/srm-server/pom.xml
  180. +2 −2 modules/srm-server/src/main/resources/org/dcache/srm/request/sql/srm.changelog-2.14.xml
  181. +1 −1 modules/srm-server/src/main/resources/org/dcache/srm/request/sql/srm.changelog-4.0.xml
  182. +6 −1 modules/srm-server/src/main/resources/org/dcache/srm/request/sql/srm.changelog-master.xml
  183. +1 −1 packages/fhs/pom.xml
  184. +2 −2 packages/pom.xml
  185. +1 −1 packages/system-test/pom.xml
  186. +1 −1 packages/tar/pom.xml
  187. +1 −1 packages/tar/src/main/container/Dockerfile
  188. +9 −1 packages/tar/src/main/container/run.sh
  189. +1 −1 plugins/hsqldb/pom.xml
  190. +1 −1 plugins/hsqldb/src/main/resources/org/dcache/hsqldb/changelog/billing-1.9.13.xml
  191. +1 −1 plugins/hsqldb/src/main/resources/org/dcache/hsqldb/changelog/billing-2.16.xml
  192. +2 −2 plugins/hsqldb/src/main/resources/org/dcache/hsqldb/changelog/billing-2.2.xml
  193. +1 −1 plugins/hsqldb/src/main/resources/org/dcache/hsqldb/changelog/billing-master.xml
  194. +1 −1 plugins/pom.xml
  195. +9 −18 pom.xml
  196. +3 −0 skel/share/defaults/billing.properties
  197. +1 −0 skel/share/defaults/chimera.properties
  198. +0 −1 skel/share/defaults/dcache.properties
  199. +1 −0 skel/share/defaults/nfs.properties
  200. +1 −0 skel/share/defaults/pnfsmanager.properties
  201. +131 −0 skel/share/httpd/static/images/dCache-logo.svg
  202. BIN skel/share/httpd/static/images/eagleredtrans.gif
  203. +24 −44 skel/share/httpd/static/index.html
  204. +4 −8 skel/share/httpd/static/styles/billing.css
  205. +156 −15 skel/share/httpd/static/styles/common.css
  206. +5 −69 skel/share/httpd/static/styles/poolinfo.css
  207. +6 −10 skel/share/httpd/static/styles/queueInfo.css
  208. +0 −6 skel/share/httpd/static/styles/restoreHandler.css
  209. +0 −51 skel/share/httpd/static/styles/usageInfo.css
  210. +1 −1 skel/share/lib/billing.sh
  211. +10 −3 skel/share/lib/database.sh
150 changes: 150 additions & 0 deletions .ci/generate-releases-xml.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
#!/bin/sh


SRM_RPM_NAME=`ls /home/marina/dcache/modules/srm-client/target/rpmbuild/RPMS/noarch/ | grep dcache-srmclient`
SRM_RPM_SUM=`md5sum /home/marina/dcache/modules/srm-client/target/rpmbuild/RPMS/noarch/$SRM_RPM_NAME | cut -d ' ' -f 1`
DEB_NAME=`ls /home/marina/dcache/packages/fhs/target/ | grep dcache`
DEB_SUM=`md5sum /home/marina/dcache/packages/fhs/target/$DEB_NAME | cut -d ' ' -f 1`
TAR_NAME=`ls /home/marina/dcache/packages/tar/target/ | grep dcache`
TAR_SUM=`md5sum /home/marina/dcache/packages/tar/target/$TAR_NAME | cut -d ' ' -f 1`
RPM_NAME=`ls /home/marina/dcache/packages/fhs/target/rpmbuild/RPMS/noarch/ | grep dcache`
RPM_SUM=`md5sum /home/marina/dcache/packages/fhs/target/rpmbuild/RPMS/noarch/$RPM_NAME | cut -d ' ' -f 1`
DATE=`date +"%Y.%m.%d"`

echo "| Download | Build date | md5 sum |"
echo "|:-----------|:-----------|----------|"
echo "| $RPM_NAME | $DATE | $RPM_SUM |"
echo "| $DEB_NAME | $DATE | $DEB_SUM |"
echo "| $TAR_NAME | $DATE | $TAR_SUM |"
echo "| $SRM_RPM_NAME | $DATE | $SRM_RPM_SUM |"

echo ; echo; echo
git log `git describe --tags --abbrev=0`...`git describe --tags --abbrev=0 HEAD^` --no-merges --format='[%h](https://github.com/dcache/dcache/commit/%H)%n: %s%n'



my_string="10.0.13"
before_last_dot=$(echo "$my_string" | rev | cut -d'.' -f2- | rev)
echo "$before_last_dot"



series=$(echo "$my_string" | rev | cut -d'.' -f2- | rev)
echo "TES series $series"


get_series() { # $1 dCache version
echo ${my_string%%.*([0-9])}
}

get_bugfix() { # $1 dCache version
echo ${my_string##*([0-9]).*([0-9]).}
}

echo "TEST 1 ${my_string##*([0-9]).*([0-9]).}"

echo "Test 2 ${my_string%%.*([0-9])}"

file="releases-$(echo $series).html"
echo "test html file name $file"

update_releases() { # $1 - dCache version
# old web pages
#[ -z ${date+x} ] || date_param="--stringparam date $date"
# xsltproc --stringparam version $1 $date_param --stringparam checksums-path "$(pwd)" $share/update-releases.xsl releases.xml > out.xml
# mv out.xml releases.xml

# new web pages
[ -z ${date} ] && date=$(date +%d.%m.%Y)
series=$(get_series $my_string)
bugfix=$(get_bugfix $my_string)
file="releases-$(echo $series).html"

echo "test file name $file"


# replace rec (= recent version, green highlighted) with even or odd
# if newest version is odd, the last version was even and vice versa
even_odd="odd"
mod=$((13 % 2))
[ $mod -eq "0" ] || even_odd="even"

tmp_file="$file.tmp"
touch $tmp_file
sed "s/rec/${even_odd}/" $file > $tmp_file

mv $tmp_file $file

echo "TEST $tmp_file"

echo $file

echo

# write new table entry on top
cat > releases.xml << EOF
<div class="test">
<style>
table.releases tbody tr.rec { background-color: #9f9; }
table.releases tbody tr.odd { background-color: #e8f6f8; }
table.releases tbody tr.even { background-color: #d0e5e8; }
</style>
<table border="1">
<thead>
<tr>
<th>Download</th>
<th>Rel. Date</th>
<th>md5 hash</th>
<th>Release Notes</th>
</tr>
</thead>
<!-- First Row -->
<tr class="rec" id="$my_string">
<td class="link">
<a href="/old/downloads/1.9/repo/$series/$DEB_NAME">
dCache $my_string (Debian package)
</a>
</td>
<td class="date">$DATE</td>
<td class="hash">$DEB_SUM</td>
<td class="notes" rowspan="3">
<a href="/old/downloads/1.9/release-notes-$series.shtml#release$my_string">
$my_string
</a>
</td>
</tr>
<!-- Second Row -->
<tr class="rec">
<td class="link">
<a href="/old/downloads/1.9/repo/$series/$RPM_NAME">
dCache $my_string (rpm)
</a>
</td>
<td class="date">$DATE</td>
<td class="hash">$RPM_SUM</td>
</tr>
<!-- Third Row -->
<tr class="rec">
<td class="link">
<a href="/old/downloads/1.9/repo/$series/$TAR_NAME">
dCache $my_string (tgz)
</a>
</td>
<td class="date">$DATE</td>
<td class="hash">$TAR_SUM</td>
</tr>
</table>
</div>
EOF

# Add old table entries below the new one
cat $file >> $tmp_file
mv $tmp_file $file
}

update_releases

33 changes: 17 additions & 16 deletions .github/workflows/mirror.yml
Original file line number Diff line number Diff line change
@@ -9,19 +9,20 @@ jobs:
to_gitlab:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Mirror + trigger CI
uses: SvanBoxel/gitlab-mirror-and-ci-action@0.6
with:
args: "https://gitlab.desy.de/dcache/dcache"
env:
FOLLOW_TAGS: "true"
FORCE_PUSH: "false"
GITLAB_HOSTNAME: "gitlab.desy.de"
GITLAB_USERNAME: "dcache"
GITLAB_PASSWORD: ${{ secrets.GITLAB_PASSWORD }}
GITLAB_PROJECT_ID: "21"
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
POLL_TIMEOUT: 60
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Mirror + trigger CI
uses: SvanBoxel/gitlab-mirror-and-ci-action@0.6
with:
args: "https://gitlab.desy.de/dcache/dcache"
env:
FOLLOW_TAGS: "true"
FORCE_PUSH: "false"
GITLAB_HOSTNAME: "gitlab.desy.de"
GITLAB_USERNAME: "dcache"
GITLAB_PASSWORD: ${{ secrets.GITLAB_PASSWORD }}
GITLAB_PROJECT_ID: "21"
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
POLL_TIMEOUT: 60
30 changes: 27 additions & 3 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
@@ -386,18 +386,42 @@ AI Generated Release Notes:
- curl -L -o chatgpt https://github.com/kardolus/chatgpt-cli/releases/latest/download/chatgpt-linux-amd64 && chmod +x chatgpt
- LAST_TAG=$(.ci/get-git-version.sh $CI_COMMIT_TAG)
- |-
git log $LAST_TAG..$CI_COMMIT_TAG | \
git log --grep=maven-release-plugin --invert-grep $LAST_TAG..$CI_COMMIT_TAG | \
OPENAI_API_KEY=$LLM_API_KEY \
OPENAI_URL=$LLM_API_ENDPOINT \
OPENAI_MODEL=$LLM_MODEL \
OPENAI_COMPLETIONS_PATH=$LLM_COMPLETIONS_PATH \
OPENAI_ROLE="You are a helpful tech writer working on release notes of the dCache project." \
OPENAI_ROLE="You are a helpful tech writer working on release notes of the dCache project, ignoring commits with messages starting with '[maven-release-plugin]'." \
./chatgpt "$LLM_PROMPT" | \
tee -a release-$CI_COMMIT_TAG.md
artifacts:
paths:
- release-*.md


#
# Prepare releases download table in html format
#
Release Notes Table:
image: almalinux:9-minimal
stage: upload
extends: .upload_rules
allow_failure: true
dependencies:
- sign_deb
- sign_rpm
- sign_srm_client_rpm
- tar
script:
- microdnf install -y git-core
- git fetch --refetch --all --tags
- .ci/generate-releases-xml.sh >> release-$CI_COMMIT_TAG.xml

#tee -a release-$CI_COMMIT_TAG.md
artifacts:
paths:
- release1-*.md

#
# prepare kubernetes env for the build
#
@@ -648,4 +672,4 @@ Run OIDC test:
- kubectl -n $K8S_NAMESPACE run oidc-tester --image=almalinux:9 --restart=Never --command -- sleep 3600
- while ! kubectl -n $K8S_NAMESPACE wait --for=condition=Ready pod oidc-tester; do sleep 1; done
- kubectl -n $K8S_NAMESPACE cp .ci/run-oidc-test.sh oidc-tester:/run-oidc-test.sh
- kubectl -n $K8S_NAMESPACE exec oidc-tester -- /bin/sh /run-oidc-test.sh
- kubectl -n $K8S_NAMESPACE exec oidc-tester -- /bin/sh /run-oidc-test.sh
2 changes: 1 addition & 1 deletion archetypes/dcache-nearline-plugin-archetype/pom.xml
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
<parent>
<groupId>org.dcache</groupId>
<artifactId>archetypes</artifactId>
<version>10.2.0-SNAPSHOT</version>
<version>11.0.0-SNAPSHOT</version>
</parent>

<artifactId>dcache-nearline-plugin-archetype</artifactId>
2 changes: 1 addition & 1 deletion archetypes/pom.xml
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@
<parent>
<groupId>org.dcache</groupId>
<artifactId>dcache-parent</artifactId>
<version>10.2.0-SNAPSHOT</version>
<version>11.0.0-SNAPSHOT</version>
</parent>

<artifactId>archetypes</artifactId>
2 changes: 1 addition & 1 deletion docs/TheBook/pom.xml
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@
<parent>
<groupId>org.dcache</groupId>
<artifactId>documentation</artifactId>
<version>10.2.0-SNAPSHOT</version>
<version>11.0.0-SNAPSHOT</version>
</parent>

<artifactId>TheBook</artifactId>
Loading