Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

artifactCleanup - protect artifacts from being deleted matching a regular expression #399

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

artifactCleanup - protect artifacts from being deleted matching a regular expression #399

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 69 additions & 16 deletions cleanup/artifactCleanup/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ which is by default 1 month. It can be run manually from the REST API, or automa
Many delete operations can affect performance due to disk I/O occurring. A new parameter now allows a delay per delete operation. See below.

To ensure logging for this plugin, edit ${ARTIFACTORY_HOME}/etc/logback.xml to add:

```xml
<logger name="artifactCleanup">
<level value="info"/>
Expand All @@ -25,15 +26,10 @@ Parameters
- `timeInterval`: The time interval to look back before deleting an artifact. Default *1*.
- `repos`: A list of repositories to clean. This parameter is required.
- `dryRun`: If this parameter is passed, artifacts will not actually be deleted. Default *false*.
- `disablePropertiesSupport`: Disable the support of Artifactory Properties (see below *Artifactory Properties support* section). Default
- `paceTimeMS`: The number of milliseconds to delay between delete operations. Default *0*.
- `disablePropertiesSupport`: Disable the support of Artifactory Properties (see below *Artifactory Properties support* section). Default *false*.

Properties
----------

- `monthsUntil`: The number of months back to look before deleting an artifact.
- `repos`: A list of repositories to clean.
- `paceTimeMS`: The number of milliseconds to delay between delete operations.
- `keepArtifacts`: Enable protection of artifacts that should not be deleted. Default *false*.
- `keepArtifactsRegex`: Regular expression for protecting artifacts to not be deleted. Default `/.*-[\.\d+]*[-+][\.\d+]*\..*/`.

Artifactory Properties support
----------
Expand All @@ -42,22 +38,25 @@ Some Artifactory [Properties](https://www.jfrog.com/confluence/display/RTF/Prope

- `cleanup.skip`: Skip the artifact deletion if property defined on artifact's path ; artifact itself or in a parent folder(s).


`artifactCleanup.json`
----------

The json contains the policies for scheduling cleanup jobs for different repositories.

The following properties are supported by each policy descriptor object:

- `cron`: The cron time when the job is executed. Default `0 0 5 ? * 1` *should* be redefined.
- `repos`: The mandatory list of repositories the policies will be applied to.
- `timeUnit`: The unit of the time interval. *year*, *month*, *day*, *hour* or *minute* are allowed values. Default *month*.
- `timeInterval`: The time interval to look back before deleting an artifact. Default *1*.
- `dryRun`: If this parameter is passed, artifacts will not actually be deleted. Default *false*.
- `dryRun`: If this parameter is passed, artifacts will not actually be deleted. Default *false*.
- `paceTimeMS`: The number of milliseconds to delay between delete operations. Default *0*.
- `disablePropertiesSupport`: Disable the support of Artifactory Properties (see above *Artifactory Properties support* section).
- `keepArtifacts`: Enable protection of artifacts that should not be deleted. Default *false*.
- `keepArtifactsRegex`: Regular expression for protecting artifacts to not be deleted. Default `~/.*-[\.\d+]*[-+][\.\d+]*\..*/`.

An example file could contain the following json:

```json
{
"policies": [
Expand All @@ -70,20 +69,55 @@ An example file could contain the following json:
"timeInterval": 3,
"dryRun": true,
"paceTimeMS": 500,
"disablePropertiesSupport": true
"disablePropertiesSupport": true,
"keepArtifacts": true,
"keepArtifactsRegex": "~/.*-[.\\d+]+-\\d{2}\\.[1-4][\\.\\w+]*/"
}
]
}
```

The regular expression used in the above example prevents the plugin from deleting artifacts that contain a version X.Y.Z and release number based on year and quarter (eg. 21.3).

```bash
my-artifact-1.2.0-21.3
my-artifact-1.2.0-21.3.zip
my-artifact-1.2.0-21.3+4.5.6
my-artifact-1.2.0-21.3+4.5.6.zip
```

**Note**: If a deprecated `artifactCleanup.properties` is defined it will only be applied if no `artifactCleanup.json` is present.

`artifactCleanup.properties` (DEPRECATED)
----------

The properties file contains the policies for scheduling cleanup jobs for different repositories.

The following properties are supported by each policy descriptor object:

- `cron`: The cron time when the job is executed. Default `0 0 5 ? * 1` *should* be redefined.
- `repos`: The mandatory list of repositories the policies will be applied to.
- `months`: The number of months back to look before deleting an artifact. Default: *6*.
- `paceTimeMS`: The number of milliseconds to delay between delete operations. Default *0*.
- `dryRun`: If this parameter is passed, artifacts will not actually be deleted. Default *false*.
- `disablePropertiesSupport`: Disable the support of Artifactory Properties (see above *Artifactory Properties support* section).
- `keepArtifacts`: Enable protection of artifacts that should not be deleted. Default *false*.
- `keepArtifactsRegex`: Regular expression for protecting artifacts to not be deleted. Default `~/.*-[\.\d+]*[-+][\.\d+]*\..*/`.

An example file could contain the following policy:

```groovy
policies = [
[ "0 0 5 ? * 1", [ "libs-release-local" ], 3, 500, true, true, true, ~/.*-\d+\.\d+\.\d+\.*/ ],
]
```

Executing
---------

To execute the plugin:

For Artifactory 4.x:
For Artifactory 4.x:
`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanup?params=timeUnit=month|timeInterval=1|repos=libs-release-local|dryRun=true|paceTimeMS=2000|disablePropertiesSupport=true"`

For Artifactory 5.x or higher:
Expand All @@ -92,29 +126,48 @@ For Artifactory 5.x or higher:
For Artifactory 5.x or higher, using the depracted `months` parameter:
`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanup?params=months=1;repos=libs-release-local;dryRun=true;paceTimeMS=2000;disablePropertiesSupport=true"`

For Artifactory 5.x or higher, using the `keepArtifactsRegex` parameter:

In order to pass a regular expression to a POST request we need to encode the expression to an URL-encoded format. You can either do this by using this
Python3 one-liner:

```bash
python3 -c "import urllib.parse, sys; print(urllib.parse.quote_plus(sys.argv[1]))" "~/.*-[\.\d+]*[-+][\.\d+]*\..*/"
```

or use [www.urlencoder.org](https://www.urlencoder.org).

Once you have encoded the regular expression you can call the cleanup script:

```bash
curl -X POST -uadmin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanup?params=timeUnit=month;timeInterval=1;repos=libs-release-local;dryRun=true;paceTimeMS=2000;disablePropertiesSupport=true;keepArtifactsRegex=~%2F.%2A-%5B%5C.%5Cd%2B%5D%2A%5B-%2B%5D%5B%5C.%5Cd%2B%5D%2A%5C..%2A%2F"
```

Admin users and users inside the `cleaners` group can execute the plugin.

There is also ability to control the running script. The following operations can occur

Operation
---------

The plugin have 4 control options:
The plugin has 4 control options:

- `stop`: When detected, the loop deleting artifacts is exited and the script ends. Example:

`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=stop"`

- `pause`: Suspend operation. The thread continues to run with a 1 minute sleep for retest. Example:

`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=pause"`

- `resume`: Resume normal execution. Example:

`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=resume"`
- `adjustPaceTimeMS`: Modify the running delay factor by increasing/decreasing the delay value. Example:

- `adjustPaceTimeMS`: Modify the running delay factor by increasing/decreasing the delay value. Example:

For Artifactory 4.x
`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=adjustPaceTimeMS|value=-1000"`
`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=adjustPaceTimeMS|value=-1000"`

For Artifactory 5.x or higher:
`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=adjustPaceTimeMS;value=-1000"`
`curl -X POST -v -u admin:password "http://localhost:8080/artifactory/api/plugins/execute/cleanupCtl?params=command=adjustPaceTimeMS;value=-1000"`
73 changes: 52 additions & 21 deletions cleanup/artifactCleanup/artifactCleanup.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
* limitations under the License.
*/

import java.util.regex.Matcher
import java.util.regex.Pattern

import org.apache.commons.lang3.StringUtils
import org.artifactory.api.repo.exception.ItemNotFoundRuntimeException
import org.artifactory.exception.CancelException
Expand Down Expand Up @@ -47,15 +50,18 @@ class Global {
// curl -i -uadmin:password -X POST "http://localhost:8081/artifactory/api/plugins/execute/cleanupCtl?params=command=adjustPaceTimeMS;value=-1000"

def pluginGroup = 'cleaners'
regex = ~/.*-[\.\d+]*[-+][\.\d+]*\..*/

executions {
cleanup(groups: [pluginGroup]) { params ->
def timeUnit = params['timeUnit'] ? params['timeUnit'][0] as String : DEFAULT_TIME_UNIT
def timeInterval = params['timeInterval'] ? params['timeInterval'][0] as int : DEFAULT_TIME_INTERVAL
def repos = params['repos'] as String[]
def repos = params['repos'] ? params['repos'] as String[] : ["__none__"] as String[]
def dryRun = params['dryRun'] ? new Boolean(params['dryRun'][0]) : false
def disablePropertiesSupport = params['disablePropertiesSupport'] ? new Boolean(params['disablePropertiesSupport'][0]) : false
def paceTimeMS = params['paceTimeMS'] ? params['paceTimeMS'][0] as int : 0
def keepArtifacts = params['keepArtifacts'] ? new Boolean(params['keepArtifacts'][0]) : false
def keepArtifactsRegex = params['keepArtifactsRegex'] ? Pattern.compile(params['keepArtifactsRegex'][0]) : regex

// Enable fallback support for deprecated month parameter
if ( params['months'] && !params['timeInterval'] ) {
Expand All @@ -65,7 +71,7 @@ executions {
log.warn('Deprecated month parameter and the new timeInterval are used in parallel: month has been ignored.', properties)
}

artifactCleanup(timeUnit, timeInterval, repos, log, paceTimeMS, dryRun, disablePropertiesSupport)
artifactCleanup(timeUnit, timeInterval, repos, log, paceTimeMS, dryRun, disablePropertiesSupport, keepArtifacts, keepArtifactsRegex)
}

cleanupCtl(groups: [pluginGroup]) { params ->
Expand Down Expand Up @@ -113,11 +119,13 @@ if ( deprecatedConfigFile.exists() ) {
def paceTimeMS = policySettings[ 3 ] ? policySettings[ 3 ] as int : 0
def dryRun = policySettings[ 4 ] ? policySettings[ 4 ] as Boolean : false
def disablePropertiesSupport = policySettings[ 5 ] ? policySettings[ 5 ] as Boolean : false
def keepArtifacts = policySettings[ 6 ] ? policySettings[ 6 ] as Boolean : false
def keepArtifactsRegex = policySettings[ 7 ] ? policySettings[ 7 ] as Pattern : regex

jobs {
"scheduledCleanup_$count"(cron: cron) {
log.info "Policy settings for scheduled run at($cron): repo list($repos), timeUnit(month), timeInterval($months), paceTimeMS($paceTimeMS) dryrun($dryRun) disablePropertiesSupport($disablePropertiesSupport)"
artifactCleanup( "month", months, repos, log, paceTimeMS, dryRun, disablePropertiesSupport )
"artifactCleanupScheduled_$count"(cron: cron) {
log.info "Policy settings for scheduled run at($cron): repo list($repos), timeUnit(month), timeInterval($months), paceTimeMS($paceTimeMS), dryRun($dryRun), disablePropertiesSupport($disablePropertiesSupport), keepArtifacts($keepArtifacts), keepArtifactsRegex($keepArtifactsRegex)"
artifactCleanup( "month", months, repos, log, paceTimeMS, dryRun, disablePropertiesSupport, keepArtifacts, keepArtifactsRegex )
}
}
count++
Expand All @@ -128,7 +136,7 @@ if ( deprecatedConfigFile.exists() ) {
}

if ( configFile.exists() ) {

def config = new JsonSlurper().parse(configFile.toURL())
log.info "Schedule job policy list: $config.policies"

Expand All @@ -141,23 +149,25 @@ if ( configFile.exists() ) {
def paceTimeMS = policySettings.containsKey("paceTimeMS") ? policySettings.paceTimeMS as int : 0
def dryRun = policySettings.containsKey("dryRun") ? new Boolean(policySettings.dryRun) : false
def disablePropertiesSupport = policySettings.containsKey("disablePropertiesSupport") ? new Boolean(policySettings.disablePropertiesSupport) : false
def keepArtifacts = policySettings.containsKey("keepArtifacts") ? new Boolean(policySettings.keepArtifacts) : false
def keepArtifactsRegex = (policySettings.containsKey("keepArtifactsRegex") && policySettings.keepArtifactsRegex) ? Pattern.compile(policySettings.keepArtifactsRegex) : regex

jobs {
"scheduledCleanup_$count"(cron: cron) {
log.info "Policy settings for scheduled run at($cron): repo list($repos), timeUnit($timeUnit), timeInterval($timeInterval), paceTimeMS($paceTimeMS) dryrun($dryRun) disablePropertiesSupport($disablePropertiesSupport)"
artifactCleanup( timeUnit, timeInterval, repos, log, paceTimeMS, dryRun, disablePropertiesSupport )
"artifactCleanupScheduled_$count"(cron: cron) {
log.info "Policy settings for scheduled run at($cron): repo list($repos), timeUnit($timeUnit), timeInterval($timeInterval), paceTimeMS($paceTimeMS) dryRun($dryRun) disablePropertiesSupport($disablePropertiesSupport), keepArtifacts($keepArtifacts), keepArtifactsRegex($keepArtifactsRegex)"
artifactCleanup( timeUnit, timeInterval, repos, log, paceTimeMS, dryRun, disablePropertiesSupport, keepArtifacts, keepArtifactsRegex )
}
}
count++
}
}
}

if ( deprecatedConfigFile.exists() && configFile.exists() ) {
log.warn "The deprecated artifactCleanup.properties and the new artifactCleanup.json are defined in parallel. You should migrate the old file and remove it."
}

private def artifactCleanup(String timeUnit, int timeInterval, String[] repos, log, paceTimeMS, dryRun = false, disablePropertiesSupport = false) {
log.info "Starting artifact cleanup for repositories $repos, until $timeInterval ${timeUnit}s ago with pacing interval $paceTimeMS ms, dryrun: $dryRun, disablePropertiesSupport: $disablePropertiesSupport"
private def artifactCleanup(String timeUnit, int timeInterval, String[] repos, log, paceTimeMS, dryRun = false, disablePropertiesSupport = false, keepArtifacts = false, keepArtifactsRegex = regex) {
log.info "Starting artifact cleanup for repositories $repos, until $timeInterval ${timeUnit}s ago with pacing interval $paceTimeMS ms, dryrun: $dryRun, disablePropertiesSupport: $disablePropertiesSupport, keepArtifacts: $keepArtifacts, keepArtifactsRegex: $keepArtifactsRegex"

// Create Map(repo, paths) of skiped paths (or others properties supported in future ...)
def skip = [:]
Expand All @@ -175,8 +185,12 @@ private def artifactCleanup(String timeUnit, int timeInterval, String[] repos, l
Global.stopCleaning = false
int cntFoundArtifacts = 0
int cntNoDeletePermissions = 0
int cntNoDeleteRegexPermissions = 0
int cntRemovedArtifacts = 0
long bytesFound = 0
long bytesFoundWithNoDeletePermission = 0
long bytesFoundWithRegexProtection = 0
long bytesRemoved = 0
def artifactsCleanedUp = searches.artifactsNotDownloadedSince(calendarUntil, calendarUntil, repos)
artifactsCleanedUp.find {
try {
Expand Down Expand Up @@ -204,21 +218,31 @@ private def artifactCleanup(String timeUnit, int timeInterval, String[] repos, l
cntNoDeletePermissions++
}
if (dryRun) {
log.info "Found $it, $cntFoundArtifacts/$artifactsCleanedUp.size total $bytesFound bytes"
log.info "\t==> currentUser: ${security.currentUser().getUsername()}"
log.info "\t==> canDelete: ${security.canDelete(it)}"

log.debug "Found $it, $cntFoundArtifacts/$artifactsCleanedUp.size total $bytesFound bytes"
log.debug "\t==> currentUser: ${security.currentUser().getUsername()}"
log.debug "\t==> canDelete: ${security.canDelete(it)}"
if (!checkName(keepArtifacts, keepArtifactsRegex, it)) {
log.info "\t==> protected by regex: ${keepArtifactsRegex}"
}
} else {
if (security.canDelete(it)) {
log.info "Deleting $it, $cntFoundArtifacts/$artifactsCleanedUp.size total $bytesFound bytes"
if (security.canDelete(it) && (checkName(keepArtifacts, keepArtifactsRegex, it))) {
bytesRemoved += repositories.getItemInfo(it)?.getSize()
log.debug "Deleting $it, [$cntFoundArtifacts/$artifactsCleanedUp.size]. Removed $bytesRemoved of total $bytesFound bytes"
repositories.delete it
} else {
log.info "Can't delete $it (user ${security.currentUser().getUsername()} has no delete permissions), " +
log.debug "Can't delete $it (user ${security.currentUser().getUsername()} has no delete permissions), " +
"$cntFoundArtifacts/$artifactsCleanedUp.size total $bytesFound bytes"
if (security.canDelete(it)) {
// Because previous block was not processed, security.canDelete(it) AND/OR checkName is FALSE.
// I am checking if permissions to delete files are proper. If that's true, that means function checkName returned FALSE
// and artifact is protected by Regex.
bytesFoundWithRegexProtection += repositories.getItemInfo(it)?.getSize()
cntNoDeleteRegexPermissions++
}
}
}
} catch (ItemNotFoundRuntimeException ex) {
log.info "Failed to find $it, skipping"
log.debug "Failed to find $it, skipping"
}

def sleepTime = (Global.paceTimeMS > 0) ? Global.paceTimeMS : paceTimeMS
Expand All @@ -235,10 +259,13 @@ private def artifactCleanup(String timeUnit, int timeInterval, String[] repos, l
log.info "$cntNoDeletePermissions artifacts cannot be deleted due to lack of permissions ($bytesFoundWithNoDeletePermission bytes)"
}
} else {
log.info "Finished cleanup, deleting $cntFoundArtifacts artifacts that took up $bytesFound bytes"
cntRemovedArtifacts = cntFoundArtifacts - cntNoDeletePermissions - cntNoDeleteRegexPermissions
log.info "Finished cleanup $repos repositories, deleting $cntFoundArtifacts artifacts that took up $bytesFound bytes"
if (cntNoDeletePermissions > 0) {
log.info "$cntNoDeletePermissions artifacts could not be deleted due to lack of permissions ($bytesFoundWithNoDeletePermission bytes)"
}
log.info "and $cntNoDeleteRegexPermissions artifacts protected by regex $keepArtifactsRegex ($bytesFoundWithRegexProtection bytes)"
log.info "After this cleanup, $cntRemovedArtifacts artifacts ($bytesRemoved bytes) got removed from $repos repositories"
}
}

Expand Down Expand Up @@ -304,3 +331,7 @@ private def mapTimeUnitToCalendar (String timeUnit) {
throw new CancelException(errorMessage, 400)
}
}

private def checkName(keepArtifacts, keepArtifactsRegex, artifactName) {
return !keepArtifacts || !((artifactName =~ keepArtifactsRegex).count > 0)
}
Loading