2.1.0

Whispers 2.1.0 release notes

❌ Breaking changes ❌

❌ Arguments ❌

Several arguments have been modified and/or adapted to improve usability.

• Human readable output is shown in logs (2.1), -H and --human (2.0) are removed.

• Version can be shown with --version (2.1), -v (2.0) is removed.

• Extended help can be shown with --info (2.1), -i (2.0) is removed.

• Debug mode can be enabled with --debug (2.1), -d (2.0) is removed.

• Logs can be redirected to a file with --log log.txt (2.1), constant /tmp/whispers.log (2.0) is removed.

• Configuration template can be created with --init (2.1), --print_config (2.0) is removed.

❌ Logging ❌

Version 2.0: Opt-in logging for tracing execution flow, useful only for debugging. Results printed to stdout using print() as a JSON dict, one result per line. Enabling logging required adding the --log argument.

Version 2.1: Logging is used to alert identified secrets during execution with WARNING level. Results are written to stdout as a JSON list at the end. This improves results parseability as a JSON list, while maintaining live results display that was previously achieved by printing secrets as JSON one per line.

✅ New features ✅

✅ Results as JSON list ✅

To improve integration and downstream processing, Whispers now outputs results as a JSON list of dictionaries with all detected secrets together (2.1), instead of one JSON dictionary per line (2.0). This list is directly loadable and parsable as JSON.

2.0.6

This release addresses two reported bugs, as well as dependency updates.

• Install dataclasses only for Python 3.6 to avoid dependency conflicts with higher versions (Fixes Skyscanner/whispers#107)
• Improve hardcoded secret detection for values starting with $ (Fixes Skyscanner/whispers#108)

What's Changed

• Bump typing-extensions from 4.0.1 to 4.1.1 by @dependabot in #21
• Bump lxml from 4.7.1 to 4.8.0 by @dependabot in #22
• Update issue templates by @adeptex in #23
• Bump astroid from 2.9.3 to 2.10.0 by @dependabot in #24
• Bump wrapt from 1.13.3 to 1.14.0 by @dependabot in #25
• Bump astroid from 2.10.0 to 2.11.0 by @dependabot in #26
• Bump astroid from 2.11.0 to 2.11.1 by @dependabot in #27
• Bump astroid from 2.11.1 to 2.11.2 by @dependabot in #28
• Bump soupsieve from 2.3.1 to 2.3.2 by @dependabot in #29
• Bump beautifulsoup4 from 4.10.0 to 4.11.0 by @dependabot in #30
• Bump beautifulsoup4 from 4.11.0 to 4.11.1 by @dependabot in #31
• Bump soupsieve from 2.3.2 to 2.3.2.post1 by @dependabot in #32
• Bump typing-extensions from 4.1.1 to 4.2.0 by @dependabot in #33
• Bump astroid from 2.11.2 to 2.11.3 by @dependabot in #34
• Bump wrapt from 1.14.0 to 1.14.1 by @dependabot in #35
• 2.0.6 by @adeptex in #36

Full Changelog: 2.0.5...2.0.6

2.0.5

Summary

• Added Mozilla SOPS placeholder detection
• Added AWS Account Number detection based on AWS ARN

What's Changed

• Detection by @adeptex in #20

Full Changelog: 2.0.4...2.0.5

2.0.4

Summary

• Target file or directory CLI argument is now position independent. For example, these two now work the same:
  • whispers -G files -H tests/fixtures
  • whispers tests/fixtures -G files -H
• File inclusion and exclusion can now be specified with CLI args (not just in config file as before):
  • whispers -f '*.json,*.yml' tests/fixtures - include globs that match (ie: include only JSON & YML files)
  • whispers -F '.*\.(json|yml)' tests/fixtures - exclude regex that matches (ie: exclude all JSON & YML files)
• Output can now be printed in human readable format by using the CLI arg --human or -H for short.

What's Changed

• Bump astroid from 2.9.0 to 2.9.3 by @dependabot in #16
• Bump jellyfish from 0.8.9 to 0.9.0 by @dependabot in #17
• Modify stdout format by @adeptex in #18
• Surface files configuration by @adeptex in #19

Full Changelog: 2.0.3...2.0.4

2.0.3

• Update dependencies

What's Changed

• Bump lxml from 4.6.4 to 4.7.0 by @dependabot in #10
• Bump lxml from 4.7.0 to 4.7.1 by @dependabot in #11
• Bump lazy-object-proxy from 1.6.0 to 1.7.0 by @dependabot in #12
• Bump lazy-object-proxy from 1.7.0 to 1.7.1 by @dependabot in #13

Full Changelog: 2.0.2...2.0.3

2.0.2

• Added --print_config argument to facilitate configuration, ie: whispers --print_config > custom.yml. This allows quickly getting a working editable template.

2.0.1

• Fix setup.py metadata
• Fix release bugs

2.0.0

Release notes

💫 Licensing changes 💫

Version 1 was released under Apache License 2.0, which states that Licensed works, modifications, and larger works may be distributed under different terms and without source code.

Version 2 is released under GNU General Public License v3.0, which is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. 💪🧠

❌ Breaking changes ❌

❌ Integration ❌

In version 1, Python integration required multiple imports and a correctly-formatted list of values (ref).

In version 2, the integration is simplified to a single import and a string of CLI arguments. The following example illustrates current Python integration:

import whispers

args = (
  "-c whispers/config.yml "
  "-r apikey,aws-secret,password "
  "-s BLOCKER,CRITICAL,MAJOR "
  "tests/fixtures"
)

for secret in whispers.secrets(args):
  print(f"[{secret.file}:{secret.line}] {secret.key} = {secret.value}")

❌ File exclusion globs are now regex ❌

In version 1, the configuration file expected file exclusion specification to be a list of globs. Whispers would then resolve included globs, resolve excluded globs, and finally subtract the two lists to get applicable scope. The entire target directory tree would be traversed twice to compute applicable files (highly resource-intensive operation!)

In version 2, file exclusions are specified as regex. Instead of resolving globs, Whispers now uses the generator directly. Every file path received from the glob generator is now checked against the file exclusion regex to determine whether the file should be excluded on-the-fly.

This highly improves performance for cases where the target directory contains a large number of files. In version 2 the tree is traversed file by file, individually checking if the file path matches a pre-compiled exclusion regex. This decreases CPU, RAM and time needed to scan directories of potentially unlimited trees and depths.

❌ Rule specification format changes ❌

In version 1 the rules were defined as a dictionary with rule ID as the key and rule config as the value. This created awkward parsing practices and unintuitive code. For example:

npmrc: 
  description: Hardcoded .npmrc authToken
  message: .npmrc authToken
  severity: CRITICAL
  key:
    regex: ^npm authToken$
    ignorecase: False

In version 2 the rules are defined as a list of dictionaries. The rule ID now has its own id key inside the rule config definition. For example:

- id: npmrc
  description: Hardcoded .npmrc authToken
  message: .npmrc authToken
  severity: CRITICAL
  key:
    regex: ^npm authToken$
    ignorecase: False

If you have any custom rule definitions, you will have to adjust them for migrating to version 2.

❌ Output file format ❌

In version 1 the output file was written in YAML with awkward indexing, which made results not very usable.

In version 2 the same JSON output as stdout is written to the output file, making it easier to parse.

❌ Log file ❌

In version 1, whispers.log is always created in the same directory from which Whispers was executed. The log file remains after execution.

In version 2, the log file will not be created by default, unless explicitly enabled with an argument: whispers --log src. The log is only useful for reviewing exceptions and bugs, not for common usage. In addition, the log will now be written to /tmp/whispers.log (Posix) or %TEMP%\whispers.log (Windows) so that it does not interfere with analysis or permissions.

Together, --log and --debug arguments can now be used to investigate exceptions and bugs. Please submit a bug report if you find something unexpected!

❌ Removed support for dynamic languages ❌

In version 1 the following language files were parsed as text and checked for common variable declaration and assignment patterns:

• JavaScript
• Java
• Go
• PHP

It is not possible to parse these languages as Abstract Syntax Trees (ASTs) in Python. The initial attempt was to detect "low hanging fruit" by parsing the files as text instead. This lead to poor functional coverage, as well as a potentially false sense of security.

In version 2 the support for these dynamic languages is dropped. This allowed bringing unit test coverage up to 100%, and in this way ensuring result reliability and true security coverage. It is recommended to rely on AST-based parsing for dynamic languages for getting reliable results. Check out Semgrep!

Python3 remains fully supported in Whispers 2.

❌ Replace Levenshtein with Jaro-Winkler ❌

In version 1, python-Levenshtein was used for key-value similarity comparisons (similar config parameter). This library is written in Cython and requires additional dependencies for installing. This made it not easily compatible with Windows systems, because additional Visual Studio dependencies needed to be present before installing Whispers.

In version 2, Jaro-Winkler algorithm is used for similarity comparisons, using the jellyfish library, for improved approximate and phonetic string matching. As an additional effect, this change allows installing Whispers on Windows through pip without Visual Studio dependencies.

This change should have no effect and behave in a consistent manner. If you have rules that specifically rely on similar for key-value comparisons, these may need to be manually tuned.

🛠️ Improvements 🛠️

🛠️ Improved support for Windows and MacOS 🛠️

Whispers now runs on Linux, MacOS, and Windows. Install it from PyPI like so: pip3 install whispers.

🛠️ Secrets detection 🛠️

• Added support for Gradle and Maven credentials
• Improved private key detection
• Added known API key formats (GitGuardian)
• Added sensitive file extensions (tell_me_your_secrets)

🛠️ Include and Exclude by Rule and Severity 🛠️

You can now specify rules and severity levels that you want to include or exclude directly with CLI args:

Exclude sensitive files from results: whispers --xrules sensitive-files
Exclude MINOR level severity from results: whispers --xseverity MINOR

It is also possible to specify included and excluded rules and severity levels via config.yml. Custom rules can be added directly to the list using the following format:

exclude:
  files:
    - \.npmrc
    - .*coded.*
    - \.git/.*
  keys:
    - SECRET_VALUE_KEY
  values:
    - SECRET_VALUE_PLACEHOLDER
  rules:
    - password
    - privatekey
    - id: starks
      message: Whispers from the North
      severity: CRITICAL
      value:
        regex: (Aria|Ned) Stark
        ignorecase: True

exclude:
  severity:
    - MINOR

If you don't specify any rules, all built-in rules will be used be default. If you do, only those that you specify will be applicable. For a full list of available rules check whispers --info.

If you don't specify any severity, all built-in severity levels will be used be default - BLOCKER, CRITICAL, MAJOR, MINOR, INFO.
If you do, only those that you specify will be applicable.

🛠️ Rule severity changes 🛠️

The following rule severity levels were adjusted for better filtering and alerting on relevant results:

• sensitive-files (MINOR)
• dangerous-functions (MINOR)
• cors (MINOR)
• creditcard (MINOR)
• comment (INFO) rule ID renamed from comments to comment

✅ New features ✅

No new features were introduced in this release. The primary objective of the present release was to optimize currently implemented logic in order to make it easier to read, understand, and work with in general. This refactoring, along with the aforementioned breaking changes, have shown to increase scanning speed of up to 7-10 times (depending on conditions) in comparison with version 1. In addition, it allowed achieving 100% unit test coverage.

Other focus areas of version 2 were improving usability, like being able to easily filter in/out results, or not writing a log file by default; and making code more Pythonic, by using built-in features and dataclass models.

Complete list of arguments, rules, and severity levels can be found in whispers --info, along with documentation in README.md