v1.6.2 patch release for `geneos deploy`

Change Log

Version v1.6.2

Released 2023-06-14

Please report issues via github or the ITRS Community Forum.

v1.6.2 Fixes

• tools/geneos Fix late found bug with deploy and home directories

Release v1.6.1

Change Log

Version v1.6.1

Released 2023-06-13

Please report issues via github or the ITRS Community Forum.

v1.6.1 Changes

• pkg/geneos Changes

  Some API changes to support work on a project for reading Gateway configuration files. Existing structures used to write config files cannot co-exist and have been renamed with an "Out" suffix down to the SamplerOut level. While the old names should have been retained and the new API requirements used new names, it was decided that this is the more common use case in the future.

• tools/geneos Command updates

  The show command can now output an instance's own configuration file (for types of Netprobe and Gateway) and also try to produce a merged Gateway file using a modified command line with the Gateway -dump-xml command line option.

  A new instance flag autostart has been added, set to true for all types except ac2 which defaults to false. Documentation updtes to follow.

  The init demo command now detects if the user has a DISPLAY environment variable set and if so also installs an ac2 instance.

  The command command can now output the details in JSON format. This format is not quite compatible with the pkg/process Run and Batch functions, but the aim is to eventually merge the formats so that they can also share the implementation later.

v1.6.1 Fixes

• Minor ongoing changes

  While adding new features there is ongoing review work and refactoring of code.

v1.6.0

Change Log

Version v1.6.0

Released 2023-06-07

Please report issues via github or the ITRS Community Forum.

v1.6.0 Changes

• #116

  Added a new geneos deploy command that combines geneos init and geneos add but takes more care over existing installations and/or creating new directories.

  As part of this work all the geneos init command will prompt the user for a directory if none is given on the command line. If the command is run from a non-interactive parent (e.g. a pipe from the shell) then the prompt is skipped and the default directory is used.

  The geneos deploy command uses the same initialisation rules but reduces the number of options. The intended audience is around automation where the deployment scripts may not have the knowledge or logic to check for existing installations.

• #114

  For geneos instances that have both the default libpaths and an environment variable LD_LIBRARY_PATH configured these are now concatenated with libpaths always first.

• #117

  Based on user feedback all the Netprobe types have been merged under the netprobe/ directory in their respective plural names, e.g. netprobe/sans. Existing installations should continue to work unchanged but you can use the geneos migrate command to automatically merge the instance directories under netprobe/ including the update of configuration files.

• #97

  The Linux Active Console is now treated like any other component and instance. At the moment, if you issue a geneos start command then all instances including Active Console(s) will be run. In a future release we may add an autostart like flag that can prevent this and require a manual start using the full geneos start ac2 abcde syntax.

• pkg/process - New features

  New functions have been added to support the running of single processes and batches based on a Program struct. This is for running tasks loaded from a config file (typically YAML) for an ongoing project. The reason for not using existing external packages was the integration with other cordial tooling. This functionality is currently maturing and is very sparsely documented and subject to major changes.

• pkg/icp and pkg/gwhub - New APIs

  These two packages are the start of Go APIs for ITRS Capacity Planner and Gateway Hub, respectively. These should not yet be used and have been included to track progress over the next few releases.

v1.6.0 Fixes

• #126

  In the config package the Load() function would fail if used with a file format set bu other defaults and run in the same directory as the binary it ran in because viper would also try to load the bare-named program binary as a config file of the type given. The package now does it's own file name construction to avoid this.

  As a consequence of the work done around this fix to make the usage of options to Load() and Save() clearer some have changed names. Existing code that wants to use v1.6.0 will experience minor API breakage. The fixes are simple refactors, so no backward compatibility has been retained.

• #124

  The work done for #117 above meant that all templates are now located under netprobe/templates and both san and floating templates had the same name. The default templates now have updated root names, e.g. san.setup.xml.gotmpl, but existing configuration may need updating if the existing templates clash.

  To help users control which configuration files are created from templates, and which to use for instance start-up, a new instance parameter setup has been introduced for this. The defaults are netprobe.setup.xml and gateway.setup.xml for the two affected component types which means no change for existing users.

  So even though the new SAN template is san.setup.xml.gotmpl, for example, running geneos rebuild san will still result in a
  netprobe.setup.xml in the instance directory.

• #120

  While this report was a misunderstanding of the way to use Daemon() the comments have been updated to give better direction on how to use the pidfile io.Writer parameters.

Patch release with small fixes and some new features

Change Log

Version v1.5.2

Releases 2023/05/31

Please report issues via github or the ITRS Community Forum.

v1.5.2 Changes

• #102 - process package additions

• #109 - New tls create command

  Move the functions of --name etc from tls new to tls create to remove dependency on the Geneos home directory.

• #106 - Change directory for TLS root and signing certs and keys

• #97 - Start of Linux AC2 support. Not yet fully functional.

• #98 - Work done and then superceeded by #109 above.

v1.5.2 Fixes

• #111 - Fix gateway instance template ports

• #103 - Fix default web server log file name

Patch release to v1.5.0

Change Log

Version v1.5.1

Releases 2023/05/25

Please report issues via github or the ITRS Community Forum.

v1.5.1 Changes

• tools/geneos:

  • #85 - TLS Issues

    Added verification for certificates against existing root and
    signing certificates.

    Added an option to recreate root and signing certs from tls init.

  • #86 - New log option for stderr logs blocked main logs

    Reworked the way that logs are followed to allow both normal and
    stderr logs to be followed. This fixes start -l, restart -l and
    by implication init -l options. Added a --nostandard/-N option
    to logs to allow normal log to be surpressed when you only want
    the error logs.

  • #88 - Floating probe configuration template output malformed

    The defaults for new floating probes used sanname instead of
    floatingname as the parameter for the template rendering.

  • #90 - Variables in SANs

    The san template was not corrected updated for the new variable
    structure.

  • #43 - Minor error in disable

    In disable check if stopped and print error if not --force

v1.5.0

We are very happy to announce release v1.5.0 of cordial

Release Notes

Version v1.5.0

Released 2023/05/22

Please report issues via github or the ITRS Community Forum.

v1.5.0 - Highlights

This release brings numerous changes to the cordial tools, especially
the geneos utility. We have tried to improve the reliability and
usability of the utility through updated and improved documentation and
subsequent fixes and changes that arose from writing and checking that
documentation.

v1.5.0 - BREAKING CHANGES

• pkg/logger:

  • REMOVED. This simplistic internal logging package has been
    deprecated in favour of more mature solutions, such as
    zerolog.

• tools/geneos:

  • The way SAN instances handle gateway connections has been fixed to
    allow gateway represented as FDQNs or IP addresses. The old way
    resulted in a mess - viper defaults to dots ('.') as configuration
    item hierarchy delimiters and this caused issues. Most users with
    SANs should not notice any change, but if you see problems please
    check the san XML file and correct the gateways section as
    necessary. One way is to remove and re-set them using:

    geneos set san mySan -g gateway.example.com ...

    Running set will rewrite the configuration in the new format bu
    there is a chance that the previous configuration will continue to
    occupy settings. You may need to manually edit the instance
    configuration file san.json anyway.

  • Like the above any variables defined for either SAN or Gateway
    instances used to generate XML from templates will have not worked
    in a case sensitive manner to mirror how Geneos treats variable
    names. To fix this the internal format of the variables section
    has been updated to move the variable name from the configuration
    key to a separate structure as it's own value. Code has been added
    to automatically convert from the old format to the new when the
    configuration file is updated however there is no fix for the
    correction of variable case name being incorrect from previous
    configurations. Please review and adjust as necessary.

  • Support for running instances as other user accounts or under
    sudo has been deprecated. Security is hard, and the support
    for these was poorly implemented. A better way should be coming in
    a future release.

    This may mean that where users has configured netprobes to run as
    different users and have previously run sudo geneos start to let
    the program do the right thing will run into issues. Please be
    careful if any of your instances run as other users and do not run
    the geneos program with sudo. There is no additional
    checking/rejection of running under sudo or any other privilege
    escalation system so this is important!

v1.5.0 - Other Changes

• There has been a significant amount of refactoring and moving around
  of the code-base. Most of this should not be user visible, but some
  public APIs have changed. As with all major changes there may be
  problems that have not been caught in testing. Please report
  anything you see as either a github
  issue or via the
  ITRS Community Forum.

  There are too many changed to list them all in detail but specific
  ones worth mentioning include:

  • memguard
    support for protected memory. Credentials (passwords, TLS keys and
    so on) should now be handled as Enclaves (for plaintext or private
    keys) or as LockedBuffers (for ciphertexts of sensitive data).

    The config package includes new methods
    for handling configuration file data as Enclaves and LockedBuffers
    to try to reduce the amount of confidential data visible in the
    process.

    The changes are ongoing and, in addition to adding a layer of data
    security to cordial, an added benefit is the interception of
    memory use errors etc. If you see errors, panic etc. please report
    them as a github
    issue

  • A number of the previous package APIs have undergone review and
    changed as needed. In particular the
    config API has been through the wringer
    and if you have any code that relies on it from v1.4 or earlier
    then it will require changes. There are new functions, which is to
    be expected, but also some existing ones have been renamed or had
    their argument signatures changed. Please review the documentation
    to see what the methods and functions have become.

  • Credentials support. There is both general purpose and
    geneos specific support for the local
    storage of credentials. Passwords and other secrets "at rest" are
    stored in Geneos AES256 format using a key file that is initial
    auto-generated. To decode these passwords you must have both the
    key file (which is by default only user readable) and the
    credentials file. There should be support for other credentials
    types, such as OAuth style client secrets and tokens, in future
    releases. The username and the domain that the credentials
    apply to are not encrypted, by design. This is however subject to
    change in a future release.

    Credentials currently works with a free-text domain that matches a
    destination using a "longest match wins" search, e.g. for a URL
    this may be a full or partial domain name, and for Geneos
    component authentication, e.g. the REST command API, the domain is
    in the form gateway:NAME. Others will be added later, probably
    including TLS certificates and keys as well as SSH password and
    private keys.

  • Releases now include selected binaries with a semantic version
    suffix. The programs in cordial use the base name of the binary
    as a key to select which configuration files to load, so that
    renaming the binary will result in a different set of
    configuration file being used, automatically.

    To make life simpler, any version suffix is automatically stripped
    if, and only if, it matches the one used to build the binary. This
    means you can now download geneos-v.1.5.0 and use it without
    having to rename it (useful for initial testing of new releases).

• tools/geneos:

  • Extensive documentation restructuring and rewriting. This is still
    work in progress but largely complet. Built-in help text (shown
    with the help command or the --help/-h option) should now
    align much more closely with real functionality and the online
    documentation is now almost completely built from the same source.

  • Addition of subsystems to group commands.

  • Move aes and tls command sources to their subsystems.

  • Add host and package subsystems and create aliases for the
    original commands, e.g.

    • add host becomes host add
    • install becomes package install
    • etc.

  • The set user, show user etc. commands are now under single
    config subystem, e.g. geneos config set mykey=value

  • The set global and related commands have been deprecated.

  • The new package subsystem command pulls all Geneos release
    management into one place

  • New login and logout commands to manage credentials.

  • New ca3 and floating components for Collection Agent 3 and Floating
    Netprobes

• tools/dv2email:

  • This new utility can be run as a Geneos Action or Effect to
    capture one or more Dataviews and send as an email. The
    configuration is extensive and the layout and contents are
    completely configurable through the use of Go templates.

v1.5.0 - Bug Fixes

• tools/geneos:

  • Version checking of local release archives was broken because of
    overloading of a common function. This is now split and checking
    should work once again.

  • Most reported issues on github have been fixed.

v1.5.0 - To Do

• Documentation needs more work and refinement. The built-in help for
  almost all commands is now up-to-date but the init and tls
  subsystems need to be reviewed further and completed. This should be
  in a patch release soon.

• tools/geneos:

  • Local storage of encrypted passwords for remote SSH access needs
    documenting

v1.5.0 Beta 2 - fixes and documentation updates

This second beta addresses some issues found during testing - thanks to those who reported issues - and also incorporates a significant number of changes (bug fixes or enhancements, you decide) that came to light while updating the help documentation for most geneos program commands. Almost all the help test in geneos has been updated with just a small subset awaiting more edits.

The geneos help text should now more closely align with functionality and has also resulted in that functionality being improved and made more consistent. Online rendered version of these changes can be browsed here: https://github.com/ITRS-Group/cordial/blob/main/tools/geneos/docs/commands/geneos.md

The CHANGELOG.md file has not had much in the way of change yet, so is not being copied here.

One more note - if you download any of the binaries attached to this release directly then please remember to rename before use as new behaviour means that the executable file name influences the configuration files loaded at start-up. We are looking at how to allow embedded semantic versioning in the file name without affecting the "basename" of the program which may be done by final release.

Once again, thanks for your patience and feedback.

v1.5.0-beta - testing invited

This beta release of cordial v1.5.0 is quite extensive and brings a number of new features and breaking changes. In addition lots of code internals, especially in the geneos tool introduce further flexibility for the future and increased security of memory and credentials.

One new highlight is the dv2html program, perhaps probably better calleddv2email and it may well get a name change before release, which lets you snapshot and email Dataviews from a Geneos Action or Alert. The README file is very much out-of-date (see below) and instead please see the [dv2html.yaml](tools/geneos/dv2html.yaml] configuration file for more details during this test period.

Documentation has not had much of an update and this is where work will be focused before final release. Any bugs found will also be addressed where possible. The aim it to have a release within a week or so.

One other change worth noting is that the geneos and dv2html binaries will look for configuration files that match their base-names, so if you download the attached geneos-v1.5.0-beta and do not rename it then it will look for configuration files in ${HOME}/.config/geneos-v1.5.0-beta and so on. So, if you download either directly then rename them before use.

The (incomplete) changelog is below:

Version v1.5.0-beta - 2023/05/16

• BREAKING CHANGES

  • pkg/logger:

    • REMOVED. This simplistic internal logging package has been removed
      in favour of more mature solutions, such as zerolog.

  • tools/geneos:

    • The way SAN instances handle gateway connections has been fixed to
      allow gateway represented as FDQNs or IP addresses. The old way
      resulted in a mess - viper defaults to dots ('.') as configuration
      item hierarchy delimiters and this caused issues. Most users with
      SANs should not notice any change, but if you see problems please
      check the san XML file and correct the gateways section as
      necessary. The easiest way is probably to remove and re-set them
      using geneos set san XXX -g gateway.example.com ...

    • Similarly to the above any variables defined in either SAN or
      Gateway configurations for use in the XML templates will have not
      worked in a case sensitive manner to align with how Geneos does
      it. To fix this the format of the variables section has been
      changed to move the variable name from the configuration key to a
      separate structure as a value. Code has been added to
      automatically convert from the old format to the new when the
      configuration file is updated however there is no fix for the
      correction of variable name case being wrong. Please review and
      adjust as necessary.

    • Support for running under sudo or for an instances where the
      user is different to the user running the command has been
      deprecated. Security is hard, and the support for these was poorly
      implemented. A better way will be coming in a later release.

      This may mean that where users has configured netprobes to run as
      different users and have run sudo geneos start to let the tool
      do the right thing will sun into issues. Please be careful if any
      of your instances run as other users and so not run the geneos
      tool with sudo. There is no additional checking/rejection of
      running under sudo or any other privilege escalation system so
      this is important!

• Changes

  • There has been a large amount of refactoring and rebalancing of the
    code-base. Most of this should not be user visible, but some
    previous public APIs have changed. As with all major changes there
    may be problems that have no been caught in testing. Please report
    anything you see as either a github issue or via the ITRS Community
    Forum.

    Specific changes worth mentioning include:

    • memguard
      support for protected memory. Most credentials should now be
      handled as Enclaves (for plaintext or private keys) or as
      LockedBuffers (for ciphertexts of sensitive data).

      The changes are ongoing and in addition to adding a layer of data
      security to cordial an added benefit is the catching of memory
      misuse etc. If you see errors, panic etc. please report them!

    • A number of the previous package APIs have undergone review and
      refactoring. In particular the pkg/config API has been through
      the wringer and if you have any code that relies on it from v1.4
      and before then it will need updating. There are many new
      functions, which is normal, but also some older entry points have
      been renamed or had the argument signatures changed. Please review
      the documentation to see what the methods and functions have
      become.

    • Credentials support. There is both general purpose and geneos
      specific support for the local storage of credentials. At rest the
      passwords are stored in Geneos AES256 format with a key that is
      auto-generated if not found. To decode these passwords you must
      have both the key file (which is by default only user readable)
      and the credentials file. Future support for other credentials,
      such as OAuth style client secrets and tokens, will be
      forthcoming. The username and the domain the the credentials apply
      to are not encrypted, by design. This is however subject to
      change.

      The credentials support currently works with a plain test domain
      that is used to match the destination "longest match wins", e.g.
      for a URL this may be a full or partial domain name, and for
      Geneos component authentication, e.g. the REST command API, the
      domain is in the form gateway:NAME. More will be added later,
      including SSH password and private keys.

  • tools/geneos:

    • Move aes and tls to their own directories as new subsystems.
    • Add host and package subsystems and create aliases for
      original commands, e.g.
      • add host becomes host add
      • install becomes package install
      • etc.
    • The set user, show user etc. commands are now under single
      config sub-command, e.g. geneos config set mykey=value
    • The set global and related commands have been deprecated.
    • The new package subsystem command pulls all Geneos release
      management into one place
    • New login and logout commands to manage credentials.

  • tools/dv2html:

    • This new program can be run as an Action or Effect to capture a
      complete Dataview and send it as email. The configuration is
      extensive and the layout and contents are completely configurable
      through the use of Go templates.

• Fixes

  • tools/geneos:

    • Version checking of local release archives was broken because of
      overloading of a common function. This is now split and checking
      should work once again.

    • Many reported issues on github have been fixed.

• To Do:

  • tools/geneos:

    • Local storage of encrypted passwords for remote SSH access needs documenting

Version v1.4.4 - 2023/04/12

• Fixes

  • New Default expand option should NOT itself default to nil

Version v1.4.3 - 2023/04/12

NOTE: Later error found in the geneos tool, so please do not use this release.

• Fixes

  • tools/geneos: fix ps not showing open ports on systems with IPv6 enabled
  • tools/geneos: make ls and ps command flags more consistent
  • tools/geneos: add an -el8 runtime to docker images when built
  • tools/geneos: fix RHEL8/Centos8 download support for localhost using new SetStringMapString() method
  • pkg/config: add SetStringMapString() methods to support settings maps (which viper doesn't support until you write a file out and read it back)
  • tools/geneos: adjust the way we choose package version, convert "-el8" to "+el8" to satisfy semver ordering
  • tools/geneos: package version number are now prefixes only

• Changes

  • tools/geneos: add -f flag to ps command to show open files. formatting subject to change for now.
  • tools/geneos: add a update ls command to show available package versions
  • pkg/config: added more ExpandOptions and support more Get* functions such as GetInt
  • pkg/geneos: added more Geneos XML config support, specifically Sampler Schemas and Standardised Formatting
  • libraries/libemail: added initial msTeams notification function