CHANGES

CHANGES
=======

* Shift 3.0 (01/09/14)
    - First public release
    - Note that --no-offline is currently enabled by default due to
        DMF corruption issues found when releasing files immediately
        after they are copied

* Shift 3.1 (01/13/14)
    - Fixed bad metadata counts during --sync preventing dirs from completing
    - Removed some site-specific config that was mistakenly included in shiftc

* Shift 3.2 (02/07/14)
    - Added note about mcp/msum >= 1.76.7 since earlier versions do not
        support all required functionality
    - Added warning message when --no-check option specified
    - Fixed leftover temp files after --status
    - Fixed hung transfers with one but not both of --no-check/--no-preserve
    - Fixed not an array reference exception during some rsync scenarios
    - Fixed use of undefined value exception during some tar scenarios

* Shift 3.3 (10/15/14)
    - Added file name to fish error messages
    - Changed final file size checks to reread src size when dst size differs
    - Changed status totals to use total+ instead of - during initialization
    - Changed mkdir and dir chattr ops to be limited by --files setting
    - Fixed tar file renaming when multiple unsplit tar files created via stdin
    - Fixed remote tar extraction due to missing fields in shift-aux tar parsing
    - Fixed parsing of @ in remote addresses (user name still dropped however)
    - Fixed local lustre striping when stripes > 160 for lustre < 2.4
    - Fixed unescape of src for tar link validation
    - Fixed display of ssize in --stats when --verify not used
    - Fixed addition to meta file if --meta=0 specified
    - Fixed reversion of metadata corruption after abrupt termination
    - Fixed truncation of existing file during file corruption recovery
    - Fixed possible truncation of directory prefix during tar creation
    - Fixed writability check of remote files
    - Fixed hang when getting striping of link to fifo (due to lfs bug)

* Shift 3.4 (01/07/15)
    - Added support for GNU long names/links during tar creation
    - Added inclusion/exclusion of file names based on regular expressions
    - Added --io[rw] and --net[rw] options to throttle by read/write only
    - Added global throttling to throttle read/writes/both by fs/host/user
    - Added --sync-fast to synchronize files without verifying integrity
    - Added built-in metadata sync mechanism for improved scalability
    - Added estimated time until completion in status
    - Added ability to specify some transport options in manager config
    - Added background spawn of dmget/dmput to avoid intermittent hangs
    - Changed disk throttling based on high/low threshold for suspend/resume
    - Fixed inadvertent tar file renaming when transfer completes with errors
    - Fixed error parsing of mcp/msum based on coreutils 8.x
    - Fixed table of contents truncation during remote tar creation

* Shift 4.0 (07/23/15)
    - Note that metadata is not backward compatible with previous versions
    - Added backgroundable, parallelizable, and restartable initialization
    - Added --newer and --older options for incremental backups
    - Added --preallocate option to preallocate files below given sparsity
    - Added --no-verify to disable verification, which is now enabled by default
    - Added preservation of extended attributes during non-tar transfers
    - Added backoff to alternate transports/checksums during retries
    - Added error handling when transfer metadata dir can't be created
    - Added ability to use --state w/o --id for brief status of xfers in state
    - Added situational attribute handling for optimized transfer initialization
    - Added ability to specify default tar split size in manager config
    - Changed handling of embedded modules to use App::FatPacker
    - Changed detailed status to show source file during tar creation
    - Changed intra-source symlink dereferencing due to initialization changes
        (location of dereferenced links may differ depending on traversal order)
    - Changed status output to omit some completed transfers beyond threshold
    - Changed --meta manager option to require --id
    - Fixed verification of files ending in whitespace
    - Fixed time estimate to reflect mgr overhead, parallelism, and chattr speed
    - Fixed sum handling of names containing trailing carriage returns
    - Fixed handling of pathological umasks
    - Fixed overrun of metadata directories when linux subdir limit reached
    - Fixed automatic striping of tar files on lustre
    - Fixed tar extraction of files with trailing spaces
    - Fixed preservation of symlink times
    - Fixed --no-cron propagation across parallel clients
    - Fixed tar header validation of symlinks containing escaped characters
    - Fixed bbftp inability to handle vt character
    - Fixed rsync inability to handle cr/lf characters (due to --files-from)
    - Fixed rsync unexpected rename behavior when one file in --files-from list
    - Fixed fish handling of errors with linefeeds
    - Removed --sync-fast since it is now equivalent to --sync --no-verify
    - Removed file/dir meter during initialization since init now backgrounded

* Shift 5.0 (07/12/16)
    - Added better detection of lan transfers in stats
    - Added error handling for malformed tar headers
    - Added --buffer option to adjust buffer size used by underlying transports
    - Added --streams option to adjust tcp streams used by tcp-based transports
    - Added --window option to set tcp window size used by tcp-based transports
    - Added --ports option to set the remote ports used by tcp-based transports
    - Added --threads option to set number of threads used by local transports
    - Added --bandwidth option to set bandwidth for stream/window calculation
    - Added adjustment of tcp window/streams based on b/w heuristics and latency
    - Added ability to set small file breakeven points for local/lan/wan cases
    - Added minimum split setting to prevent file system overload with metadata
    - Added rescan of mesh keys between batches to pick up newly generated keys
    - Added support for bbcp as underlying transport
    - Added support for all remote transports to be used as local transports
    - Added manager setting for lustre default stripe count
    - Added multi-threading of single/multi-file batches to built-in transports
    - Added multi-threading of single/multi-file batches to built-in hashing
    - Added multi-threaded tcp-based remote transport based on fish protocol
    - Added support for --verify-fast during fish gets
    - Added output across transfers of all users when --status invoked as root
    - Added zero-padding to duration and estimated time when --status=pad used
    - Changed status emails to limit length of original command sent
    - Changed brief status so at least one completed transfer is always shown
    - Changed --encrypt to --secure, which also changes ssh cipher/mac selection
    - Changed dmput handling so -r is no longer used in automatic offlining
    - Changed help output into functional units
    - Changed external invocations to eliminate all extra shell processes
    - Changed client selection to use selection hook instead of random policy
    - Changed shift-aux sums so file issues are errors and not bad checksums
    - Changed handling of estimated completion to reflect actual operation rates
    - Changed extraction of tar files to remove relative path components
    - Changed --stats output to omit rows without non-empty values
    - Fixed gridftp support using unbuffer utility to interlace stderr/stdout
    - Fixed umask for root transfers so won't inadvertently expose files
    - Fixed status after --wait, which sometimes did not appear when redirected
    - Fixed getting/setting of acls and xattrs on symlinks
    - Fixed existence check of target path when using openssh 7.x
    - Fixed infinite loop in built-in hashing when source file shrinks
    - Fixed reported rate when operations report in after transfer stopped
    - Fixed improper dst truncation in some non-tar corruption recovery cases
    - Fixed infinite loop when extracting tar files less than 512 bytes
    - Fixed host/process failures due to dmgets on every command line file
    - Fixed --include/--exclude options to handle malformed regular expressions
    - Fixed truncation of built-in local copies when dst larger than src
    - Fixed toc file empty blocks of increasing size during split tar creation
    - Fixed built-in hashing chopping off range when file has backslash/newline
    - Fixed correction of corruption in multiple byte ranges during tar creation
    - Fixed distribution of clients to remote hosts when more clients than hosts
    - Fixed built-in transport detection when user $PATH is empty
    - Fixed sum file rename when no regular files during tar creation
    - Fixed -d with unwritable or trailing slash dst (bug report by J. Otey)
    - Fixed exception in fish protocol when input stream is invalid
    - Fixed transport selection order when first transport not suitable
    - Fixed crontab handling with csh variants
    - Fixed exception when fish input stream is unreadable
    - Fixed mkdir errors during parallelization in some scenarios
    - Fixed sum file rename when transfer w/o regular files grouped with regular
    - Fixed abort due to embedded use of Time::HiRes in some perl versions
    - Removed use of File::Copy for built-in local copies

* Shift 5.1 (12/21/17)
    - Added --plot to visualize transfer performance when piped to gnuplot
    - Added support for setting stripe size and stripe pool via --stripe
    - Added code location in fish/fish-tcp invalid protocol returns
    - Added colorized status using --status=color
    - Added total time throttled to --stats output
    - Added --no-recall to prevent automated dmget handling
    - Added ability to prevent restart using no_restart file in mgr meta dir
    - Added disk overrun protection when cp would exceed throttling threshold
    - Added support for wildcard expansion when args piped in via stdin
    - Added check of overwrite of same file at transfer initiation
    - Changed rates in --stats to ignore --sync transfers to prevent skewed max
    - Changed tar preallocation to allocate during first tar file operation
    - Fixed mgr sync when user has many transfers/many clients in one transfer
    - Fixed high estimated completion when few small files copied before large
    - Fixed --status/--stats in overflow dir when no transfers in parent dir
    - Fixed failed auth in remote --wait due to agent killed by child process
    - Fixed error handling in remote --wait to show error output instead of ref
    - Fixed collection of os type and perl version from non-origin hosts
    - Fixed backout from fish-tcp when client does not have threading support
    - Fixed duplication of built-in transport error messages in thread splits
    - Fixed empty file name in detailed status during tar renames
    - Fixed metadata recovery after host/process failures preventing retries
    - Fixed cleanup of temporary files during transfer and at completion
    - Fixed detailed status line wrap when one large file and some smaller files
    - Fixed stall due to repeated timeout when very large dir hierarchy
    - Fixed --stripe forcing nearest power of 2 instead of specified
    - Fixed overspawn of dmget processes causing process failures
    - Fixed incorrect dereferencing of dangling symlinks during tar creation
    - Fixed status of transfers that only have errors in final tar rename
    - Fixed exception when unable to open sum file
    - Fixed zero-length files when copying dangling symlinks to lustre
    - Fixed file system load recording for throttling during sum/cksum
    - Fixed aux/mgr taint sanitization of insecure env variables on some OS's
    - Fixed mgr exit delay and exit code during background sync
    - Fixed tar creation when mount point name differs during parallel find
    - Fixed tar corruption of dir entries with specific (rare) name format
        (IMPORTANT - SEE "BUGS")
    - Removed deprecated options

* Shift 5.2 (04/23/18)
    - Added ability to show stats in CSV format using --stats=csv
    - Fixed lfs setstripe calls to use -S instead of -s (lustre >= 2.3.0)
    - Fixed descent into directories matching --exclude during initialization
    - Fixed hash errors when remote source mapped to local file system

* Shift 6.0 (01/16/19)
    - Note that metadata is not backward compatible with previous versions
    - Added depth-first file stage processing using --pipeline
    - Added CSV history output using --history=csv
    - Added disablement of email status for states given in --no-mail
    - Added disablement of preservation for attributes given in --no-preserve
    - Added continuous real-time display of status using --monitor
    - Added error handling for manager disk exhaustion
    - Added overwrite of read-only files using --force
    - Added ability to ignore unrecoverable errors using --restart=ignore
    - Added faster get/set of external file attributes using shift-bin
    - Added plotting by client using --plot=client
    - Added alert state for items that must be externally checked/fixed
    - Added encrypted data streams to fish-tcp transport using --secure
    - Added better support for scp-style "user@" syntax
    - Added silent corruption detection for previously completed transfers
    - Added --last-sum option to query silent corruption database
    - Added fadvise of source after transfer when shift-bin available
    - Added per file dynamic striping via perl-based expressions
    - Added manager config for hash algorithm and hash leaf size
    - Added advanced --older/--newer expressions using atime/ctime/mtime
    - Added --doing manager option to see past/present doing logs
    - Added abort during tar creation if first split file already exists
    - Added same file detection across remote hosts
    - Changed --no-check option to --no-sanity
    - Changed manager --meta output to handle deeper data structures
    - Changed manager db_file config setting to mount_db
    - Changed dmget handling to use -a during tar creation to keep files online
    - Changed manager meta file format from base64 to yEnc
    - Changed manager log structured files to seekable compressed format
    - Changed manager doing logs to log structured format
    - Changed HMAC authentication from SHA1 to SHA512 in fish-tcp
    - Changed manager sync mechanism to better support multiple failovers
    - Changed estimated completion to use running avgs instead of elapsed time
    - Fixed mount detection for DMF file systems reported as dmapi
    - Fixed mount detection for lustre file systems with multiple MDS's
    - Fixed overwrite of symlinks when exist at destination
    - Fixed deprecated File::Glob usage on newer perls
    - Fixed Net::Ping invocations during latency measurements on redhat/centos
    - Fixed detection of automounted file systems
    - Fixed hang on PBS-controlled hosts during file system detection
    - Fixed --sync on DMF to copy file instead of summing when time/size match
    - Fixed DMF recalls during file overwrite when using built-in transports
    - Fixed incorrect striping when --stripe=0 given and transport is mcp
    - Fixed warning output in some --status=color scenarios
    - Fixed exception in detailed status when very large file with small files
    - Fixed race condition between check and creation of user metadata dir
    - Fixed tracking of corruption stats during --sync
    - Fixed stalls after host/process failures due to bad metadata recovery
    - Fixed throttling deadlock due to file not counting own preallocation
    - Fixed error detection in some gridftp scenarios
    - Removed manager lustre_default_stripe config setting
    - Removed manager min_split config setting
    - Removed previously allowed inverse options --preserve and --verify

* Shift 6.1 (02/07/19)
    - Added manager config to globally disable use of cron for restarts
    - Fixed corruption tracking causing negative sizes/rates in --status
    - Fixed inadvertent call of test shift-mgr during mgr synchronization
    - Removed warning message when --no-cron specified

* Shift 6.2 (06/24/19)
    - Added multi-threading of directory traversal
    - Added multi-threading of attr preservation when external programs invoked
    - Added --interval to dynamically adjust batch size around fixed frequency
    - Added use of gzip index files to increase metadata read performance
    - Added --no-silent to disable silent corruption detection
    - Added ability to escape multiple lines over stdin in shift-aux
    - Added syntax checking of numeric options that can take units
    - Changed --files and --size to be minimums instead of absolutes
    - Changed time remaining to more accurately reflect manager overhead
    - Fixed remote host selection that was running per file instead of per batch
    - Fixed detection of operations already performed by another client
    - Fixed slow processing of doing logs due to small backward read size
    - Fixed local/remote fadvise that had old initial implementation
    - Fixed --no-recall handling of remote files
    - Fixed possible exceptions during some thread joins
    - Fixed Net::Ping exceptions when Time::HiRes not available
    - Fixed exception when stat of remote file fails
    - Fixed use of setfacl when shift-bin available
    - Fixed run state when --restart=ignore ignores final operations
    - Fixed manager --doing=n so it retrieves nth doing in past
    - Fixed output of error table in stats due to tainted file names
    - Fixed option abbreviation so deprecated options don't invoke other options
    - Fixed double lustre striping during attribute preservation stage
    - Fixed high memory utilization during traversal of very large directories
    - Fixed stalls due to inadvertent newlines left in logs after restarts
    - Fixed preservation of lustre striping due to bad conditional

* Shift 6.3 (09/18/19)
    - Fixed erroneous crontab entries for extra clients when --clients > 1
    - Fixed slash prepended to relative paths during tar creation
    - Fixed Tie::DB_FileLock exceptions that can abort manager updates
    - Fixed tainted file names preventing manager synchronization
    - Fixed missing lock during writes of user mount db
    - Fixed use of dequeue_timed preventing directory traversal in older perls
    - Fixed double locks causing deadlock during manager synchronization
    - Fixed thread race condition resulting in zombie shift-aux chattr processes
    - Fixed rounding of lustre stripe count causing allocated stripes off by 1
    - Fixed potential noop in shift-aux chattr when thread creation fails
    - Fixed stalls due to deletion of local mount information during restarts
    - Fixed asymmetric fish-tcp performance during some wan scenarios
    - Fixed potential duplicate error messages in various transports
    - Fixed duplication of corrupt byte ranges resulting in negative rates/sizes
    - Fixed infinite retry of corruption rectification when no progress made
    - Fixed missing write error handling in fish and fish-tcp transports
    - Fixed metadata counts during tar creation renames with --restart=ignore
    - Fixed bad newline handling in shift-aux escape/unescape commands

* Shift 7.0 (10/04/19)
    - Note that metadata is not backward compatible with previous versions
    - Fixed vulnerability in root --stats/--status due to use of Storable
        when metadata directly accessible by users (bug report by J. Neff)
    - Fixed race condition causing permission denied during some remote mkdirs
    - Fixed built-in checksums when thread creation fails
    - Fixed chattrs when thread creation fails
    - Fixed leftover shift-bin processes due to unneeded chattr thread spawns
    - Fixed exception when setting binmode on remote file handles
    - Fixed local dmget calls during traversal when source is not local
    - Fixed duplicated source tar lines in dmget input files
    - Fixed deceptive thread creation errors during fish-tcp initialization

* Shift 7.1 (03/05/21)
    - Note that the shiftc included with 7.0 was the wrong version (still 6.3)
    - Added latency collection on windows systems
    - Added tcp buffer size collection on osx
    - Added periodic latency measurement to track network issues
    - Added support of \Q metacharacter in --include/--exclude
    - Added manager setting for maximum files per batch
    - Added manager setting for time between latency measurement
    - Added origin IP address to transfer metadata
    - Added save of manager exceptions to associated user debug file
    - Added --plot by file system
    - Changed tracking of built-in operations for stats output
    - Changed file metadata operations to report metadata rate
    - Changed corruption tracking to not count file sizes differing
    - Fixed globs while locating ssh agents to reduce system audit messages
    - Fixed potential for multiple simultaneous keepalive invocations
    - Fixed ephemeral port usage during remote bbftpd invocation
    - Fixed missing handling of --no-recall in some scenarios
    - Fixed accuracy of rsync rates during partial file transfers
    - Fixed failure to detect completion properly in some tar scenarios
    - Fixed parallelization due to overwrite of per user mount metadata
    - Fixed manager exception during read of corrupt/incomplete gzi files
    - Fixed handling of manager disk exhaustion to prevent finalized metadata
    - Fixed potential divide by zero exceptions during throttling
    - Fixed lustre handling in shift-bin to current lustre API

* Shift 8.0 (09/16/21)
    - Note that metadata is not backward compatible with previous versions
    - Added plots by tool, subnet, and batch size
    - Added detection of BeeGFS file systems
    - Changed plots from line to heatmap for improved scalability
    - Changed processing so batches only contain single operation type
    - Changed monitoring to support manager hosts across shared file system
    - Changed naming of built-in tools in --stats and new --plot by tool
    - Fixed detection of DMF file systems when using xdsm mount option
    - Fixed detection of GPFS file systems when mmlsmgr not user-accessible
    - Fixed --wait status retrieval when integrated with Mesh framework
    - Fixed shift-bin striping with lustre progressive layouts when --stripe=0
    - Fixed use of XS Data::MessagePack when embedded pure perl version differs
    - Fixed scalability of built-in yEnc encoding/decoding
    - Fixed unnecessary metadata traversal when showing detailed status
    - Fixed application of lustre striping expression to directories
    - Removed bbcp and gridftp support since reported fatal bugs never fixed

* Shift 8.1 (01/08/24)
    - Added silent corruption database search using --search with --last-sum 
    - Added custom lustre striping of directories using DR in expressions
    - Added alternative metadata locking for broken NFSv3 locks
    - Added debugging output when manager unable to map local/remote files
    - Changed embedded mesh handling to latest mesh version
    - Fixed fish-tcp --secure SSL handling with more recent IO::Socket::SSL
    - Fixed mount collection when file server is an alias to many IPs
    - Fixed handling of @kfi forms in lustre mount collection
    - Fixed remote mount point in GPFS mount collection
    - Fixed mcp default stripe handling overriding --stripe=0
    - Fixed handling of --mgr so no longer always overrides with default
    - Fixed inherited retry count of operations in run state during restart
    - Fixed clean of files associated with old monitoring processes
    - Fixed exception during silent corruption processing of some tar operations
    - Fixed escape of commas in email command line
    - Fixed synchronization of monitor files between managers
    - Fixed error handling of manager invocations

* Shift 8.2 (03/15/24)
    - Fixed collection and application of default acls in shift-bin
    - Added additional documentation in shiftc