2.5.0.0 (Released Nov 12, 2020)

New features

Yugabyte Platform

• Azure Cloud integration for Yugabyte Platform (in beta):
  Yugabyte Platform is natively integrated with Azure cloud to simplify deploying, monitoring, and managing YugabyteDB deployments. This feature automates a number of operations including orchestration of instances, secure deployments, online software upgrades, and scheduled backups, as well as monitoring and alerting. (6094, 6020)

• Yugabyte Platform operations now allow promoting a Yugabyte TServer only node to run YB-Master and YB-TServer process. (5831)

Core database

• Enhanced multi-region capabilities with geo-partitioning and follower reads
  The YugabyteDB 2.5 release adds row-level geo-partitioning capabilities as well as follower reads to the extensive set of multi-region features that YugabyteDB already had.

• Enterprise-grade security features:
  Authentication using the highly secure SCRAM-SHA-256 is now supported to limit security risks from brute force attacks and sniffing, including LDAP support for better user management and the ability to audit all database operations.

• Table-level partitions
  This allows users to split what is logically one large table into smaller sub-tables, using the following types of table partitioning schemes that PostgreSQL supports: range partitioning, list partitioning, and hash partitioning. Read more about table partitioning in YugabyteDB.

• Event triggers
  This is now supported in addition to regular table-level triggers in YSQL. While regular triggers are attached to a single table and capture only DML events, event triggers are global to a particular database and are capable of capturing DDL events. The event-based trigger framework enables detecting changes made to the data, and automating any subsequent tasks that need to be performed, which are useful in a number of use cases such as implementing a consolidated, central audit table (2379)

• Simplified cluster administration:

• Online rebuild of indexes
  This is supported for both the YSQL and YCQL APIs. This means that new indexes can be added to tables with pre-existing data while concurrent updates are happening on the cluster. The online index rebuild process creates the newly added index in the background, and transactionally enables the index once the rebuild of all the data is completed. This feature allows flexibility of adding indexes as the application needs evolve to keep queries efficient.

• Cluster execution statistics and running queries
  This can be analyzed in detail, allowing administrators to gain insights into how the database is performing. The pg_stat_statements extension, which enables tracking execution statistics of all SQL statements executed by the cluster, is supported and enabled by default. Support for pg_stat_activity has also been added, which shows information related to the activity performed by each connection. Yet another useful feature in this category is the ability to view all the live queries being executed by the cluster at any point in time.

• Detailed query plan and execution analysis
  This can now be performed with commands such as EXPLAIN and EXPLAIN ANALYZE. These commands display the execution plan generated by the planner for a given SQL statement. The execution plan shows details for any SQL statement such as how tables will be scanned (plain sequential scan, index scan), what join algorithms will be used to fetch required rows from the different tables, etc.

Improvements

Yugabyte Platform

• Enhancements to on-premises deployment workflows:
  • Do not fail universe creation if cronjobs can't be created for on-premises. (5939)
  • Remove pre-provision script requirement for airgapped installations. (5929)
  • "Sudo passwordless" in on-premises cloud provider configuration toggle is renamed.
  • Allow yugabyte user to belong to other user groups in Linux. (5943)
  • Added a new "Advanced" section in on-prem cloud provider configuration which includes:
    • Use hostnames
    • Desired home directory
    • Node exporter settings
  • Improvements to installation of Prometheus Node Exporter utility workflow. (5926)
    • The node exporter option is now available in the cloud configuration under advanced settings.
    • Supports bringing your own node exporter user.
• Make YEDIS API optional for new Universes and no change in behavior of existing universes. (5207)
• UI/UX improvements for YB Platform:
  • Add visual feedback when backup or restore is initiated from modal. (5908)
  • Minor fixes to primary cluster widget text issue. (5988)
  • Show pre-provision script in UI for non-passwordless sudo on-premises provider. (5550)
  • Update backup target and backup pages. (5917)
• For YugabyteDB universes with replication factor (RF) > 3, change the default min_num replicas for even distribution of AZs across all regions. (5426)
• Added functionality to create IPv6-enabled universe in Kubernetes. (5309, 5235)

Core database

• Support for SQL/JSON Path Language (jsonb_path_query). (5408)
• Remove range-loop-analysis and fallthrough errors in our code. (5812)
• Incorrect index update if used expression result is still unchanged. (5898)
• As part of the tablet splitting feature:
  • Implemented cleanup of the tablet for which all replicas have been split for. (4929)
  • Compaction improvements (5523)
• Improve performance for sequences by using higher cache value by default. Controlled by a YB-TServer configuration flag --ysql_sequence_cache_minval. (6041)
• Added compatibility mode in the yb_backup.py script for YugabyteDB version earlier than v2.1.4. (5810)
• Stability improvements to core database - make exponential backoff on lagging RAFT followers send NOOP. (5527)
• Added use of separate metrics objects for RegularDB and IntentsDB. (5640)
• Improvements to 2DC (xCluster replication) by adding TLS support for cleaning up cdc_state table. (5905)

Bug fixes

Yugabyte Platform

• Fix for Universe disk usage shows up empty on the universe page. (5548)
• Fix on on-premises backup failures due to file owned by the root user. (6062)
• Fix for a bug where user operation to perform a change to nodes count by AZ was doing a full move. (5335)
• Fixes for Yugabyte Platform data backup script for Replicated-based installations
• Fixes to Client Certificate start time to use UTC during download. (6118)
• Fixes for migration if no access keys exist yet. (6099)
• Fix to resolve issues caused by starting a YB-TServer node when another YB-Master node is down in the universe. (5739)
• Use the correct disk mount point while calculating disk usage of logs. (5983)

Core database

• Fix for bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index columns. (5811)
• Improve fault tolerance of DDLs and reduce version mismatch errors in YSQL. (3979, 4360)
• Fixes to incorrect column-ids in the restored table if the original table was altered. (5958)
• Fixes timeout bug in Yugabyte Platform when there are read replicas. This fix will ignore read replica YB-TServers when running AreLeadersOnPreferredOnly. (6081)
• Fixes to restore of YSQL Backups after dropping and recreating a database. (5651)
• Fixes to a bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index columns. This is fixed by setting the read time for backfill write requests to be the safe time chosen for backfill read requests rather than the hard-coded time 50. (5811)
• Fixes to delete backups for TLS-enabled universes. (5980)

Known Issues

Yugabyte Platform

...

v2.2.5.0 (Released Dec 2, 2020)

Yugabyte Release Notes v2.2.5 Dec 2, 2020

Bug Fixes

Yugabyte Platform

• Fix for updating user profile when SMTP username or password is empty (6257)
• Fix for editing YugabyteDB Universe placement info logic (6250)
• Fixes to delete unnecessary Prometheus snapshot when backing up YB Platform data (6175)
• Fix for an issue where Client Certificates are not valid on download due to incorrect validity time (6118)
• Safeguard extra migration for pre-provisioning (6181)

Core Database

• Fixes to the logic for SSL write to continue retrying, as long as SSL_write produces new data to send (6266)
• Fix TServer crash with YEDIS workloads and snapshot restore (6338)
• Disable Rocksdb flush on all DeleteTablet calls (6353)
• Shutdown status resolver before destroying it in ResolveIntents (6170)

Known Issues

Yugabyte Platform

• Tasks page reports an incorrect status of failed backup (6210)

Platform Support Package Improvements (Only applicable to Non-Replicated installations)

Yugabyte Platform

• It is required to pull the latest platform-support-packages.tar.gz binaries for the fix related to issue (6175) - “Fixes to delete unnecessary Prometheus snapshot when backing up YB Platform data”

v2.2.4.0 (Released Oct 30, 2020)

Yugabyte Release Notes v2.2.4 Oct 30, 2020

New features

Yugabyte Platform

Yugabyte Platform operations now allow promoting a Yugabyte TServer only node to run Yugabyte Master and TServer process (5831)

Improvements

Yugabyte Platform

Enhancements to on-prem Cloud Provider configuration
When creating onprem universes, YW will automatically run provisioning if airgap is enabled now, instead of asking the users to run the pre-provision script. Users will need to run the script only when creating universes without passwordless sudo access.
Do not fail universe creation if cronjobs can't be created for on-prem (5939)
Remove pre-provision script requirement for air-gapped installations (5929)
"Sudo passwordless" in on-prem cloud provider configuration toggle is renamed to “Manually Provision Node”
Added a new "Advanced" section in on-prem cloud provider configuration which includes
Use hostnames
Desired home directory
Node exporter settings
Pre-provisioning of ‘yugabyte’ user is supported for on-prem Cloud Provider. (YB devops script won't run create_user if yugabyte user already exists) (6086)
Improvements to installation of Prometheus Node Exporter utility workflow (5926)
Prometheus Node exporter option is now available in the cloud configuration under advanced settings
Supports bringing your own node exporter user
UI/UX improvements for YB Platform
Add visual feedback when backup or restore is initiated from modal (5908)
Add new button for downloading root cert in Certificates page (5521)
Use the correct disk mount while calculating the disk usage of logs (5983)
Core Database
YSQL Support ROWS_PER_TRANSACTION option for COPY FROM stdin (6069)
Bug Fixes
Yugabyte Platform

[Replicated] Fixed an issue that would leave Replicated in a bad state if the network connection drops while uploading an airgap bundle, preventing the installation from proceeding. (https://release-notes.replicated.com/release-notes/2.49.0/)
Fix for the check if changing node count by AZ affects master node placement and requires a full move (5335)
Fix delete backup failure for TLS enabled universes (5980)
Universe disk usage shows up empty on the universe page (5548)
Fix for starting a YB TServer Node when another YB Master is down causes issues (5739)
Fix platform backup script for replicated installations (6130)

Core Database
[docdb] Fix a rare bug on restore which could clear more metadata than necessary (5788)
[docdb] Fix a rare race condition between transactions and snapshots that could lead to a deadlock (6170)

[YSQL] Fix an issue with setting bind address for YSQL metrics endpoint for IPv6 scenarios (4901)
[YCQL] Fix for bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index column(s) (5811)
[YCQL] Fix a bug with update and expression indexes, where an index row would get deleted if the column is updated without changing the indexed value (expression) (5898)
[YSQL] Fix OOM with \copy or COPY FROM stdin for large files (5603)
Known Issues
Yugabyte Platform
Client Certificates are not valid on download due to incorrect validity time (6118)
Tasks page reports incorrect status of failed backup (6210)

Platform Support Package Improvements
(Only applicable to Non-Replicated installations)
Yugabyte Platform
Fix for On-prem YB Platform backup fails due to file owned by root (6062)

v2.3.3.0 (Released October 26, 2020)

New features

Yugabyte Platform

• Introducing beta support for Azure IaaS orchestration in Yugabyte Platform cloud provider configuration. (6094, 6020)
• Yugabyte Platform operations now allow promoting a Yugabyte TServer only node to run YB-Master and YB-TServer process. (5831)

Core database

• [YSQL] Introducing support for Event Triggers that are global to a particular database and are capable of capturing DDL events. (2379)

Improvements

Yugabyte Platform

• Enhancements to on-premises deployment workflows:
  • Do not fail universe creation if cronjobs can't be created for on-premises. (5939)
  • Remove pre-provision script requirement for airgapped installations. (5929)
  • "Sudo passwordless" in on-premises cloud provider configuration toggle is renamed.
  • Allow yugabyte user to belong to other user groups in Linux. (5943)
  • Added a new "Advanced" section in on-prem cloud provider configuration which includes:
    • Use hostnames
    • Desired home directory
    • Node exporter settings
  • Improvements to installation of Prometheus Node Exporter utility workflow. (5926)
    • The node exporter option is now available in the cloud configuration under advanced settings.
    • Supports bringing your own node exporter user.
• Make YEDIS API optional for new Universes and no change in behavior of existing universes. (5207)
• UI/UX improvements for YB Platform:
  • Add visual feedback when backup or restore is initiated from modal. (5908)
  • Minor fixes to primary cluster widget text issue. (5988)
  • Show pre-provision script in UI for non-passwordless sudo on-premises provider. (5550)
  • Update backup target and backup pages. (5917)
• For YugabyteDB universes with replication factor (RF) > 3, change the default min_num replicas for even distribution of AZs across all regions. (5426)
• Added functionality to create IPv6-enabled universe in Kubernetes. (5309, 5235)

Core database

• Support for SQL/JSON Path Language (jsonb_path_query). (5408)
• Remove range-loop-analysis and fallthrough errors in our code. (5812)
• Incorrect index update if used expression result is still unchanged. (5898)
• Various improvements to RaftConsensus tests. (4548, 5912)
• As part of the tablet splitting feature:
  • Implemented cleanup of the tablet for which all replicas have been split for. (4929)
  • Compaction improvements (5523)
• Improve performance for sequences by using higher cache value by default. Controlled by a YB-TServer configuration flag --ysql_sequence_cache_minval. (6041)
• Added compatibility mode in the yb_backup.py script for YugabyteDB version earlier than v2.1.4. (5810)
• Stability improvements to core database - make exponential backoff on lagging RAFT followers send NOOP. (5527)
• Added use of separate metrics objects for RegularDB and IntentsDB. (5640)
• Improvements to 2DC (xCluster replication) by adding TLS support for cleaning up cdc_state table. (5905)

Bug fixes

Yugabyte Platform

• Fix for Universe disk usage shows up empty on the universe page. (5548)
• Fix on on-premises backup failures due to file owned by the root user. (6062)
• Fix for a bug where user operation to perform a change to nodes count by AZ was doing a full move. (5335)
• Fixes for Yugabyte Platform data backup script for Replicated-based installations
• Fixes to Client Certificate start time to use UTC during download. (6118)
• Fixes for migration if no access keys exist yet. (6099)
• Fix to resolve issues caused by starting a YB-TServer node when another YB-Master node is down in the universe. (5739)
• Use the correct disk mount point while calculating disk usage of logs. (5983)
• Fixes to delete backups for TLS-enabled universes. (5980)

Core database

• Fix for bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index columns. (5811)
• Improve fault tolerance of DDLs and reduce version mismatch errors in YSQL. (3979, 4360)
• Fixes to incorrect column-ids in the restored table if the original table was altered. (5958)
• Fixes timeout bug in Yugabyte Platform when there are read replicas. This fix will ignore read replica YB-TServers when running AreLeadersOnPreferredOnly. (6081)
• Fixes to restore of YSQL Backups after dropping and recreating a database. (5651)
• Fixes to a bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index columns. This is fixed by setting the read time for backfill write requests to be the safe time chosen for backfill read requests rather than the hard-coded time 50. (5811)

Known Issues

Yugabyte Platform

• Azure IaaS orchestration:
  • No pricing information provided. (5624)
  • No support for regions with zero availability zones (AZs). (5628)

v2.2.3.0 (Released September 30, 2020)

New features

Yugabyte Platform

• Enhancements to RBAC with a new “Backup Admin” role which allows all read-only permissions plus back up operations #5311
• Display table sizes in the Tables page #3581
• On-prem Cloud provider instances table list can be sorted and filtered #4757
• API for disabling auto-generation of database nodes certificate #5545
• Backup retention policy for auto-deletion of older backups #4493

Core database

• [YCQL] Introducing Audit Logging feature for YCQL operations #5887
• [YSQL] Support for ALTER COLUMN type that does not require on-disk changes #4424
• [YSQL] Support for Transactional batch size in COPY FROM command syntax #2855, #5453
• [YCQL] Finer grained permission control for TRUNCATE #5443
• [YSQL] Enable 2DC replication bootstrap for YSQL Tables #5601

Improvements

Yugabyte Platform

• Backup improvements
  • YCQL, YSQL, and YEDIS backups are split into different tabs on the backup UI screen
  • Back up operation creates a single restore object regardless of “transactional” field selection
  • Added a new field parallel thread for multi-threading backups
  • YSQL backups only support namespace level backup
  • Backup table list has been revamped with the new UX and shows more details - expiration time, duration, backup ty
  • Universe can be restored with a single action
  • Backup objects can be manually deleted
  • Scheduled backups cron expression shows the next run time in local browser time and the label displays that cron supports UTC only #4709
• Improve the Replication graph for xDC replication set up to display the graph and metric names cleanly #5429
• UI improvements for displaying On-prem instances
  • Instance ID is now optional and also shown in the nodes page #4760
  • “In Use” column has been renamed to “Universe Name”
  • Supports adding multiple instances in different rows instead of comma-separated entries
• Supports generating API tokens for SSO-enabled users
• Yugabyte Platform restart cancels any tasks that were in progress
• Remove sudo requirement in DB backup script #5440

Core database

• [DocDB] Removal of unreachable (dead) nodes from YugabyteDB UI #4759
• [DocDB] Improvements with selecting better default flags for Index backfill #5494
• [DocDB] Lower the leader load balancer parallelism for performance improvements #5461

Bug fixes

Yugabyte Platform

• [Platform] Security hardening: Added CSRF token support
• [Platform] Support for PingFederate OIDC SSO login for Non Replicated deployments
• [Platform] Allow on-prem nodes to be reused after releasing them from a universe #5703

Core Database

• [YCQL] Fix for CQL Index scans when ORDER BY columns exist in the table but not in chosen index #5690
• [YSQL] Avoids Pushdown down of UPDATE with RETURNING clause #5366
• [YSQL] Fix for UPDATE operation with partial and expression Indexes #4939
• [YSQL] OOM issue fix in COPY FROM query #2855
• [YSQL] OOM issue fix in COPY TO query #5205
• [DocDB] Crash fix for DNS-enabled Yugabyte Universes #5561
• [DocDB] Fix for in-memory state not getting updated correctly when using read-from-follower query semantics #1052
• [DocDB] For Yugabyte Universes using cross-cluster async replication (2DC), added RPC throttling on idle CDC tables to avoid wasting CPU #5472
• [DocDB] Fix for rare deadlock scenario when the node switches from Leader to Follower state of the tablet and there are pending operations on Leader side #5741
• [DocDB] Fix to ensure all Yugabyte Tablet Servers are accounted for when Master fails over #5501

Known Issues

Yugabyte Platform

• Universe disk usage shows up empty on the universe page #5548

Platform Support Package Improvements (Only applicable to Non-Replicated installations)

Yugabyte Platform

• Platform backup script needs to preserve attributes on copy from Prometheus snapshot #5612
• Platform backup fails if /opt/yugabyte/release directory does not exist #5615
• Platform backup script needs to stop Prometheus service during the restore of Prometheus data #5685

Note:

Prior to v2.0, YSQL was still in beta. Upon the release of v2.0, a backward-incompatible file format change was made for YSQL. For existing clusters running pre-2.0 release with YSQL enabled, you cannot upgrade to v2.0 or later. Instead, export your data from existing clusters and then import the data into a new cluster (v2.0 or later).

v2.3.2.0 (Released September 29, 2020)

YSQL

• Enable pg_stat_statements extension by default. #5750
• Clean up unused PgStatement objects on hot path after usage to avoid memory consumption. #5374
• Fix restarting writes (INSERT and UPDATE) on transaction conflicts and avoid surfacing transaction errors to user applications. #4291
• Enable batching of updates for YSQL procedures and functions. #5257

YCQL

• Add additional system queries (SELECT_LOCAL, SELECT_LOCAL_TOKENS, and SELECT_PEERS_TOKENS) for caching. #5753
• Fix crash for YCQL index scan when ORDER BY columns exist in table, but not in the chosen index, and raise an error. #5690

Core database

• Allow setting a custom replacement policy per table through yb_admin. #5368
• Do not clean up transaction that potentially was committed. #5789 #5783 #5755
• Fix CDC shutdown races. Moved thread pool shutdown before other CDC objects cleanup. Added explicit CDC poller clear in CDC consumer shutdown. #5583
• Fix preparer deadlock after switching from leader to follower. #5741
• Fix RaftGroupMetadata construction issues found using PVS Studio. #5578
• Change rlimit flags to have at-least semantics. Only change the process soft limit if it is smaller than the limit specified by the corresponding flag value. #5715
• Add extra logging when initializing syscatalog. #5708
• Set default webserver_interface to rpc_bind_address IP/host component. #5320
• Add environmental variables functionality to yugabyted. New env variables for YSQL (YSQL_USER, YSQL_PASSWORD, and YSQL_DB) and YCQL(YCQL_USER, YCQL_PASSWORD, and YCQL_KEYSPACE). (#5499
• Improve cleanup of aborted transactions. #5669
• Fix dumping an empty batch by updating MaxReplicateOpId. #5679
• Set default staleness for read from followers (using max_stale_read_time_ms) to 60 seconds. Previously, a follower could return requested data no matter how state the data is. #5590
• Calculate the placement UUID from Raft upon tserver registration. #5558
• Enable ignoring deleted tablets. #5122
• Remove unreachable nodes from UI (/tablet-servers page) after nodes are removed from the cluster. #4759

Yugabyte Platform

• Fix issue parsing multiple comma-separated addresses for rpc_bind_addresses. #5634
• Add functionality to delete old backups after a specified time period. In the Create Backup dialog, added Number of Days to Retain Backup field. #4493
• Combine the Platform and community Grafana dashboards. Grafana dashboard shows the node status and other panels correctly. #5707
• Sum table sizes across all nodes of a universe. #5745
• Fixes releasing OnPrem nodes should not delete them from the universe metadata. Fixes releasing a node should only attempt to shut down processes on that node. Nodes can be re-added using Node > Actions > Add flow. #5703 #5791 #5792
• Add additional property to MetricsPanel to handle width of Replication graph and increase height. Aggregates replication metrics values from all tservers into an average and max value graph across all tservers. #5429 #5613 #5632
• Update yb-client JAR version to 0.8.1. #5759
• Stop and restart relevant services during Yugabyte Platform backup. Backups stop and restart services. Restores stop and restart Yugabyte Platform and Prometheus, if necessary. #5685
• Authenticate API tokens even when using OAuth. This allows curl requests. #5587
• Fix Metrics tab and columns misalignment on table page. #5582
• Rename In Use column to Universe Name and show NOT USED if the node is not in use. #5713
• Do not generate node certificates for custom certificates. #5545
• Make instance name field optional for on-premises provider. #5234
• Fix button for Metrics tab to reset active and focus state when key is unpressed. #5620
• Add Docker- and VM-aware options to the backup script and will include Prometheus snapshots. #5120 #5615 #5612 #5614

Note:
Prior to v2.0, YSQL was still in beta. Upon release of v2.0, a backward-incompatible file format change was made for YSQL. For existing clusters running pre-2.0 release with YSQL enabled, you cannot upgrade to v2.0 or later. Instead, export your data from existing clusters and then import the data into a new cluster (v2.0 or later).

v2.3.1.0 (Released September 15, 2020)

Features and changes

YSQL

• Fixes OOM when using COPY <table> FROM <file> to load data from a large file to a table. #5453
• Enable 2DC replication bootstrap for YSQL tables. This allows the producer to know where to start replicating from. #5601
• Fix CREATE TABLE is 4-5x slower using Docker on Mac than not using Docker. Speeds up table creation by buffering writes to postgres system tables, caching pinned objects, and significantly reducing write RPC calls. #3503
• Roll back the catalog version made in commit 46f3701 so that 2.3 upgrades can proceed. #5408

Core database

• Quickly evict known unresponsive tablet servers from the tablet location cache. Applies only to follower reads. For example, when tablet servers are not replying to RPC calls or dead — not sending heartbeats to the master for 5 minutes. This could also happen after decommissioning nodes. #1052

Yugabyte Platform

• Add search input and data sorting to the on-premises instances table list. Click arrows next to column titles to sort. Use the Search form to search multiple columns. #4757
• Add the ability to change the user role (Admin, ReadOnly, or BackupAdmin) from the UI by an admin. Also, fix stale users list after creation or deletion of a user and disable Save buttons at Customer Profile tabs for ReadOnly users. #5311
• Before deleting a universe, stop the master/tserver processes and release the instance so that if started again, the processes are not still running. #4953

Note:

Prior to version 2.0, YSQL was still in beta. As a result, the 2.0 release included a backward-incompatible file format change for YSQL. If you have an existing cluster running releases earlier than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from your existing cluster and then import into a new cluster (v2.0 or later) to use existing data.

v2.3.0.0 (Released September 8, 2020)

YSQL

• Fix OOM when running large COPY TO statements by creating a new memory context for the loop over retrieved rows and resetting it after processing each row. #5205
• Fix OOM on file-sourced YB relations in the COPY FROM statement. Reset memory context regularly, per row, when memory is resettable and when rows are read from a file (not stdin). #5561
• Support transactional batch size for COPY FROM command with OOM fix. Batch sizes can be passed in with ROWS_PER_TRANSACTION in the COPY OPTION syntax. #2855 #5453
• For index backfill flags, use better default values. Set index_backfill_rpc_timeout_ms default from 60000 to 30000 and change backfill_index_timeout_grace_margin_ms default from 50 to 500. #5494
• Remove spurious error message "0A000: Alter table is not yet supported" from CREATE OR REPLACE VIEW. #5071
• Prevent consistency violations when a partitioned table has foreign key constraints due to erroneous classification as a single-row transaction. #5387
• Fix restore from a distributed backup fails for tables using SPLIT INTO without a primary key. #4993
• Fix the wrong result by avoiding pushdown of UPDATE statement with the RETURNING clause. #5366
• Improve error message when UPDATE changes partition so its clear what went wrong, fixed yb_pg_foreign_key pg_regress test for semantic merge conflict when updating primary keys and row-level partitioning. #659 #5179 #5310
• Allow the UPDATE statement to change primary key columns. #659
• Support row-level partitioning. #5179
• Block usage of TABLEGROUP with the SPLIT clause. For CREATE TABLE, usage is blocked in the grammar. For CREATE INDEX, if NO TABLEGROUP was provided, then presplitting for the index is allowed; otherwise, an error is issued. #5352
• Buffered operations may share single RPC with read operation. Reducing the number of RPC calls speeds up the CREATE TABLE statement. For example, the total number of RPC calls is dropped from 66 to 58 for a simple table like CREATE TABLE t(k INT PRIMARY KEY). #5177
• Add new ysqlsh describe metacommands for tablegroups: \dgr[+] [grpname] to describe tablegroups, \dgrt[+] [grpname] lists all tables/indexes within the specified tablegroup (or within all tablegroups if grpname is not specified), and \d <table_name> is modified to include tablegroup information in the footer, if any. #5088
• Add ALTER TABLEGROUP statements to support ALTER TABLEGROUP tablegroup_name RENAME TO ... and ALTER TABLEGROUP tablegroup_name OWNER TO .... Also changes pg_tablegroup entry corresponding to tablegroup_name to properly reflect new grpname and grpowner (if the user has proper permissions or ownership to issue the ALTER TABLEGROUP statement. #5249
• Add support for indexes to opt out of tablegroups (NO TABLEGROUPS) or select their own tablegroup (TABLEGROUP group_name). #5293
• For ysql_dump, enable serializable-deferrable mode by default. Add new --no-serializable-deferrable flag to disable the default mode. #5128
• Support ALTER COLUMN type for variants that don't require on-disk changes. Specifically only allowing ALTER COLUMN type for varch(n) and varbit(n). For example, changing column type from varchar(50) to varchar(255). #4424
• Enable USING clause in DELETE statement and FROM clause in UPDATE statement. #738 #5262
• Support GRANT/REVOKE/ALTER DEFAULT PRIVILEGES for tablegroups. #5087 #5160
• Implement tablegroup query layer changes. #4525
• Replace "Foreign Scan" with "Seq Scan" for EXPLAIN statement. #2076
• Fix initdb when index backfill is enabled. #5027
• Properly handle empty delete with backfill by sending appropriate messages to the client, including setting it skipped and rows_affected_count to 0. #5015
• DROP INDEX statement invalidates table cache entry for the index table but should invalidate the table cache entry for the indexed table. #4974
• yb-admin create_database_snapshot command should not require ysql. prefix for database name. #4991
• For non-prepared statements, optimize pg_statistic system table lookups and update debugging utilities. #5051
• Correctly show beta feature warnings by default. #5322

YCQL

• For WHERE clause in CREATE INDEX statement, return a Not supported error. #5363
• Fix TSAN issue in partition-aware policy for C++ driver 2.9.0-yb-8 (yugabyte/cassandra-cpp-driver). #1837
• Support YCQL backup for indexes based on JSON-attribute. #5198
• Correctly set release_version for system.peers queries. #5407
• Reject TRUNCATE operations when ycql_require_drop_privs_for_truncate flag is enabled. When enabled, DROP TABLE permission is required to truncate a table. Default is false. #5443
• Fix missing return statement in error case of the SetPagingState method in statement_params.cc. #5441
• Enable backfilling of transactional tables by default. #4708
• Fix ycqlsh should return failure when known that the create (unique) index has failed. #5161

Core database

• Fix core dump related to DNS resolution from cache for Kubernetes universes. #5561
• Fix yb-master fails to restart after errors on the first run. #5276
• Show better error message when using yugabyted and yb-master fails to start. #5304
• Disable ignoring deleted tablets on load by default. #5122
• [CDC] Improve CDC idle throttling logic to reduce high CPU utilization in clusters without workloads running. #5472
• For server_broadcast_addresses flag, provide default port if not specified. #2540
• [CDC] Fix CDC TSAN destructor warning. #4258
• Add API endpoint to download the root certificate file. #4957
• Do not load deleted tables and tablets into memory on startup. #5122
• Set the follower lag for leaders to 0 by resetting timestamp to the maximum value when a peer becomes a leader and when the peer loses leadership. #5502
• Flow keyspace information from yb-master to yb-tserver. #3020
• Implement meta cache lookups throttling to reduce unnecessary thrashing. #5434
• Fix rpcz/statements links in yb-tserver Web UI when pgsql_proxy_bind_address and cql_proxy_bind_address are 0.0.0.0. #4963
• DumpReplayStateToStrings should handle too many WAL entries scenario. Also, log lines only display fields critical for debugging and do not show customer-sensitive information. #5345
• Find the difference between the replica map and consensus state more quickly using map lookup. #5435
• Improve failover to a new master leader in the case of a network partition or a dead yb-tserver by ...

v2.2.2.0 (Released August 19, 2020)

YSQL

• Fix failed backup if restored table was deleted before restoration. #5274
• Newly elected YB-Master leader should pause before initiating load balancing. #5221
• Fix backfilling to better handle large indexed table tablets. #5031
• Fix DROP DATABASE statement should work with databases deleted on YB-Master. #4710
• For non-prepared statements, optimize pg_statistic system table lookups. #5051
• [CDC] Avoid periodic querying of the cdc_state table for xDC metrics if there are no replication streams enabled. #5173

YCQL

• Implement DNS cache to significantly reduce CPU loads due to a large number of DNS resolution requests (especially for YCQL connections). Adds dns_cache_expiration_ms flag (default is 1 minute). #5201
• Fixed incorrect names de-mangling in index creation from CatalogManager::ImportSnapshot(). #5157
• Fixed crashes when inserting literals containing newline characters. #5270
• Reuse CQL parser between processors to improve memory usage. Add a new cql_processors_limit flag to control processor allocation. #5057

Core database

• Fix yugabyted fails to start UI due to class binding failure. #5069
• Show hostnames in YB-Master and YB-TServer Admin UI when hostnames are specified in --webserver_interface, --rpc_bind_addresses, and --server_broadcast_addresses flags. #5002
• Skip tablets without intents during commits, for example, the case of an update of a non-existing row. #5321
• Fix log spew when applying an unknown transaction and release a mutex as soon as possible when the transaction is not found. #5315
• Fix snapshots cleanup when snapshots removed before restart. #5337
• Replace SCHECK with LOG(DFATAL) when checking for restart-safe timestamps in WAL entries. #5314
• Replace all CHECK in the load balancer code with SCHECK or RETURN_NOT_OK. #5182
• Implement DNS cache to significantly reduce CPU loads due to a large number of DNS resolution requests (especially for YCQL connections). Adds dns_cache_expiration_ms flag (default is 1 minute). #5201
• Avoid duplicate DNS requests when handling system.partitions table requests. #5225
• Fix leader load balancing can cause CHECK failures if stepdown task is pending on the next run. Sets the global leader balance threshold while allowing progress to be made across tables. Adds new load_balancer_max_concurrent_moves_per_table flag to limit number of leader moves per table. #5181 and #5021
• Set the async YBClient initialization future in TabletPeer constructor to ensure tablet bootstrap logic can resolve transaction statuses. #5215
• Handle write operation failures during tablet bootstrap. #5224
• Drop index upon failed backfill. #5144 #5161
• Fix WAL overwriting by new leader and replay of incorrect entries on tablet bootstrap. #5003 #3759 #4983
• Avoid taking a unique lock when starting a lookup request. #5059
• Set 2DC lag metrics to 0 if not the leader and if the replication is deleted. #5113
• Set the table to ALTERING state when fully_* is populated. #5139
• Not the leader errors should not cause a replica to be marked as failed. #5072
• Use the difference between follower's hybrid time and its safe time as a measure of staleness. #4868

Yugabyte Platform

• Add Master section below Tablet Server section in Metrics page. #5233
• Add rpc_connections_alive metrics for YSQL and YCQL APIs. #5223
• Fix restore payload when renaming table to include keyspace. Disable keyspace field when restoring universe backup. #5178
• Pass in ssh_user to air-gap provision script and add to the on-premise template. #5132
• Add the --recursive flag to AZCopy for multi-table restore. #5163
• Fix transactional backup with specified tables list by including keyspace in payload. #5149
• Fix undefine object property error when Prometheus is unavailable and navigating to the Replication tab. #5146
• Backups should take provider-level environment variables, including home directory. #5064
• Add hostname and display in the Nodes page along with node name and IP address. #4760
• Support the option of DNS names instead of IP addresses for nodes. Add option in Admin UI to choose between using hostnames or IP addresses for the on-premises provider. #4951 #4950
• Disable glob before running cleanup of old log files using zip_purge_yb_logs.sh. Fixes issue on Red Hat. #5169
• Fix Replication graph units and missing graph in Replication tab when metrics exist. #5423

Note:
Prior to version 2.0, YSQL was still in beta. As a result, the 2.0 release included a backward-incompatible file format change for YSQL. If you have an existing cluster running releases earlier than 2.0 with YSQL enabled, then you will not be able to upgrade to version 2.0+. Export from your existing cluster and then import into a new cluster (v2.0 or later) to use existing data.

v2.2.0.0 (Released July 15, 2020)

YSQL

Transactional distributed backups

YugabyteDB now supports distributed backup and restore of YSQL databases, including backup of all tables in a database. #1139 and #3849

Online index backfills

• YugabyteDB can now build indexes on non-empty tables while online, without failing other concurrent writes. When you add a new index to a table that is already populated with data, you can now use the YSQL CREATE INDEX statement to enable building these indexes in an online manner, without requiring downtime. For details on how the online backfill of indexes works, see the Online Index Backfill design document.
• Backfilling an index while online is disable by default. To enable online index backfilling, set the yb-tserver --ysql_disable_index_backfill flag to false when starting YB-TServers. For details on how this works, see Online Index Backfill.

Colocated tables

Database-level colocation for YSQL, which started as a beta feature in the 2.1 release, is now generally available in the 2.2 release. Traditional RDBMS modeling of parent-child relationships as foreign key constraints can lead to high-latency JOIN queries in a geo-distributed SQL database. This is because the tablets (or shards) containing the child rows might be hosted in nodes, availability zones, and regions different from the tablets containing the parent rows. By using colocated tables, you can let a single tablet be shared across all tables. Colocation can also be at the overall database level, where all tables of a single database are located in the same tablet and managed by the same Raft group. Note that tables that you do not want to reside in the overall database’s tablet because of the expectation of large data volume can override the feature at table creation time and hence get independent tablets for themselves.

What happens when the “colocation” tablet containing all the tables of a database becomes too large and starts impacting performance? Check out automatic tablet splitting [BETA].

Deferred constraints on foreign keys

Foreign keys in YSQL now support the DEFERRABLE INITIALLY IMMEDIATE and DEFERRABLE INITIALLY DEFERRED clauses. Work on deferring additional constraints, including those for primary keys, is in progress.

Application developers often declare constraints that their data must obey, leaving it to relational databases to enforce the rules. The end result is simpler application logic, lower error probability, and higher developer productivity. Automatic constraint enforcement is a powerful feature that should be leveraged whenever possible. There are times, however, when you need to temporarily defer enforcement. An example is during the data load of a relational schema where there are cyclic foreign key dependencies. Data migration tools usually defer the enforcement of foreign key dependencies to the end of a transaction by which data for all foreign keys would ideally be present. This should also allow YSQL to power Django apps.

yugabyted for single-node clusters

The yugabyted server is now out of beta for single-node deployments. New users can start using YugabyteDB without needing to understand the underlying architectures acts as a parent server, reducing the need to understand data management by YB-TServers and metadata management by YB-Masters.

See it in action by following the updated Quick start. For details, see yugabyted in the Reference section.

Automatic tablet splitting [BETA]

• YugabyteDB now supports automatic tablet splitting, resharding your data by changing the number of tablets at runtime. For details, see Tablet splitting and Automatic Resharding of Data with Tablet Splitting. #1004, #1461, and #1462
• To enable automatic tablet splitting, use the new yb-master --tablet_split_size_threshold_bytes flag to specify the size when tablets should split.

TPC-C benchmarking

New results are now available for benchmarking the performance of the YSQL API using the TPC-C suite. For the new TPC-C results and details on performing your own benchmark tests to evaluate YugabyteDB, see TPC-C.

Other notable changes

• Support pre-splitting using CREATE INDEX...SPLIT INTO for range-partitioned table indexes. For details, see Pre-splitting #4235
• Fix crash for nested SELECT statements that involve null pushdown on system tables. #4685
• Fix wrong sorting order in pre-split tables. #4651
• To help track down unoptimized (or "slow") queries, use the new yb-tserver --ysql_log_min_duration_statement. #4817
• Enhance the yb-admin list_tables command with optional flags for listing tables with database type (include_db_type), table ID (include_table_id), and table type (include_table_type). This command replaces the deprecated yb-admin list_tables_with_db_types command. #4546
• Add support for ALTER TABLE on colocated tables. #4293
• Improve logic for index delete permissions. #4980
• Add fast path option for index backfill when certain statements can create indexes without the unnecessary overhead from online schema migration (for example, CREATE TABLE with unique column constraint). Skip index backfill for unsupported statements, including DROP INDEX, CREATE UNIQUE INDEX ON, and index create in postgres nested DDL. [#4918]
• Suppress incomplete startup packet messages in YSQL logs. #4813
• Improve YSQL create namespace failure handling. #3979
• Add error messages to table schema version mismatch errors so they are more understandable. [#4810]
• Increase the default DDL operations timeout to 120 seconds to allow for multi-region deployments, where a CREATE DATABASE statement can take longer than one minute. #4762

YCQL

Transactional distributed backups

YugabyteDB supports distributed backup and restore of YCQL databases and tables. #1139 and #3849

Online index backfills

• YugabyteDB can now build indexes on non-empty tables while online, without failing other concurrent writes. When you add a new index to a table that is already populated with data, you can now use the YCQL CREATE INDEX statement to enable building these indexes in an online manner, without requiring downtime. For details on how the online backfill of indexes works, see the Online Index Backfill design document.
• In YCQL, backfilling an index while online is enabled by default. To disable, set the yb-tserver --ycql_disable_index_backfill flag to false when starting YB-TServers. Note: Do not use this flag in a production cluster yet. For details on how online index backfill works, see Online Index Backfill #2301 and #4708

Online schema changes for YCQL [BETA]

Most applications have a need to frequently evolve the database schema, while simultaneously ensuring zero downtime during those schema change ope...