MatrixOne-v2.0.2

Release date: January 25, 2025
MatrixOne version: v2.0.2

MatrixOne 2.0.2 introduces a series of improvements and bug fixes that enhance system performance, stability, and usability. Below are the major updates and new features.

Key Improvements

1. Point query performance optimization
2. Replay implementation optimization to reduce TN restart time
3. Support querying statistical information of subscribed tables
4. ETLMerge optimization
5. Optimize logtail reconnection to minimize impact on CN operations and reduce CN downtime
6. Implement json_set

Bugfix

1. Fix bugs related to full-text indexing
2. Resolve the issue of CN hanging during upgrades
3. Fix some operator-related bugs
4. Fix bugs related to high-concurrency DDL
5. Fix bugs in window functions

What's Changed

• cp to 2.0 "[Tech Request]: optimize filters on composite columns (#20695)" by @badboynt1 in #20703
• fix a bug for case when expression by @badboynt1 in #20710
• support json_set/json_insert/json_replace(#2.0-dev) by @YANGGMM in #20592
• Optimize analyze code 2.0 dev by @qingxinhome in #20638
• cp to 2.0 "optimize stats for runtime filters when table not flushed" by @badboynt1 in #20719
• Lock mo_databases for DDL/DML(2.0-dev) by @ouyuanning in #20617
• fileservice: more disk cache events logging by @reusee in #20706
• to 2.0: fix show accounts double counting size. by @gouhongshen in #20714
• fix bad write format error - 2.0 by @aptend in #20716
• [improvement] logtail: do not abort txns when reconnect. by @volgariver6 in #20732
• rm tag 20544(2.0-dev) by @Ariznawlll in #20709
• Blob supports maximum storage64 m data 2.0 dev by @qingxinhome in #20720
• [improvement] logservice: add some logs by @volgariver6 in #20704
• [improvement] stats: add some logs and UT by @volgariver6 in #20620
• fix stats when merge filters on composite key by @badboynt1 in #20749
• to 2.0: apply reader pk filter on workspace entry. by @gouhongshen in #20750
• fix tests by @zhangxu19830126 in #20746
• fix task concurrency control by @w-zr in #20757
• optimize operator of fill by @iamlinjunhong in #20723
• Refactor fault injection by @Wenbin1002 in #20523
• to 2.0: optimization of the reader Equal filter. by @gouhongshen in #20533
• Print index name when doing explain or explain analyze by @qingxinhome in #20741
• improve lock meta table for DML (#20763) by @ouyuanning in #20774
• fix get role name of mo admin role in non-sys account by @ck89119 in #20776
• optimize workspace dump memory usage by @Wenbin1002 in #20779
• Fix panic caused by resolve index table in subscription by @ck89119 in #20793
• remove some logic from multi_update (#20788) by @ouyuanning in #20795
• refine string by @daviszhen in #20798
• Cdc task auto detect db/tbl by @ck89119 in #20701
• align merge with main by @w-zr in #20488
• fileservice: drain response reader to ensure connection reuse by @reusee in #20792
• fix ReWriteCheckpointAndBlockFromKey by @jiangxinmeng1 in #20807
• [improvement] logtail: some refactor and add UTs by @volgariver6 in #20756
• refine bytes converting by @daviszhen in #20805
• add date to bytes by @daviszhen in #20814
• [improvement] clusterservice: at least one tn in the service. by @volgariver6 in #20811
• to 2.0: support subscription in mo table size/rows. by @gouhongshen in #20784
• fix a bug that cause panic on loop join by @badboynt1 in #20826
• fix merge start log by @w-zr in #20832
• cherrypick from 2.0.1hotfix to 2.0dev: use fixed function id by @daviszhen in #20830
• [improvement] proxy: filter duplicate set stmts. by @volgariver6 in #20829
• fix w-w conflict for add/drop FK (#20804) by @ouyuanning in #20836
• optimize workspace dump 2.0 by @Wenbin1002 in #20772
• composite date bytes by @daviszhen in #20823
• reset create node and delete node in replay by @jiangxinmeng1 in #20824
• add pitr checking by @ck89119 in #20838
• fix panic on 2.0-dev by @w-zr in #20841
• revert maximum value of max_allowed_packet to 1G by @ck89119 in #20851
• Locate sql parser error new 2.0 dev by @qingxinhome in #20758
• Revert check pitr in cdc by @ck89119 in #20854
• optimize replay by @jiangxinmeng1 in #20860
• fix 2.0-dev merge oom by @w-zr in #20861
• remove unused fields in s3writer(2.0-dev) by @ouyuanning in #20862
• TxnExecuter context missing roleId when executes the DDL statement by @qingxinhome in #20858
• Dynamic workspace threshold by @Wenbin1002 in #20848
• enable multiple merges on one table at once by @w-zr in #20847
• change code owner by @ouyuanning in #20875
• delete bvt tage #19912 by @aressu1985 in #20820
• 2.0 dev mo set json by @Songxx-7 in #20872
• Fix the bug that the file cannot be found when restoring using incremental backup-2.0 by @LeftHandCold in #20878
• [improvement] logservice: add truncate log by @volgariver6 in #20882
• refine convertslice by @daviszhen in #20863
• Fix data race for ParallelJobScheduler 2.0 by @LeftHandCold in #20885
• fail expired task by @w-zr in #20891
• fix big txn dump by @Wenbin1002 in #20900
• abort lock table when tn restart by @iamlinjunhong in #20914
• fix getChangedTableList panic caused by null requests. by @gouhongshen in #20906
• fix opt bug: not ResolveAlias for sk (#20903) by @ouyuanning in #20918
• make sure the default lock timeout is large enough to reduce sensitivity to 2.0 by @zhangxu19830126 in #20920
• Fix get snapshot panic for GC by @LeftHandCold in #20924
• Add stats io time consumed for stats 2.0 dev by @qingxinhome in #20800
• fix panic in on duplicate key update by @ouyuanning in #20884
• Remove GC's panic(), use Warn log instead: GC-PANIC by @LeftHandCold in #20910
• fileservice: more logs for memory cache updating by @reusee in #20889
• Update schema upgrade version to V2.0.2-2.0-dev by @qingxinhome in #20905
• [bug] fix ut TestSpeedupAbortAllTxn by @volgariver6 in #20928
• fix remote lock EOF and OOM when lock table bind by @iamlinjunhong in #20939
• [bug] logtail: remove "set ready" in init method by @volgariver6 in #20934
• fix a bug in apply index by @badboynt1 in #20946
• Revert "Cdc task auto detect db/tbl (#20701)" by @ck89119 in #20952
• Fix bug: eng is nil in multiupdate by @ouyuanning in #20961
• [bug] datasync: fix ut ...

v2.0.1-hotfix-20241211

Release date: December 12, 2024
MatrixOne version: v2.0.1-hotfix-20241211
MOI PROD Impact Scope: All CN Affected

This hotfix resolves compatibility issues in v2.0.1 to ensure improved stability and consistent functionality.

What's Changed

• Add debug log for 2.0.1 by @XuPeng-SH in #20729
• fix functionid by @daviszhen in #20743

Full Changelog: v2.0.1...v2.0.1-hotfix-20241211

MatrixOne-v2.0.1

Release date: December 10, 2024
MatrixOne version: v2.0.1

MatrixOne 2.0.1 introduces a series of improvements and bug fixes that enhance system performance, stability, and usability. Below are the major updates and new features.

Key Improvements

• Index and Query Optimization
• Memory and Resource Optimization
• Password Management
• Backup and Restore Enhancements
• Experimental support for full text search

Bugfix

This release addresses several critical bugs that improve the overall stability and reliability of the system. Some of the key bug fixes include:

• Transaction Management:

  • Fixed deadlocks and transaction state inconsistencies in distributed environments.
  • Resolved race conditions in transaction handling and session management.
  • Improved error handling for lock conflicts and concurrent delete operations.

• SQL Execution:

  • Fixed issues with recursive CTEs, view creation, and ON DUPLICATE KEY UPDATE statements.
  • Resolved problems with incorrect row counts and handling of compressed files during LOAD DATA.

• Logtail and Cache:

  • Addressed data race conditions and performance issues in logtail processing.
  • Fixed issues with cache usage and hit status monitoring.

• Full-Text Index:

  • Fixed panics related to full-text index creation and runtime filters.
  • Improved delete operations for tables with full-text indexes.

• Backup and Restore:

  • Fixed missing workspace data during the restore process.
  • Resolved errors when restoring tables or databases with special keywords or case-sensitive names.

• Performance and Stability:

  • Addressed memory allocation issues in batch processing and deduplication operations.
  • Resolved panics and race conditions in various modules, including query execution and system diagnostics.

Additionally, many other minor bug fixes were implemented to further enhance system reliability. For the full list of changes, refer to the What's Changed.

What's Changed

• Fix missing workspace data in restore process by @aptend in #19736
• uses CI release/2.0-dev for 2.0-dev by @Rosyrain in #19692
• refactor: improve checkTableDefChange by @huby2358 in #19681
• sync to 2.0: reduce the comparison during the reader primary key iterator scan by @gouhongshen in #19677
• optimize folding in EvalExpression(2.0-dev) by @zengyan1 in #19686
• [enhancement] proxy: reduce the interval time of refresh cn nodes by @volgariver6 in #19704
• Operator analyze counter set reuse2.0-dev by @qingxinhome in #19708
• abort all active txn on invalid service and make new txn can work to 2.0 by @zhangxu19830126 in #19749
• Fix test global checkpoint5 2.0 by @LeftHandCold in #19742
• fix lockservice dead lock with txn by @zhangxu19830126 in #19773
• fix dedup by @jiangxinmeng1 in #19775
• [bug] return retry error if snapshot ts is smaller than ts of catalog by @volgariver6 in #19780
• malloc: add IgnoreMunmapError by @reusee in #19782
• Password management policy to 2.0-dev by @YANGGMM in #19785
• fix group concat without data by @iamlinjunhong in #19778
• set sink mysql timeout to 10min by @ck89119 in #19791
• Fix TestAppendAndGC for 2.0 by @LeftHandCold in #19795
• handle index table missing table_kind identifier by @qingxinhome in #19796
• Backup adapts to 2.0 by @LeftHandCold in #19768
• Improve cdc send sql performance by @ck89119 in #19797
• fix: fetch result from correct resultset 2.0-dev by @xzxiong in #19809
• fix txn status invalid to 2.0 by @zhangxu19830126 in #19817
• fix ut by @ck89119 in #19836
• check user after open check switch by @YANGGMM in #19830
• fix transfer deletes by @jiangxinmeng1 in #19831
• [bug] logtail client: change the channel to buffered one by @volgariver6 in #19839
• Fix TestAppendAndGC2 by @LeftHandCold in #19824
• refactor: reduce memory alloc in makeBatchRows by @huby2358 in #19793
• refactor DML plan (cp #18406) by @aunjgr in #19848
• objectio: make metadata cache capacity dynamic adjustable by @reusee in #19829
• use customized driver by @ck89119 in #19849
• allow now function pushdown to block filters by @badboynt1 in #19855
• fix a bug cause remote run cannot stop immediately. by @m-schen in #19841
• fix Incorrect time unit for password lock check(#2.0-dev) by @YANGGMM in #19853
• fifocache: add context params to methods by @reusee in #19805
• [fix] : Instead of sending the value scan's batch all at once, split it up to reuse memory. by @jensenojs in #19845
• make ut for multi_update faster (#19861) by @ouyuanning in #19874
• [bug] proxy: fix UT by @volgariver6 in #19880
• metric: redefine duration buckets by @reusee in #19878
• to 2.0: fix the batchApplyTombstone got unsorted rowOffsets by @gouhongshen in #19870
• fix nested recursive cte by @iamlinjunhong in #19899
• [Bug]: use load_file() function to load stage data, mo report 'ERROR 20101 (HY000): internal error: Invalid alloc size 7775731712 by @cpegeric in #19905
• malloc: do not unmap to avoid VMA fragmentation by @reusee in #19886
• bug fix fulltext phrase search by @cpegeric in #19902
• add debug tool for remote run (#19833) by @ouyuanning in #19892
• feat: keep S3IOInput as 6-prec float value. by @xzxiong in #19865
• refactor: remove openSysTable 2.0 by @aptend in #19823
• Add metric for GC by @LeftHandCold in #19887
• cherry-pick optimize performance for filling hashkeys on string vectors (#19881) by @badboynt1 in #19918
• fix bug: sysbench delete pk will flush s3 by @ouyuanning in #19927
• support ip whitelist feat (#2.0-dev) by @YANGGMM in #19919
• cherry pick 'support pushdown cast expression to block filters for date time type' by @badboynt1 in #19938
• ignore bvt case 2.0-dev by @w-zr in #19935
• update log by @jiangxinmeng1 in #19932
• do some lock if lock no rows after lock_op by @ouyuanning in #19943
• [bug] clusterservice: ignore tn store which is expired by @volgariver6 in #19954
• support password_history and password_reuse_interval take effect separately(#2.0-dev) by @YANGGMM in #19950
• enable transfer in metric and cu table on 2.0-dev by @w-zr in #19916
• optimize workspace size by @Wenbin1002 in #19929
• cherry-pick: support observe cache usage and hit status (#19914) by @xzxiong in #19926
• add system variable profiling_history_size(#2.0-dev) by @YANGGMM in #19946
• cherry-pick: couple of updates by @daviszhen in #19968
• update logtail entry type by @jiangxinmeng1 in #19901
• [bug] fix ut, which raises "address already in use" error by @volgariver6 in #19957
• fix panic in fillStrhashmap by @badboynt1 in #19971
• fix create view with hint(#2.0-dev) by @YANGGMM in #19978
• fix bug: panic on create fulltext idx by @ouyuanning in #19975
• Fix TestBackupData2 for 2.0 by @LeftHandCold in #19986
• fix show index panic(#2.0-d...

v1.2.3-hotfix-20241101

What's Changed

• fix vector free panic(1.2.3-tmp-shulian) by @zengyan1 in #19680
• Fix alter table column null property-shulian by @qingxinhome in #19688
• fix decimal output by @daviszhen in #19643
• cherry-pick: fix select version by @daviszhen in #19695
• fix decimal panic(1.2.3-shulian) by @zengyan1 in #19716

Full Changelog: v1.2.3-hotfix-20240916...v1.2.3-hotfix-20241101

MatrixOne-v2.0.0

We are excited to announce the release of MatrixOne v24.2.0.0!

MatrixOne is a hyper-converged, cloud-native database built for modern data demands. Designed to deliver high performance, scalability, and MySQL compatibility, MatrixOne provides a seamless HTAP (Hybrid Transactional/Analytical Processing) experience, allowing users to efficiently handle transactions, analytics, time-series data, and streaming processing in one unified platform.

Highlights of this Release

MatrixOne v24.2.0.0 introduces significant enhancements for enterprise-level high availability, disaster recovery, and expanded AIGC support.
This version features robust improvements to support Generative AI applications, including unstructured data processing, full-text search, optimized vector search, and enhanced MySQL compatibility. Key features include external data access, snapshot backups, point-in-time recovery (PITR), CDC, and disaster recovery through log-based replication for primary-standby clusters. MatrixOne continues to evolve as a leading platform for intelligent, AI-driven data management, providing enterprises with the ideal solution for their data infrastructure needs.

Use Cases

MatrixOne is well-suited for the following application scenarios. We welcome users facing similar challenges to reach out and explore a trial deployment with us.

Generative AI Applications

MatrixOne’s hyper-converged architecture is ideal for Generative AI, offering comprehensive support for multimodal data, real-time data retrieval, and intelligent data processing. In text and image generation scenarios, MatrixOne facilitates rapid response times and high-quality outputs through efficient data management, vector and hybrid search capabilities, data preprocessing with Python UDFs, and GPU-accelerated real-time inference. MatrixOne’s low-latency architecture supports Generative AI workloads like large-scale data storage, online inference, and adaptive feedback, empowering enterprises to drive innovation with speed and efficiency.

Time-Series Data Applications

Modern IoT ecosystems generate massive volumes of real-time data from diverse sources like industrial production lines, smart grids, and autonomous systems. MatrixOne is built to handle such demands with millisecond-level, high-concurrency writes and rapid, scalable data retrieval. MatrixOne’s real-time analytics seamlessly integrates with machine learning models, making it an ideal solution for predictive maintenance, energy optimization, and intelligent monitoring.

Mixed Workload Scenarios

Traditional single-node databases often struggle with the simultaneous processing demands of transactional and analytical workloads in enterprise applications like ERP, CRM, and OA systems. MatrixOne’s support for mixed workloads within a single database enables real-time analytics, continuous reporting, and efficient data-driven decision-making without the need for additional analytical databases or sharding. With MatrixOne’s scalability and high concurrency support, enterprises can confidently meet growing data demands while keeping performance at peak levels.

Enterprise SaaS

With the rise of enterprise SaaS applications, supporting multi-tenancy while ensuring cost-efficiency and data isolation is essential. MatrixOne’s native multi-tenant architecture provides load isolation, independent scaling, and unified management for each tenant. This architecture reduces management overhead, ensures data separation, and improves operational efficiency, making MatrixOne an optimal database choice for SaaS platforms.

Key New Features

Multi-mode Data Management

MatrixOne now supports direct access to external object storage, remote file systems, and local storage through Stage objects, as well as datalink access to files in storage systems. This capability significantly simplifies data pipeline construction for Generative AI applications, reducing development overhead and maintenance costs.

Full-text Indexing for Text and JSON Data

Full-text indexing on JSON and TEXT columns greatly enhances performance in AIoT applications, especially when combined with MatrixOne’s JSON data type, which minimizes data redundancy and boosts efficiency.

Vector Search

Enhanced vector search capabilities now provide rapid, large-scale vector retrieval, a critical feature for Generative AI applications involving large language models (LLMs) and retrieval-augmented generation (RAG).

Snapshot-based Backup and Recovery

Cluster and tenant-level data snapshots capture the database state at specific points in time, ensuring rapid recovery while minimally impacting performance. Snapshots support cross-tenant restoration, bolstering MatrixOne’s disaster recovery.

Primary-standby Log Replication for High Availability

Log replication enables transaction log synchronization between primary and standby databases, supporting high availability and disaster recovery. Standby databases can take over in case of primary database failure, ensuring uninterrupted operations.

Point-in-time Recovery (PITR)

PITR captures all data changes post-snapshot, allowing precise restoration to a historical moment in case of accidental operations or data loss. This approach reduces storage costs, enhances recovery efficiency, and provides flexibility for critical business continuity and compliance.

MatrixOne to MySQL CDC

Change Data Capture (CDC) from MatrixOne to MySQL supports real-time disaster recovery for users transitioning from MySQL, maintaining data continuity.

Table-level Publish-Subscribe

Building on previous database-level publish/subscribe, table-level publish-subscribe in this release enables more granular control over data change synchronization, providing enhanced flexibility for data management.

Additional Updates

SQL Enhancements

• Added support for rename table, create pitr, drop pitr, alter pitr, restore pitr, show pitrs.
• Optimized show publications and show subscriptions.
• Enhanced load data infile command to support user-defined column order.

Data Types

• Support datalink data type

Indexes and Constraints

• Support Full-text Index

Functions and Operators

• Support JSON functions: json_row, jq, try_jq, json_extract_string, json_extract_float64 functions.
• Enhanced date manipulation for now() function.

Tools

• mo-backup: Supports PiTR management.
• mo_cdc: supports CDC task management.

MySQL Compatibility

• Support Encode()/Decode() function

Quick Start

Community users and enterprise developers can try MatrixOne with the following command:

docker pull matrixorigin/matrixone:2.0.0

For more details, including architectural insights, installation guides, and tutorials, visit our documentation site. Join our discussions or share feedback on GitHub or in our community WeChat group.

Known Issues

• Standby clusters currently do not support synchronization of data in external tables or stages.
• Standby clusters support only cold backups and cannot be opened in read-only mode.
• CDC supports only table-level data synchronization.
• Snapshot backups support cluster and tenant levels, with restoration possible at the cluster, tenant, database, or table level.
• Snapshots and PiTR backups cannot recover deleted tenant data.

New Contributors

• @zuyu made their first contribution in #16402
• @Aoang made their first contribution in #14897
• @Cyberleu made their first contribution in #19014

Full Changelog: v1.2.4...v2.0.0

v1.2.3-hotfix-20241016

What's Changed

• fix Intersection2Vector panic by @gouhongshen in #19383

Full Changelog: v1.2.3-hotfix-20241010...v1.2.3-hotfix-20241016

v1.2.3-hotfix-20241010

Full Changelog: v1.2.3-hotfix-20240916...v1.2.3-hotfix-20241010

v1.2.4

Release Date: September 23, 2024
MatrixOne Version: v1.2.4

This hotfix addresses several critical issues and includes minor improvements aimed at enhancing stability and performance.

What's Changed

• make query with limit run in single parallel by @badboynt1 in #18803
• cp to 1.2-dev 'fix ndv calculation to make it more accurate' by @badboynt1 in #18848
• fix json extract function(#1.2 hotfix) by @YANGGMM in #18849
• cp to 1.2-dev 'fix the condition for runtime filters (#18496)' by @badboynt1 in #18860

Full Changelog: v1.2.3-hotfix-20240916...v1.2.4

v1.2.3-hotfix-20240916

Release Date: September 16, 2024
MatrixOne Version: v1.2.3-hotfix-20240916

This hotfix addresses several critical issues and includes minor improvements aimed at enhancing stability and performance.

Bug Fixes

• Resolved a transaction retrieval issue in GetOrCreateTxnWithMeta.
• Fixed incorrect value retrieval for unique and secondary keys.
• Corrected a panic issue in KillRoutineConnections.
• Fixed a proxy panic when killing a connection.
• Resolved a bug where sessions could not be transferred after using LOAD DATA LOCAL INFILE.
• Fixed range checkpoint inconsistencies.

What's Changed

• add summary for txnBlock(1.2-dev only) by @aptend in #18691
• fix GetOrCreateTxnWithMeta not get txn error by @w-zr in #18705
• bug fix: unique key/secondary key get incorrect value (#18711) by @ouyuanning in #18717
• set srv.pu before creating hakeeper client by @w-zr in #18672
• Method for reading query results during upgrade modification by @qingxinhome in #18731
• fix KillRoutineConnections panic by @YANGGMM in #18726
• [bug] proxy: fix panic when kill connection by @volgariver6 in #18756
• support order by limit optimization for non-sorted columns by @badboynt1 in #18760
• cp to 1.2-dev 'fix a bug that cause hashmap data wrong' by @badboynt1 in #18779
• [tech] pprof: dump malloc profile by @volgariver6 in #18768
• remove the 1024bytes limit for group_concat result. by @m-schen in #18786
• add support rollback running txn to 1.2 by @zhangxu19830126 in #18775
• [bug] proxy: session cannot be transferred after load data local infile by @volgariver6 in #18787
• fix range checkpoint by @jiangxinmeng1 in #18799

Full Changelog: v1.2.3...v1.2.3-hotfix-20240916

MatrixOne-v1.2.3

Release Date: September 11, 2024
MatrixOne Version: v1.2.3

Compared to the previous version, v1.2.3 focuses on bug fixes, performance optimizations, and minor feature enhancements, without introducing major new features.

Improvements

• Memory Optimization: Enhancements in memory usage, particularly for duplicate checks, SEMI Join, and TableScan operations.
• Enhanced Logging: Added logs for account restrictions, account suspensions, and additional operations, improving debugging and monitoring capabilities.
• Performance Enhancements: Optimized handling of LIMIT 0 queries, improved dynamic cache management, and refined query performance for JSON type ordering.

Bug Fixes

This release addresses multiple issues related to memory usage, transaction handling, proxy connection stability, and more. These fixes improve the overall stability and reliability of MatrixOne.

What’s Changed

• [cherry-pick] : cherry-pick 17161 17020 16522 to save memory useage for dup check and clean up code by @jensenojs in #17211
• add log for account restricted by @YANGGMM in #17327
• fileservice: fix TestFuzzingDiskS3 by @reusee in #17470
• [CherryPick]: Reduce mp.Grow2 in SEMI Join and TableScan Operator by @arjunsk in #17482
• fix restore slow （#1.2-dev） by @YANGGMM in #17288
• Cp 1.2.2 hotfix by @sukki37 in #17554
• fileservice: fix missing free in tests by @reusee in #17544

For a full list of changes and improvements, refer to the complete changelog.

This release is part of our ongoing effort to ensure MatrixOne’s stability, performance, and usability, offering refinements that contribute to a smoother and more efficient database experience.