v4-write-process #345

761417898 · 2024-12-25T08:31:43Z

Implemented the V4 tree model and table model write processes. Support string data types.. The following still need to be implemented/fixed:

Implemented the read process for the V4 table model.
Support encryption.
Verified support for aligned time series in the table model.

# Conflicts: # cpp/src/common/tablet.cc # cpp/src/common/tablet.h # cpp/src/writer/tsfile_writer.cc # cpp/src/writer/tsfile_writer.h

cpp/src/common/device_id.h

cpp/src/common/schema.h

cpp/src/common/tsfile_common.h

cpp/src/file/tsfile_io_writer.cc

cpp/src/writer/tsfile_writer.cc

cpp/src/writer/tsfile_writer.h

zwhzzz0821 · 2024-12-26T03:18:59Z

cpp/src/common/device_id.h

+
+    int segment_num() override { return static_cast<int>(segments_.size()); }
+
+    std::vector<std::string> get_segments() const override { return segments_; }


If read only
you can return const std::vector<std::string>& to avoid copy

zwhzzz0821 · 2024-12-26T03:24:18Z

cpp/src/common/device_id.h

+            return tableName_;
+        }
+
+        size_t lastSeparatorPos = device_id_.find_last_of('.');


use PATH_SEPARATOR_CHAR instead of '.'

zwhzzz0821 · 2024-12-26T03:25:00Z

cpp/src/common/device_id.h

+    std::string get_device_name() const override { return device_id_; };
+
+    std::string get_table_name() override {


You can verify that you need to return a copy

zwhzzz0821 · 2024-12-26T03:26:45Z

cpp/src/common/schema.h

+        if (ret == common::E_OK) {
+            if (RET_FAIL(common::SerializationUtil::write_str(measurement_name_,
+                                                              out))) {
+                for (auto &prop : props_) {


const auto& prop

zwhzzz0821 · 2024-12-26T03:29:14Z

cpp/src/common/schema.h

+        : table_name_(table_name),
+          measurement_schemas_(measurement_schemas),
+          column_categories_(column_categories) {
+        int idx = 0;


use size_t to avoid potential problems

zwhzzz0821 · 2024-12-26T03:40:05Z

cpp/src/common/allocator/my_string.h

+    std::string to_string() {
+        return std::string(buf_, len_);
+    }


There is already one such method. You can view the develop branch

zwhzzz0821 · 2024-12-26T04:11:58Z

cpp/src/common/tablet.h


    int add_timestamp(uint32_t row_index, int64_t timestamp);

+    // int set_value(int row_index, int schema_index, double val);


zwhzzz0821 · 2024-12-26T04:20:53Z

cpp/src/file/tsfile_io_writer.cc

    int ret = write_byte(CHUNK_GROUP_HEADER_MARKER);
    if (ret != common::E_OK) {
        return ret;
    }
-    ret = write_string(device_name);
+    device_name->serialize(write_stream_);


ret = device_name->serialize(write_stream_);

zwhzzz0821 · 2024-12-26T04:22:59Z

cpp/src/file/tsfile_io_writer.cc

+        auto cur_device_name = cur_device_name_;
+        if (*iter.get()->device_name_ == *cur_device_name) {
            use_prev_alloc_cgm_ = true;
            cur_chunk_group_meta_ = iter.get();
            break;


why use cur_device_name?

zwhzzz0821 · 2024-12-26T04:29:32Z

cpp/src/writer/tsfile_writer.h

    int write_typed_column(storage::ChunkWriter *chunk_writer,
                           int64_t *timestamps, bool *col_values,
-                           common::BitMap &col_notnull_bitmap,
-                           int32_t row_count);
+                           common::BitMap &col_notnull_bitmap, int start_idx,
+                           int end_idx);
    int write_typed_column(storage::ChunkWriter *chunk_writer,
                           int64_t *timestamps, int32_t *col_values,
-                           common::BitMap &col_notnull_bitmap,
-                           int32_t row_count);
+                           common::BitMap &col_notnull_bitmap, int start_idx,
+                           int end_idx);
    int write_typed_column(storage::ChunkWriter *chunk_writer,
                           int64_t *timestamps, int64_t *col_values,
-                           common::BitMap &col_notnull_bitmap,
-                           int32_t row_count);
+                           common::BitMap &col_notnull_bitmap, int start_idx,
+                           int end_idx);
    int write_typed_column(storage::ChunkWriter *chunk_writer,
                           int64_t *timestamps, float *col_values,
-                           common::BitMap &col_notnull_bitmap,
-                           int32_t row_count);
+                           common::BitMap &col_notnull_bitmap, int start_idx,
+                           int end_idx);
    int write_typed_column(storage::ChunkWriter *chunk_writer,
                           int64_t *timestamps, double *col_values,
-                           common::BitMap &col_notnull_bitmap,
-                           int32_t row_count);
+                           common::BitMap &col_notnull_bitmap, int start_idx,
+                           int end_idx);


Is it better to use template?

…t string data types.

ColinLeeo · 2025-01-20T17:04:17Z

cpp/src/common/tablet.cc

        return E_OUT_OF_RANGE;
    }


如果row_index 是 uint32_t的话，就把max_row_num_设置成uint32_t把，现在有些参数是int32的有些是 uint32的。

ColinLeeo · 2025-01-20T18:10:49Z

cpp/src/writer/tsfile_writer.h

@@ -61,13 +65,26 @@ class TsFileWriter {
    int register_aligned_timeseries(
        const std::string &device_id,
        const std::vector<MeasurementSchema *> &measurement_schemas);
+    void register_table(const std::shared_ptr<TableSchema> &table_schema);


I dont think pass a shared pointer to writer is a good idea. The TableSchema is internal info and shold not pass a sharedPtr directly

ColinLeeo · 2025-01-20T18:16:57Z

cpp/src/common/schema.h

+        TableSchema(const std::string &table_name,
+                    const std::vector<std::shared_ptr<MeasurementSchema> >
+                    &column_schemas,
+                    const std::vector<ColumnCategory> &column_categories)


If you can make sure the owner of measurement schema, dont use shared_ptr.

jt2594838 · 2025-01-21T02:16:32Z

cpp/src/common/row_record.h

+        if (type_ == common::STRING) {
+            return value_.strval_;
+        } else {
+            std::cout << "not String type" << std::endl;


jt2594838 · 2025-01-21T02:43:02Z

cpp/src/common/tablet.cc

+    // value_matrix_ = (void **)malloc(sizeof(void *) * schema_count);
+    // for (size_t c = 0; c < schema_count; c++) {
+    //     const MeasurementSchema &schema = schema_vec_->at(c);
+    //     value_matrix_[c] =
+    //         malloc(get_data_type_size(schema.data_type_) * max_row_num_);
+    // }


jt2594838 · 2025-01-21T02:46:43Z

cpp/src/common/tablet.cc

+template <>
+void Tablet::process_val(uint32_t row_index, uint32_t schema_index, common::String val) {
+    value_matrix_[schema_index].string_data[row_index].dup_from(val, page_arena_);
+    bitmaps_[schema_index].set(row_index); /* mark as non-null */


The bitmaps of a Tablet in the Java edition are marked when the value is null. I think we should make them consistent.

jt2594838 · 2025-01-21T02:50:13Z

cpp/src/common/tsblock/tuple_desc.cc

@@ -44,6 +44,10 @@ uint32_t TupleDesc::get_single_row_len(int *erro_code) {
                totol_len += sizeof(double);
                break;
            }
+            case common::STRING: {
+                totol_len += DEFAULT_RESERVED_SIZE_OF_STRING + STRING_LEN;


totol_len -> total_len

jt2594838 · 2025-01-21T02:58:11Z

cpp/src/common/tsfile_common.h

+  /*
+  FORCE_INLINE bool is_same_measurement_name(const common::String &s1, const
+  common::String &s2)
+  {
+    return s1.equal_to(s2);
+  }
+  */
+private:


jt2594838 · 2025-01-21T03:19:20Z

cpp/src/file/tsfile_io_reader.h

+        bool read_file_created_;
+    };
+
+    // class TsFileIOReaderSet


jt2594838 · 2025-01-21T03:22:00Z

cpp/src/file/tsfile_io_writer.cc

    std::cout << "ChunkGroupMeta = {insert_target_name_="
-              << cgm->insert_target_name_ << ", chunk_meta_list_={";
+              << cgm->device_id_ << ", chunk_meta_list_={";


{insert_target_name_= -> {device_id_=

zwhzzz0821 · 2025-01-21T04:14:12Z

cpp/src/common/device_id.h

+    static const char PATH_SEPARATOR = '.';
+    static const int DEFAULT_SEGMENT_NUM_FOR_TABLE_NAME = 3;


include /common/constant/tsfile_constant

zwhzzz0821 · 2025-01-21T04:19:31Z

cpp/src/common/device_id.h

+                [](const std::string& a, const std::string& b) {
+                    return a.empty() ? b : a + PATH_SEPARATOR + b;
+                });
+            final_segments.push_back(table_name);


Suggested change

final_segments.push_back(table_name);

final_segments.emplace_back(std::move(table_name));

zwhzzz0821 · 2025-01-21T04:28:02Z

cpp/src/common/schema.h

+                        } else if (RET_FAIL(common::SerializationUtil::write_str(
+                            prop.second, out))) {
+                        }
+                        if (ret != common::E_OK) break;


Suggested change

if (ret != common::E_OK) break;

if (IS_FAIL(ret)) break;

zwhzzz0821 · 2025-01-21T04:28:49Z

cpp/src/common/schema.h

+                            value, in))) {
+                        }
+                        props_.insert(std::make_pair(key, value));
+                        if (ret != common::E_OK) break;


Suggested change

if (ret != common::E_OK) break;

if (IS_FAIL(ret)) break;

zwhzzz0821 · 2025-01-21T04:31:43Z

cpp/src/common/schema.h

+        const std::string &get_table_name() { return table_name_; }
+
+        std::vector<std::string> get_measurement_names() const {
+            std::vector<std::string> ret;


ret.reserve(column_schemas_.size())

fix some issues

761417898 added 27 commits December 9, 2024 08:52

v4 writer

e9fb2d2

tmp

2fe33ee

tmp

a6d212b

write table

0c0e51e

fix device_id.h

94fa48e

write table, split deviceid

b21d479

fix get_device_id

a92ae2c

fix tablet get_value

1ffa310

fix device id

67b4db9

complete the develop of the registration and writing process

dc8ba7d

fix tablet

87ca0cf

fix v4 write

50fe3f4

fix device_id.h

f6764ee

fix flush()

349c129

Implement the Table Model Write Process

2924a72

Implement the Table Model Write Process

402bc9c

Implement the Table Model Write Process

dc9b350

fix mem leak

7ac7233

try to fix mem leak

142217a

try to fix mem leak

59095f3

try to fix mem leak

358a71b

try to fix mem leak

03d4b10

try to fix mem leak

810bf12

Merge remote-tracking branch 'origin/develop' into feature/v4-write

64d04e4

# Conflicts: # cpp/src/common/tablet.cc # cpp/src/common/tablet.h # cpp/src/writer/tsfile_writer.cc # cpp/src/writer/tsfile_writer.h

try to fix men leak

77c4149

fix men leak

7038d0d

Implement the v4 writing process

a9438c4

761417898 mentioned this pull request Dec 25, 2024

[CPP] Support V4 TsFile #103

Open

jt2594838 requested changes Dec 25, 2024

View reviewed changes

zwhzzz0821 reviewed Dec 26, 2024

View reviewed changes

761417898 added 15 commits January 17, 2025 00:56

fix mem leak

eed8618

fix mem leak

89771d4

fix compilation error

2390bba

fix compilation error

c7b5bbe

fix compilation error

8098038

fix compilation error

9882039

bugfix

9986dbf

merge develop

c602470

bugfix

5bd2c0d

fix LZ4 compressor

bd3e16e

tmp

36efa31

Implemented the V4 tree model and table model write processes; Suppor…

717cb2d

…t string data types.

Implemented the V4 tree model and table model write processes; Suppor…

5faebb7

…t string data types.

Implemented the V4 tree model and table model write processes; Suppor…

4284302

…t string data types.

Implemented the V4 tree model and table model write processes; Suppor…

01cee15

…t string data types.

ColinLeeo reviewed Jan 20, 2025

View reviewed changes

jt2594838 approved these changes Jan 21, 2025

View reviewed changes

zwhzzz0821 approved these changes Jan 21, 2025

View reviewed changes

761417898 added 5 commits January 21, 2025 19:47

Fix some issues

f899bdd

Fix some issues

79dc252

Fix some issues

79e14e3

Fix some issues

cefc80c

try to fix compilation error

b01ffe6

ColinLeeo approved these changes Jan 22, 2025

View reviewed changes

761417898 added 4 commits January 22, 2025 13:16

try to fix compilation error

54da3ab

fix some issues

9e6cd21

fix some issues

2ab81f0

Merge pull request #3 from 761417898/feature/v4

2f80ca9

fix some issues

761417898 merged commit 00002d6 into apache:develop Jan 22, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v4-write-process #345

v4-write-process #345

761417898 commented Dec 25, 2024 •

edited

Loading

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

761417898 Jan 1, 2025

zwhzzz0821 Dec 26, 2024

ColinLeeo Jan 20, 2025 •

edited

Loading

ColinLeeo Jan 20, 2025 •

edited

Loading

ColinLeeo Jan 20, 2025

jt2594838 Jan 21, 2025

jt2594838 Jan 21, 2025

jt2594838 Jan 21, 2025

jt2594838 Jan 21, 2025

jt2594838 Jan 21, 2025

jt2594838 Jan 21, 2025

jt2594838 Jan 21, 2025

zwhzzz0821 Jan 21, 2025

zwhzzz0821 Jan 21, 2025

zwhzzz0821 Jan 21, 2025

zwhzzz0821 Jan 21, 2025

zwhzzz0821 Jan 21, 2025


		int segment_num() override { return static_cast<int>(segments_.size()); }

		std::vector<std::string> get_segments() const override { return segments_; }

		std::string get_device_name() const override { return device_id_; };

		std::string get_table_name() override {


		int add_timestamp(uint32_t row_index, int64_t timestamp);

		// int set_value(int row_index, int schema_index, double val);

		static const char PATH_SEPARATOR = '.';
		static const int DEFAULT_SEGMENT_NUM_FOR_TABLE_NAME = 3;

	final_segments.push_back(table_name);
	final_segments.emplace_back(std::move(table_name));

v4-write-process #345

v4-write-process #345

Conversation

761417898 commented Dec 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ColinLeeo Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

ColinLeeo Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

761417898 commented Dec 25, 2024 •

edited

Loading

ColinLeeo Jan 20, 2025 •

edited

Loading

ColinLeeo Jan 20, 2025 •

edited

Loading