Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new tsdb track to test _tsid hashing #464

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
ec724cc
feature: add a new tsdb track to test _tsid hashing
salvatore-campagna Oct 4, 2023
d964e53
fix: include a parameter which enables more dimension fields
salvatore-campagna Oct 4, 2023
5673d5d
fix: inclrrect uncompressed-bytes
salvatore-campagna Oct 4, 2023
b67661b
fix: increase index.mapping.dimension_fields.limit
salvatore-campagna Oct 5, 2023
ef1e822
fix: introduce standard index_mode
salvatore-campagna Oct 10, 2023
899c73e
fix: include additional force merge
salvatore-campagna Oct 11, 2023
789cdbc
fix: just wait before force merging
salvatore-campagna Oct 11, 2023
33400ae
fix: assign different names to tasks
salvatore-campagna Oct 11, 2023
93d04db
fix: run force merge twice
salvatore-campagna Oct 12, 2023
2f78d42
fix: no such operation type
salvatore-campagna Oct 12, 2023
cbd6c6e
fix: update lucene merge policy
salvatore-campagna Oct 12, 2023
5681352
fix: disable lifecycle
salvatore-campagna Oct 13, 2023
fcfc9aa
fix: move template out of settings
salvatore-campagna Oct 13, 2023
286054b
fix: include downsampling after indexing
salvatore-campagna Dec 18, 2023
019efda
Merge branch 'master' into feature/tsdb-synthetic-track
salvatore-campagna Dec 18, 2023
d62f047
fix: block writes to all indices
salvatore-campagna Dec 22, 2023
e97319d
fix: include ingest_mode for datastream and index
salvatore-campagna Dec 22, 2023
8257b87
fix: gate usage of target-data-stream
salvatore-campagna Dec 22, 2023
9e14f2c
fix: index pattern and template
salvatore-campagna Dec 22, 2023
8504660
fix: gate usage of delete operation
salvatore-campagna Dec 22, 2023
a8e6680
fix: remove comma
salvatore-campagna Dec 22, 2023
8159c85
fix: index and data_stream modes
salvatore-campagna Dec 22, 2023
076e48f
fix: force mode time_series
salvatore-campagna Jan 26, 2024
e35c3ec
Merge branch 'elastic:master' into feature/tsdb-synthetic-track
salvatore-campagna Jan 26, 2024
0042771
fix: use different names for index and datastream
salvatore-campagna Jan 26, 2024
3590bd3
fix: remove datastream indexing mode
salvatore-campagna Jan 26, 2024
4edaeba
fix: gate settings not available in serverless deployments
salvatore-campagna Jan 26, 2024
f0bf7ab
fix: cleanup tsdb synthetic dataset track
salvatore-campagna Jan 26, 2024
934d28b
fix: delete only the tsdb_index
salvatore-campagna Jan 27, 2024
27025f0
fix: replace _all with tsdb_synth index
salvatore-campagna Jan 27, 2024
2308717
fix: merging stuck
salvatore-campagna Jan 29, 2024
5fbc2d1
fix: enable create op_type while indexing into tsdb_synth index
salvatore-campagna Jan 30, 2024
217c893
fix: comprssed and uncompressed file size
salvatore-campagna Feb 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
339 changes: 339 additions & 0 deletions tsdb_synth/README.md

Large diffs are not rendered by default.

118 changes: 118 additions & 0 deletions tsdb_synth/challenges/default.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
{
"name": "append-no-conflicts",
"description": "Indexes the whole document corpus and runs downsampling",
"default": true,
"schedule": [
{# non-serverless-cluster-settings-marker-start #}{%- if build_flavor != "serverless" or serverless_operator == true -%}
{
"name":"increase-max_buckets_setting",
"tags": ["setup"],
"operation": {
"operation-type": "raw-request",
"method": "PUT",
"path": "/_cluster/settings",
"body": {
"transient": {
"search.max_buckets" : 300000
}
},
"include-in-reporting": false
}
},
{%- endif -%}{# non-serverless-cluster-settings-marker-end #}
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default({}) | tojson}}
}
},
{
"name": "check-cluster-health",
"operation": {
"operation-type": "cluster-health",
"index": "tsdb_synth",
"request-params": {
"wait_for_status": "{{cluster_health | default('green')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": "index",
"warmup-time-period": 240,
"clients": {{bulk_indexing_clients | default(8)}},
"ignore-response-error-level": "{{error_level | default('non-fatal')}}"
},
{
"name": "refresh-after-index",
"operation": "refresh"
},
{
"operation": {
"operation-type": "force-merge",
"request-timeout": 7200{%- if force_merge_max_num_segments is defined %},
"max-num-segments": {{ force_merge_max_num_segments | tojson }}
{%- endif %}
}
},
{
"name": "wait-until-merges-finish",
"operation": {
"operation-type": "index-stats",
"index": "_all",
"condition": {
"path": "_all.total.merges.current",
"expected-value": 0
},
"retry-until-success": true,
"include-in-reporting": false
}
},
{
"name": "refresh-after-force-merge",
"operation": "refresh"
},
{# serverless-post-ingest-sleep-marker-start #}{%- if post_ingest_sleep|default(false) -%}
{
"name": "post-ingest-sleep",
"operation": {
"operation-type": "sleep",
"duration": {{ post_ingest_sleep_duration|default(30) }}
}
},
{%- endif -%}{# serverless-post-ingest-sleep-marker-end #}
{
"name": "block-source-index-writes",
"operation": {
"operation-type": "raw-request",
"path": "/tsdb_synth/_block/write",
"method": "PUT"
}
},
{
"name": "downsample-1m",
"operation": {
"operation-type": "downsample",
"fixed-interval": "1m"
}
},
{
"name": "downsample-1h",
"operation": {
"operation-type": "downsample",
"fixed-interval": "1h"
}
},
{
"name": "downsample-1d",
"operation": {
"operation-type": "downsample",
"fixed-interval": "1d"
}
}
]
}
2 changes: 2 additions & 0 deletions tsdb_synth/files.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
documents.json.bz2
documents-1k.json.bz2
Loading
Loading