gg-21461 #180

ygerzhedovich · 2019-07-22T15:07:51Z

gg-21461

(cherry picked from commit b5f3f72)

(cherry picked from commit 7883609)

(cherry picked from commit 7043f6c)

…terator

(cherry picked from commit 1a0fd3e)

…or 2.1 pds

…er doesn't configured explicitly. (cherry picked from commit c8bd480)

(cherry-picked from commit #fa8e5da60546bf2e46cc16b13baf0a21c394e7b6)

(cherry picked from commit cafab0d) # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtPartitionTopologyImpl.java # modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsCacheWalDisabledOnRebalancingTest.java

…fTest

…ence.GridCacheOffheapManager#restorePartitionStates method. (cherry picked from commit 621f6af)

…) doesn't work. (cherry picked from commit 9350760)

…e-ignite into ignite-2.5-master

…asicIndexSelfTest

…ame make DR paused in case of persistence. (cherry picked from commit e093716)

…ueries. (cherry picked from commit 03283da)

…e same name make DR paused in case of persistence." This reverts commit 3616b6a.

…Left flaky failed on TC (cherry picked from commit fe0ec74)

…ctivateSimple_5_Servers_5_Clients_FromClient flaky failed on TC (cherry picked from commit 2578931)

…ormation about cache group. (cherry picked from commit 82409a9)

…or all x.x-master branches

… ignite system properties (cherry picked from commit 05cdf46)

… called (cherry picked from commit d1b0787)

…ientInForceServerModeStopsOnExchangeHistoryExhaustion is flaky.

… from 8.4.9 to 8.5.7 if POJO primary key is used

(cherry picked from commit 7baa018)

…emote filter class is missing (#177) (cherry picked from commit a28138f)

…es and test coverage. Phase #1.

…message, if applicable. (cherry picked from commit 4b045f5)

…about group and cache id. (cherry picked from commit f3017ea)

(cherry picked from commit f7f0b89)

(cherry picked from commit 4f05cd9)

(cherry picked from commit fb2553a)

…ce.file.FilePageStore instances (cherry picked from commit 806b1b6)

(cherry picked from commit 0a17bb9)

…p for experemental commands. (cherry picked from commit 86ee0b5)

…pi without IgniteConfiguration.setIncludeEventTypes(EventType.EVT_TASK_FINISHED, EventType.EVT_TASK_FAILED) leads to memory leak - Fixed apache#6690. (apache#293) (cherry picked from commit e2d6632)

(cherry picked from commit d8753ae)

…ning

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

pavlukhin · 2019-07-23T06:28:34Z

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

+                byte[] originalObjBytes = val.getBytesNoCopy();
+
+                // read size more then available space or more then origin length
+                if(len > inlineSize - fieldOff - 3 || len > originalObjBytes.length){


len > originalObjBytes.length seems to mean unexpected (garbage) data in current position. Cannot we do comparison in branch if (type == Value.JAVA_OBJECT) using following strategy:

Prepare (virtually) an expected byte content which is saved by regular inline procedures.

Compare that content with bytes stored in the index page.

Exact match means that JAVA_OBJECT inline is (almost 100%) supported. Otherwise it is not (tree corruption is possible, might be warning is a good idea). In both cases we can finish tree scan here.

Also using of InlineIndexHelper.compare is dangerous below, because it is more broad method and it is used in different places. But here we are just solving a compatibility issue from certain point in history.

for your comparison proposal - by fact it do the same way.

objections to use InlineIndexHelper.compare don't understand. Let's discuss internally.

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

pavlukhin · 2019-07-24T12:45:30Z

...s/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/database/H2Tree.java

@@ -148,7 +167,9 @@ protected H2Tree(
            if (metaInfo.useUnwrappedPk())
                throw new IgniteCheckedException("Unwrapped PK is not supported by current version");

-            this.inlineSize = metaInfo.inlineSize();
+            inlineSize = metaInfo.inlineSize();


For my understanding. How is it possible that inlineSize passed to a constructor is not equal to metaInfo.inlineSize()?

inlinesize can be different due to use different set of columns to calculate it ( as example before support JAVA_OBJECT type and after)

Where do we calculate it?

@pavlukhin see org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex#computeInlineSize

pavlukhin · 2019-07-24T12:46:46Z

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

        for (InlineIndexHelper ih : inlineHelpers) {
            if (fieldOff >= inlineSize)
                return false;

            if (ih.type() != Value.JAVA_OBJECT) {
+                if(ih.size() < 0)


Formatting.

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

pavlukhin · 2019-07-24T12:58:06Z

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

+                byte[] inlineBytes = PageUtils.getBytes(pageAddr, off + fieldOff + 3, len);
+
+                for (int i = 0; i < len; i++) {
+                    if (inlineBytes[i] == originalObjBytes[i])


Seems more straightforward to me:

for (int i = 0; i < len; i++) { if (inlineBytes[i] != originalObjBytes[i]) { inlineObjectSupportedDecision(false, i + " byte compare"); return true; } }

pavlukhin · 2019-07-24T13:02:41Z

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java

        for (InlineIndexHelper ih : inlineHelpers) {
            if (fieldOff >= inlineSize)
                return false;

            if (ih.type() != Value.JAVA_OBJECT) {
+                if(ih.size() < 0)
+                    varLenPresents=true;


I suppose we can initialize it constructor.

yes, but no reason to do it

antkr and others added 30 commits July 1, 2019 21:56

GG-14719 jackson dependency upgraded to 2.9.9

b457b03

(cherry picked from commit b5f3f72)

GG-14930 Upgraded Jetty dependency to 9.4.18

53899a0

(cherry picked from commit 7883609)

GG-19121 commons-codec dependency upgraded to 1.12

422e6ce

(cherry picked from commit 7043f6c)

GG-20558 Recovery mode for page memory used only for binary restore

e33a69a

GG-20536 Holding exception at the record reading until next call of i…

859c409

…terator

GG-14970 IdleVerify command should print end time of execution.

c41982f

(cherry picked from commit 1a0fd3e)

GG-20761 [GG-20749] remove migration pendingTree compatibility test f…

7719fe9

…or 2.1 pds

GG-20420 NullPointerException If transaction failed and failure handl…

5a616de

…er doesn't configured explicitly. (cherry picked from commit c8bd480)

GG-20193 fix IGNITE-9858 adapted into the branch

3b97ac7

(cherry-picked from commit #fa8e5da60546bf2e46cc16b13baf0a21c394e7b6)

GG-20464 backport to 8.5-master of GG-19518 IoStatisticsBasicIndexSel…

1c9dbe2

…fTest

GG-20798 Fix failing tests.

b823cc6

GG-19492 Add debug output for o.a.i.internal.processors.cache.persist…

44091fa

…ence.GridCacheOffheapManager#restorePartitionStates method. (cherry picked from commit 621f6af)

GG-18891 CommandHandlerParsingTest#testExperimentalCommandIsDisabled(…

b01ce11

…) doesn't work. (cherry picked from commit 9350760)

Merge branch 'ignite-2.5-master' of https://github.com/gridgain/apach…

20df5b0

…e-ignite into ignite-2.5-master

GG-20810 backport to 8.5-master of GG-20807 Fix failing IoStatisticsB…

ad9070d

…asicIndexSelfTest

GG-19515: GG: destroying of the cache and re-creating with the same n…

3616b6a

…ame make DR paused in case of persistence. (cherry picked from commit e093716)

GG-20822: Compact log and drop log level down to info for cancelled q…

b04ec7e

…ueries. (cherry picked from commit 03283da)

Revert "GG-19515: GG: destroying of the cache and re-creating with th…

21de580

…e same name make DR paused in case of persistence." This reverts commit 3616b6a.

GG-21061 [GG-21057] Fixed IgniteCacheReplicatedQuerySelfTest.testNode…

b773b6c

…Left flaky failed on TC (cherry picked from commit fe0ec74)

GG-21074 [GG-21055] Fixed IgniteClusterActivateDeactivateTest.testDea…

d8a004f

…ctivateSimple_5_Servers_5_Clients_FromClient flaky failed on TC (cherry picked from commit 2578931)

GG-20185 control.sh validate_indexes SQL Index issue must contain inf…

dfff292

…ormation about cache group. (cherry picked from commit 82409a9)

GG-20917 [8.5-next]-[GG-20908] Set max possible maintenance version f…

e093eeb

…or all x.x-master branches

GG-20729 Add ability to set default parallelism of rebuild indexes in…

0adf555

… ignite system properties (cherry picked from commit 05cdf46)

GG-20845 Added explicit indexes to WALRecord.RecordType.

ca61a70

GG-19486 Fixed IgniteOutOfMemoryException when IgniteCache#clear() is…

91c2faa

… called (cherry picked from commit d1b0787)

GG-20875 Fix assertion on counter reservation.

f593ded

GG-21105: [8.5-next]-[GG-21101] IgniteCacheClientReconnectTest.testCl…

4b3419f

…ientInForceServerModeStopsOnExchangeHistoryExhaustion is flaky.

GG-19609 [8.5-master]-[GG-19255] Inline index compatibility is broken…

7157097

… from 8.4.9 to 8.5.7 if POJO primary key is used

GG-21186 Fixed IgniteDataStorageMetricsSelfTest

784d1e0

pavlukhin and others added 19 commits July 16, 2019 14:54

GG-19412 Exclude test resource Wrapper.ser from license check

2676c12

(cherry picked from commit 7baa018)

GG-20850: Fix @ignore link.

ebb2603

GG-19412 Fail Continuous Query registration and do not fail node if r…

2ca3f15

…emote filter class is missing (#177) (cherry picked from commit a28138f)

GG-20567 Updated dependencies to fix possible vulnerabilities

f6576fc

GG-20594 GG-20593 GG-20592 GG-19428 [IGNITE-9560] Security Engine fix…

4c22ea9

…es and test coverage. Phase #1.

GG-19117 GridCacheProcessor should add info about cache in exception …

b8a547f

…message, if applicable. (cherry picked from commit 4b045f5)

GG-21225 control.sh validate_indexes SQL Index issue add information …

60eda4f

…about group and cache id. (cherry picked from commit f3017ea)

GG-21472 [GG-21254] Fixed *.testFailAfterStart failed on TC

884f661

(cherry picked from commit f7f0b89)

GG-21472 [GG-21254] Fixed *.testFailAfterStart failed on TC

f196f2c

(cherry picked from commit f7f0b89)

GG-20905 Print warning if awaiting next wal segment it too long.

6674af2

(cherry picked from commit 4f05cd9)

GG-21338 Fixed checkpointReadLock() may hang during node stop

143257c

(cherry picked from commit fb2553a)

GG-17345 Reduce heap occupation by o.a.i.i.processors.cache.persisten…

eab2af9

…ce.file.FilePageStore instances (cherry picked from commit 806b1b6)

GG-17345 Checking file length in FilePageStore#exists()

347ae68

(cherry picked from commit 0a17bb9)

GG-21232 control.sh if experimental command disabled - don't show hel…

3fe68c2

…p for experemental commands. (cherry picked from commit 86ee0b5)

GG-20567 Removed outdated dependency from pom file

7bbf4d0

GG-21630 [GG-20866] [IGNITE-11966] Fixed using AdaptiveLoadBalancingS…

a5a2d85

…pi without IgniteConfiguration.setIncludeEventTypes(EventType.EVT_TASK_FINISHED, EventType.EVT_TASK_FAILED) leads to memory leak - Fixed apache#6690. (apache#293) (cherry picked from commit e2d6632)

GG-21395 Fixed unguarded log.info() usages

d62c77d

GG-21696 [GG-21389] Flush records on cache stop

9a3b007

(cherry picked from commit d8753ae)

GG-21461: improve algorithm of detecting use of java objects for inli…

8d83dbc

…ning

pavlukhin reviewed Jul 23, 2019

View reviewed changes

ygerzhedovich added 2 commits July 23, 2019 11:26

GG-21461: fixes after review.

8a70bd0

GG-21461: Corrupt index tree fix

67a599b

pavlukhin reviewed Jul 24, 2019

View reviewed changes

...java/org/apache/ignite/internal/processors/query/h2/database/H2TreeInlineObjectDetector.java Outdated Show resolved Hide resolved

pavlukhin reviewed Jul 24, 2019

View reviewed changes

ygerzhedovich added 2 commits July 24, 2019 18:44

GG-21461: fixes after review

98ea82f

GG-21461: fixes after review

27f7124

vveider force-pushed the ignite-2.5-master branch from 9f0de85 to c77bc90 Compare December 4, 2019 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gg-21461 #180

gg-21461 #180

ygerzhedovich commented Jul 22, 2019

pavlukhin Jul 23, 2019

ygerzhedovich Jul 23, 2019

pavlukhin Jul 24, 2019

ygerzhedovich Jul 24, 2019

pavlukhin Jul 25, 2019

ygerzhedovich Jul 25, 2019

pavlukhin Jul 24, 2019

ygerzhedovich Jul 24, 2019

pavlukhin Jul 24, 2019

ygerzhedovich Jul 24, 2019

pavlukhin Jul 24, 2019

ygerzhedovich Jul 24, 2019

gg-21461 #180

Are you sure you want to change the base?

gg-21461 #180

Conversation

ygerzhedovich commented Jul 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment