refactor(clustering/rpc): tunings and fixes for rubustness #13771

chronolaw · 2024-10-17T22:17:05Z

Summary

Please squash and merge.

KAG-5617

Related tickets:
KAG-5602

Checklist

The Pull Request has tests
A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #[issue number]

kong/db/strategies/off/init.lua

chobits · 2024-10-30T06:41:39Z

kong/db/declarative/import.lua

+
+  -- If the item belongs to a specific workspace,
+  -- use it directly without using the default one.
+  local ws_id = item.ws_id or workspace_id(schema, options)


TODO: need to add a test to check the ws_id value if the schema is non-workspaceable. https://konghq.atlassian.net/browse/KAG-5706

This comment is not a block.

chobits · 2024-10-30T06:46:19Z

kong/db/declarative/import.lua

+    for _, wid in ipairs {ws_id, GLOBAL_WORKSPACE_TAG} do
+      local key = unique_field_key(entity_name, wid, "cache_key", cache_key)
+
+      -- store item_key or nil into lmdb
+      t:set(key, idx_value)


not a blocker, just a comment for other reviewers.

TODO: we only want to save the entity with key containing specific ws_id, not a GLOBAL workspace id like*, which does not consume too much memory space for lmdb. https://konghq.atlassian.net/browse/KAG-5704

chobits · 2024-10-30T06:51:34Z

kong/db/declarative/import.lua

+-- * <entity_name>|*|*|<pk_string> => serialized item
+-- * <entity_name>|<ws_id>|*|<pk_string> => actual item key
+--
+-- * <entity_name>|*|<unique_field_name>|sha256(field_value) => actual item key
+-- * <entity_name>|<ws_id>|<unique_field_name>|sha256(field_value) => actual item key
+--
+-- * <entity_name>|*|<foreign_field_name>|<foreign_key>|<pk_string> => actual item key
+-- * <entity_name>|<ws_id>|<foreign_field_name>|<foreign_key>|<pk_string> => actual item key


not a blocker, -> https://github.com/Kong/kong/pull/13771/files#r1821977274

chronolaw · 2024-10-30T09:58:50Z

kong/db/declarative/export.lua

@@ -359,7 +359,8 @@ local sync_emitter = {

  emit_entity = function(self, entity_name, entity_data)
    self.out_n = self.out_n + 1
-    self.out[self.out_n] = { type = entity_name , row = entity_data, version = self.sync_version, }
+    self.out[self.out_n] = { type = entity_name , row = entity_data, version = self.sync_version,
+                             ws_id = kong.default_workspace, }


Check it later

Here ws_id is a fallback, if row(entity) has the field ws_id, do_sync() will use it but not delta.ws_id.

chronolaw · 2024-10-30T10:09:56Z

kong/clustering/services/sync/rpc.lua

        if not res then
          return nil, err
        end
      end

-      local res, err = insert_entity_for_txn(t, delta_type, delta_row, nil)
+      local res, err = insert_entity_for_txn(t, delta_type, delta_row, opts)


row.ws_id ? delta.ws_id?

In hooks.lua, delta item will always has the field ws_id, so opts.workspace is a fallback.

local deltas = { { type = name, pk = pk, ws_id = ws_id, entity = is_delete and ngx_null or entity, }, }

* fix(clustering/sync): set ws_id in do_sync() * comments

* fix(clustering/sync): delta.version may be null * isempty * default_ws_changed

* rename db id to pk * change id to pk in Lua * change pk to json * schema:extract_pk_values

Co-authored-by: Datong Sun <[email protected]>

* change db filed name * rename in rpc.sync * rename in export

…#13815) * fix(incremental sync): fix cache_key handling for select_by_cache_key * fix some comments * fix: remove unnecessary comment

* refactor(dbless): clean logic of selec_by_field * check schema_field

…13816)

team-gateway-bot · 2024-11-01T09:12:48Z

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13771-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13771-to-master-to-upstream
git checkout -b cherry-pick-13771-to-master-to-upstream
ancref=$(git merge-base 5cab556adc6b0c74c11b83baeb2b81010881e8dc b37bdd165487396379420cef355d16f3fb8a8583)
git cherry-pick -x $ancref..b37bdd165487396379420cef355d16f3fb8a8583

chronolaw · 2024-11-01T09:38:46Z

Cherry-picked in https://github.com/Kong/kong-ee/pull/10460

* fix(db): pagesize must not be less than 2 * fix(db): set global key in lmdb * fix(db): use request_aware_table correctly * fix(dbless): get default workspace correctly * fix(clustering): check latest_version and d.ws_id * fix(clustering): fix dp do_sync() * fix(clustering):fix declarative export sync * fix(clustering):check latest_version == ngx_null in CP * fix(clustering): inc sync add ws (#13790) * fix(clustering/sync): set ws_id in do_sync() * fix(clustering): delta version is null (#13789) * fix(clustering/sync): delta.version may be null * refactor(clustering): change sync data structure (#13794) * fix(incremental sync): added comment for 2 times do_sync (#13801) * fix(incremental sync): fix cache_key handling for select_by_cache_key (#13815) * fix(incremental sync): fix cache_key handling for select_by_cache_key * refactor(dbless): clean logic of select_by_field (#13817) * refactor(dbless): clean logic of selec_by_field * fix(incremental sync): fixed ws_id processing in _set_entity_for_txn (#13816) --------- Co-authored-by: Xiaochen Wang <[email protected]> Co-authored-by: Datong Sun <[email protected]>

chronolaw added the skip-changelog label Oct 17, 2024

pull-request-size bot added the size/XS label Oct 17, 2024

github-actions bot added core/db cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee labels Oct 17, 2024

github-actions bot assigned chronolaw Oct 17, 2024

pull-request-size bot added size/S and removed size/XS labels Oct 21, 2024

chronolaw force-pushed the refactor/tuning_inc_sync branch from eada306 to 522dbf3 Compare October 22, 2024 00:16

pull-request-size bot added size/M and removed size/S labels Oct 22, 2024

chobits reviewed Oct 22, 2024

View reviewed changes

kong/db/strategies/off/init.lua Show resolved Hide resolved

chronolaw force-pushed the refactor/tuning_inc_sync branch from 427835a to a566ec6 Compare October 22, 2024 03:15

github-actions bot added the schema-change-noteworthy label Oct 22, 2024

chronolaw changed the title ~~refactor(clustering/rpc): tunings for rubustness~~ refactor(clustering/rpc): tunings and fixes for rubustness Oct 22, 2024

github-actions bot added the core/clustering label Oct 24, 2024

pull-request-size bot added size/L and removed size/M labels Oct 24, 2024

chronolaw force-pushed the refactor/tuning_inc_sync branch from 150fa2d to c625a60 Compare October 28, 2024 03:15

github-actions bot added the core/db/migrations label Oct 28, 2024

chobits force-pushed the refactor/tuning_inc_sync branch from 9281822 to b7963e4 Compare October 29, 2024 09:31

chronolaw force-pushed the refactor/tuning_inc_sync branch from b7963e4 to a7ba490 Compare October 29, 2024 10:53

chronolaw marked this pull request as ready for review October 29, 2024 11:57

team-gateway-bot added the author/community PRs from the open-source community (not Kong Inc) label Oct 29, 2024

chobits reviewed Oct 30, 2024

View reviewed changes

chobits approved these changes Oct 30, 2024

View reviewed changes

chronolaw removed the author/community PRs from the open-source community (not Kong Inc) label Oct 30, 2024

team-gateway-bot added the author/community PRs from the open-source community (not Kong Inc) label Oct 30, 2024

chronolaw commented Oct 30, 2024

View reviewed changes

chronolaw and others added 18 commits October 31, 2024 17:48

check latest_version and d.ws_id

a8d4479

fix dp do_sync()

8c1b950

fix declarative export sync

e227a34

typo fix

ff56567

check latest_version == ngx_null in CP

9386ffd

style fix

d061361

fix(clustring): inc sync add ws (#13790)

15624ec

* fix(clustering/sync): set ws_id in do_sync() * comments

fix(clustering): delta version is null (#13789)

f3ed16b

* fix(clustering/sync): delta.version may be null * isempty * default_ws_changed

rpc.lua style clean

3467d0c

sync comments clean

7294a20

refactor(clustering): change sync data structure (#13794)

cacf764

* rename db id to pk * change id to pk in Lua * change pk to json * schema:extract_pk_values

comments clean in rpc.lua

fc26c42

fix(incremental sync): added comment for 2 times do_sync (#13801)

60ef6c3

Co-authored-by: Datong Sun <[email protected]>

Refactor/rename delta row (#13807)

c657588

* change db filed name * rename in rpc.sync * rename in export

comments clean

8d34725

check delta.entity nil

d6aa49a

fix(incremental sync): fix cache_key handling for select_by_cache_key (…

8774284

…#13815) * fix(incremental sync): fix cache_key handling for select_by_cache_key * fix some comments * fix: remove unnecessary comment

refactor(dbless): clean logic of select_by_field (#13817)

63c17e2

* refactor(dbless): clean logic of selec_by_field * check schema_field

chronolaw force-pushed the refactor/tuning_inc_sync branch from f35c3d3 to 63c17e2 Compare October 31, 2024 09:48

fix(incremental sync): fixed ws_id processing in _set_entity_for_txn (#…

b37bdd1

…13816)

StarlightIbuki approved these changes Nov 1, 2024

View reviewed changes

AndyZhang0707 requested a review from bungle November 1, 2024 08:38

bungle approved these changes Nov 1, 2024

View reviewed changes

bungle merged commit 0072d7b into master Nov 1, 2024
32 checks passed

bungle deleted the refactor/tuning_inc_sync branch November 1, 2024 09:12

github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Nov 1, 2024

AndyZhang0707 removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Nov 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(clustering/rpc): tunings and fixes for rubustness #13771

refactor(clustering/rpc): tunings and fixes for rubustness #13771

chronolaw commented Oct 17, 2024 •

edited

Loading

chobits Oct 30, 2024

chobits Oct 30, 2024 •

edited

Loading

chobits Oct 30, 2024

chronolaw Oct 30, 2024

chronolaw Oct 30, 2024

chronolaw Oct 30, 2024

chronolaw Oct 30, 2024 •

edited

Loading

team-gateway-bot commented Nov 1, 2024

chronolaw commented Nov 1, 2024

refactor(clustering/rpc): tunings and fixes for rubustness #13771

refactor(clustering/rpc): tunings and fixes for rubustness #13771

Conversation

chronolaw commented Oct 17, 2024 • edited Loading

Summary

Checklist

Issue reference

chobits Oct 30, 2024

Choose a reason for hiding this comment

chobits Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

chobits Oct 30, 2024

Choose a reason for hiding this comment

chronolaw Oct 30, 2024

Choose a reason for hiding this comment

chronolaw Oct 30, 2024

Choose a reason for hiding this comment

chronolaw Oct 30, 2024

Choose a reason for hiding this comment

chronolaw Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

team-gateway-bot commented Nov 1, 2024

chronolaw commented Nov 1, 2024

chronolaw commented Oct 17, 2024 •

edited

Loading

chobits Oct 30, 2024 •

edited

Loading

chronolaw Oct 30, 2024 •

edited

Loading