feat: cio optimization #2550

sshanzel · 2024-12-17T15:08:06Z

CIO optimization by removing/registering users based on their current activity, along with downgrading their digest if we need to.

Once we've agreed on the processing code-wise, I will start working on the rollout plan, as it should basically use the same processing.

sshanzel · 2024-12-18T14:29:15Z

.infra/application.properties

@@ -9,7 +9,7 @@ debezium.source.database.password=%database_pass%
 debezium.source.database.dbname=%database_dbname%
 debezium.source.database.server.name=api
 debezium.source.table.include.list=public.comment,public.user_comment,public.comment_mention,public.source_request,public.post,public.user,public.post_report,public.source_feed,public.settings,public.reputation_event,public.submission,public.user_state,public.notification_v2,public.source_member,public.feature,public.source,public.post_mention,public.content_image,public.comment_report,public.user_post,public.banner,public.post_relation,public.marketing_cta,public.squad_public_request,public.user_streak,public.bookmark,public.user_company,public.source_report,public.user_top_reader,public.source_post_moderation
-debezium.source.column.exclude.list=public.post.tsv,public.post.placeholder,public.source.flags,public.user_top_reader.image
+debezium.source.column.exclude.list=public.post.tsv,public.post.placeholder,public.source.flags,public.user_top_reader.image,public.user.cioRegistered,public.user.acceptedMarketing,public.user.followingEmail,public.user.notificationEmail


We don't need to track changes from any of these columns.

Are you sure though?
For example acceptedMarketing should trigger userUpdated no? to sync them to CIO (when they not in your flow)

Ah, right. We have these props in CIO, they should definitely be synced. Let me check the worker.

Maybe I should not add these columns, but only when running the initial rollout.

Also please note, this list doesn't exclude these columns from triggering cdc but from being sent in the debezium payload. To ignore them you will need to add a filter, it's more complex process but you can find examples in this file

I think it does, didn't we fix that issue a while back by finally updating debezium?

It's not an issue just misunderstanding of what this config does from our side. But it's better to check with adhoc infra any change in debezium props

sshanzel · 2024-12-18T14:30:05Z

src/cio.ts

+  user: ChangeObject<User>,
+) => {
+  const { id } = user;
+  const changed = JSON.parse(JSON.stringify(user));


Not sure if we have a util function that can turn regular objects into ChangeObject type.

structuredClone maybe?

Right. Yeah, I think that's better than json-parse-stringify.

Edit: introduced a function instead that makes the same parse and stringify of our objects to pubsub. Structured clone returns Date object still for which we need either a string or number, etc.

src/common/googleCloud.ts

src/common/mailing.ts

src/cron/validateActiveUsers.ts

idoshamun

we can't commit the initial rollout csv yet. we need to do it only when we're ready

sshanzel · 2025-01-02T08:23:23Z

we can't commit the initial rollout csv yet. we need to do it only when we're ready

Yes, this was just a sample. Just 1k records to test it.

sshanzel · 2025-01-02T08:24:41Z

I will add more details in the rollout plan confluence page on how to proceed.

idoshamun

Looks good but let's not merge until we meet to do the initial rollout

rebelchris

Overall looks good to me, just monitor very closely I guess.

pulumi · 2025-01-07T19:03:28Z

🍹 The Update (preview) for dailydotdev/api/prod was successful.

Resource Changes

    Name                                            Type                           Operation
~   vpc-native-update-highlighted-views-cron        kubernetes:batch/v1:CronJob    update
~   vpc-native-private-deployment                   kubernetes:apps/v1:Deployment  update
~   vpc-native-clean-zombie-users-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-deployment                           kubernetes:apps/v1:Deployment  update
~   vpc-native-calculate-top-readers-cron           kubernetes:batch/v1:CronJob    update
~   vpc-native-update-views-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-hourly-notification-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-sync-subscription-with-cio-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-generate-search-invites-cron         kubernetes:batch/v1:CronJob    update
~   vpc-native-update-source-tag-view-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-update-trending-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-user-companies-cron     kubernetes:batch/v1:CronJob    update
~   vpc-native-update-tags-str-cron                 kubernetes:batch/v1:CronJob    update
+-  vpc-native-debezium-props                       kubernetes:core/v1:Secret      create-replacement
~   vpc-native-update-tag-recommendations-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-images-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-daily-digest-cron                    kubernetes:batch/v1:CronJob    update
+-  vpc-native-debezium-deployment                  kubernetes:apps/v1:Deployment  create-replacement
~   vpc-native-personalized-digest-deployment       kubernetes:apps/v1:Deployment  update
~   vpc-native-update-current-streak-cron           kubernetes:batch/v1:CronJob    update
-   vpc-native-api-migration-fea8fa4d               kubernetes:batch/v1:Job        delete
~   vpc-native-check-analytics-report-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-bg-deployment                        kubernetes:apps/v1:Deployment  update
+   vpc-native-api-migration-f14825cf               kubernetes:batch/v1:Job        create
~   vpc-native-generic-referral-reminder-cron       kubernetes:batch/v1:CronJob    update
~   vpc-native-temporal-deployment                  kubernetes:apps/v1:Deployment  update
~   vpc-native-ws-deployment                        kubernetes:apps/v1:Deployment  update
~   vpc-native-update-source-public-threshold-cron  kubernetes:batch/v1:CronJob    update

sshanzel added 3 commits December 17, 2024 22:49

feat: cio optimization

717fbc3

feat: retry policy

92971da

refactor: function name

bd96bc1

This comment has been minimized.

Sign in to view

sshanzel added 3 commits December 18, 2024 16:07

feat: bq query

3b66822

feat: query from bq

b9048d8

test: cron job

a9adb02

This comment has been minimized.

Sign in to view

sshanzel added 2 commits December 18, 2024 22:16

chore: prepare cron job

f2d68b0

chore: no registered test

bb510ff

This comment has been minimized.

Sign in to view

fix: missing await

6af995b

This comment has been minimized.

Sign in to view

sshanzel commented Dec 18, 2024

View reviewed changes

refactor: ordering of variables

0c0e4a3

This comment has been minimized.

Sign in to view

Merge branch 'main' into MI-691

a1c8939

sshanzel marked this pull request as ready for review December 18, 2024 14:38

sshanzel requested review from capJavert and a team as code owners December 18, 2024 14:38

sshanzel requested review from rebelchris, omBratteng, ilasw, nensidosari and AmarTrebinjac December 18, 2024 14:38

This comment has been minimized.

Sign in to view

refactor: code cleanup

4df7881

This comment has been minimized.

Sign in to view

idoshamun reviewed Dec 22, 2024

View reviewed changes

src/common/googleCloud.ts Show resolved Hide resolved

idoshamun reviewed Dec 22, 2024

View reviewed changes

src/common/mailing.ts Outdated Show resolved Hide resolved

idoshamun reviewed Dec 22, 2024

View reviewed changes

src/cron/validateActiveUsers.ts Outdated Show resolved Hide resolved

sshanzel added 2 commits January 2, 2025 12:10

fix: missing transaction

ec456da

refactor: unnecessary limit

772d21c

This comment has been minimized.

Sign in to view

Merge branch 'main' into MI-691

9d7c6f4

This comment has been minimized.

Sign in to view

feat: initial rollout through csv

dc0aa64

This comment has been minimized.

Sign in to view

idoshamun reviewed Jan 2, 2025

View reviewed changes

chore: minor code cleanups

ac4b96e

This comment has been minimized.

Sign in to view

chore: uncomment ignoring of cio registered

f4f036c

This comment has been minimized.

Sign in to view

chore: unnecessary csv file

6cdf31c

This comment has been minimized.

Sign in to view

sshanzel requested a review from idoshamun January 2, 2025 09:19

idoshamun approved these changes Jan 2, 2025

View reviewed changes

rebelchris approved these changes Jan 3, 2025

View reviewed changes

fix: condition to not process anything related to cio registered

47a6b42

This comment has been minimized.

Sign in to view

Merge branch 'main' into MI-691

279773e

This comment has been minimized.

Sign in to view

Merge branch 'main' into MI-691

edbace0

sshanzel merged commit bca573b into main Jan 8, 2025
8 checks passed

sshanzel deleted the MI-691 branch January 8, 2025 08:05

feat: cio optimization #2550

feat: cio optimization #2550

Uh oh!

Conversation

sshanzel commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sshanzel Dec 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

idoshamun left a comment

Choose a reason for hiding this comment

Uh oh!

sshanzel commented Jan 2, 2025

Uh oh!

sshanzel commented Jan 2, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

idoshamun left a comment

Choose a reason for hiding this comment

Uh oh!

rebelchris left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

pulumi bot commented Jan 7, 2025

Resource Changes

Uh oh!

Uh oh!

Uh oh!

sshanzel commented Dec 17, 2024 •

edited

Loading

sshanzel Dec 19, 2024 •

edited

Loading