WIP: Compute balances in database #247

FriesischScott · 2025-06-09T11:24:30Z

As discussed in #234 the goal of this PR is to drop the separate balance tables for users and groups and instead compute the balances from all expenses in the database.

I agree with @krokosik that we need to be certain we can handle the required loads for large instances with lots of expenses but I think postgresql will be up for the task.

This is a list of the required balances:

Friend balances
Overall group balances for one user
All balances for a specific group

here I've implemented the second and third and I think @krokosik has started working on the first.

Because prisma can't handle the necessary computations through it's javascript interface I've chosen to use typedSQL to write native SQL queries and still benifit from prisma's typing capabilities.

One minor downside is that typedSQL currently requires a live database connection to generate the types. However with prisma 7 the location of the generated client will move out of node_modules and we will have to set an output directory like this.

We could then commit the generated sql types in the sql subfolder of the prisma client and only generate the regular client during build. This should already be possible now (only in prisma 7 it's mandatory) and I'm working on setting it up.

This part no longer relies on the GroupBalances table

Uses a prisma typedSQL query

Only the `typedSQL` files are committed. The client is excluded from eslint.

prisma/seed.ts

prisma/schema.prisma

guenzd · 2025-06-09T15:47:19Z

I get this error when I try to run with existing data:

FriesischScott · 2025-06-09T15:48:47Z

I get this error when I try to run with existing data:

Yes, I haven't finished fixing a few type issues. You should be able to run it with pnpm dev but build is not yet working.

guenzd · 2025-06-09T16:06:28Z

I get this error when I try to run with existing data:

Yes, I haven't finished fixing a few type issues. You should be able to run it with pnpm dev but build is not yet working.

Then there are more runtime issues :/

FriesischScott · 2025-06-09T23:44:14Z

I get this error when I try to run with existing data:

Yes, I haven't finished fixing a few type issues. You should be able to run it with pnpm dev but build is not yet working.

Then there are more runtime issues :/

Can you try again now?

Turns out I wasn't up to date on the migrations and for some reason when the column is bigint the sum is returned as numeric. I'm now casting explicitly to bigint.

1000 users in 101 groups with 10k expenses each

Type Hash for faster lookup with =

FriesischScott · 2025-06-18T12:32:06Z

I've updated the seed file to create 1000 users and 101 groups with 10k expenses each. The largest group has 30 members all the others have 10. It should give us a better understanding of the performance.

It does take quite a while to create all the expenses but it's a temporary change anyway. If we merge we can just revert the file completely.

krokosik · 2025-06-18T19:00:06Z

Fantastic work, thank you 🙌
I know I promised to take a swing at it this week, but unfortunately it's not likely to happen due to other obligations :/

FriesischScott · 2025-06-19T08:35:51Z

Fantastic work, thank you 🙌 I know I promised to take a swing at it this week, but unfortunately it's not likely to happen due to other obligations :/

No problem at all. Take your time. Better not to rush things :)

krokosik

Very sorry for taking so long with the review. I had to take care of other stuff as well as more pressing current issues. I requested some changes that would minimize this PR further as I believe such risky changes should be as easy to review as possible. I ran some initial tests on my machine and it is looking good, so I would like to proceed with this, but only after they have been resolved and the branch gets rebased onto main. Furthermore, if you could give me write access to your fork, I will implement friend balances as well :)

krokosik · 2025-07-07T15:48:04Z

prisma/seed.ts

The original seed script is useful for local dev work, while this one takes a long time to execute. Please move it to a separate file

My plan was to only use this as and intermediate script and revert to the original before we merge. If you think it's worth keeping around I'll move it to a separate script.

prisma/sql/getAllBalancesForGroup.sql

krokosik · 2025-07-07T15:52:25Z

src/lib/simplify.test.ts

+const sortByIds = (a: getAllBalancesForGroup.Result, b: getAllBalancesForGroup.Result) => {
+  if (a.paidBy === b.paidBy) {
+    return a.borrowedBy - b.borrowedBy;
  }
-  return a.userId - b.userId;
+  return a.paidBy - b.paidBy;


I would prefer to keep this PR as minimal as possible. As such, please keep the property names identical and create a type GroupBalance = getAllBalancesForGroup.Result alias

Just to confirm, you want me to keep the existing firendId including the typo?

While I agree the naming even without the typo is quite unfortunate, I would prefer to make such a critical PR as small and easy to review as possible. We can think about another PR afterwards.

krokosik · 2025-07-07T15:54:40Z

src/server/api/routers/group.ts

+    const groupBalances = await ctx.db.$queryRawTyped(getGroupsWithBalances(ctx.session.user.id));
+
+    const _groups = groupBalances
+      .map((b) => {
+        return {
+          id: b.id,
+          name: b.name,
+        };
+      })
+      .filter((obj, index, self) => index === self.findIndex((t) => t.id === obj.id));
+
+    const groupsWithBalances = _groups.map((group) => {
+      const balancesForGroup: Record<string, bigint> = {};
+      groupBalances
+        .filter((b) => {
+          return b.id == group.id;
+        })
+        .forEach((b) => {
+          if (b.currency != null && b.balance != null) {
+            balancesForGroup[b.currency] = b.balance;
+          }
+        });


While we remain on the dev branch, I propose the following sanity check:
utilize both the legacy and new method and compare their results. When a mismatch is detected, throw an Error. We will remove it before creating a release.

krokosik · 2025-07-07T16:10:35Z

Oh and one more thing, this change would break the current splitwise import functionality, as it does not import expenses as of now. #207 needs to be implemented first and as such I don't expect to go ahead with this PR for 1.5 release

FriesischScott · 2025-07-09T18:21:22Z

Oh and one more thing, this change would break the current splitwise import functionality, as it does not import expenses as of now. #207 needs to be implemented first and as such I don't expect to go ahead with this PR for 1.5 release

No problem. More time to polish.

alexanderwassbjer · 2025-08-05T19:53:25Z

This is really good work! Calculating it based on the expenses rather then the balance table is amazing! 👏

FriesischScott · 2025-10-09T00:11:31Z

Thanks for the ping. I'm of course happy to finish this.

I'll start by merging in all the changes and fixing the conflicts. Shouldn't take more than a day or two until I have something up to date and working.

krokosik · 2025-10-09T09:20:07Z

Okay nice, since I would like to include my changes as well in this PR and have some thoughts I would like to share, would you like to connect on Discord to chat more conveniently?

FriesischScott · 2025-10-11T14:30:22Z

I've fixed the conflicts and properly formatted the SQL files. So far everything still seems to work as intended. The check currently fails because Github Actions doesn't allow to change the command for the postgres container to load the pg_cron extension.

The easiest fix would probably be if you could update the custom postgres image to contain the shared_preload_libraries.

FriesischScott · 2025-10-11T14:33:17Z

Okay nice, since I would like to include my changes as well in this PR and have some thoughts I would like to share, would you like to connect on Discord to chat more conveniently?

Yeah, I think that would help. I'll email you my username.

alexanderwassbjer · 2025-10-11T18:23:21Z

I've fixed the conflicts and properly formatted the SQL files. So far everything still seems to work as intended. The check currently fails because Github Actions doesn't allow to change the command for the postgres container to load the pg_cron extension.

The easiest fix would probably be if you could update the custom postgres image to contain the shared_preload_libraries.

Good work! Looking forward to this change.

krokosik · 2025-10-14T16:54:23Z

@FriesischScott I've sent you a friend request. I am now working on an extensive seeding script with fakerjs and some sociale network graph generation algorithms. It would be nice to have a deterministic dev db and we could establish some data consistency check on it and more importantly compare the existing balance tables with the query calculated ones.

FriesischScott · 2025-10-14T16:56:12Z

@FriesischScott I've sent you a friend request. I am now working on an extensive seeding script with fakerjs and some sociale network graph generation algorithms. It would be nice to have a deterministic dev db and we could establish some data consistency check on it and more importantly compare the existing balance tables with the query calculated ones.

Good plan!

krokosik · 2025-10-14T16:57:19Z

The build error seems to stem from the fact you don't provide startup commands for cron. See the repo compose files for reference.

krokosik · 2025-10-16T18:03:24Z

@FriesischScott The seed script has been merged :)
It should serve as a replacement over your largeSeed file and is more nuanced with edited and deleted expenses.
You can tweak the number of users and groups to get more expenses and I recommend checking out the CONTRIBUTING.md. Experiment with the params by calling pnpm tsx src/dummies to get a SEED_STATISTICS.md report until you have some desirable data.

The next step would be to implement checks. Currently we still create balances, which also prevents us from inserting expenses in parallel as all the transactions create deadlocks. I would propose writing some consistency checks, that would compare as many cases of query calculated balances to the table ones as possible, preferably all of them.

Once that is done, we can plan next steps, as the situation with direct user-user balances requires us to think through the transition from current Total balances to user-user ones.

FriesischScott added 10 commits June 5, 2025 23:52

Add typed query to get groups with balances

a9ed276

Return groups without expenses

beb89da

Order groups descending by newest

df6a7ad

Compute blances for group overview in db

ab4c7ee

This part no longer relies on the GroupBalances table

Compute balance list in db

56af112

Uses a prisma typedSQL query

Merge branch 'main' into sql-balances

de25193

Include group id in all balances

24d1b99

Fix linting

70e58c6

Move prisma client to src/prisma/client

a40ca8c

Only the `typedSQL` files are committed. The client is excluded from eslint.

Fix prettier warning

cee753c

krokosik reviewed Jun 9, 2025

View reviewed changes

prisma/seed.ts Outdated Show resolved Hide resolved

krokosik reviewed Jun 9, 2025

View reviewed changes

prisma/schema.prisma Outdated Show resolved Hide resolved

FriesischScott added 14 commits June 9, 2025 18:26

Use path alias to move prisma client

23fece2

gitignore prisma client files

b3002be

Fix sql imports

20fb843

Set prisma client output

44a3bdd

Run lint --fix

a36344d

Fix type issues

cd32056

Fix prettier warnings

c619e26

Add postgresql service container

646c2e0

Run migrations before generating sql types

7de67eb

Merge branch 'main' into sql-balances

bb996e9

Remove sql types

fa22c43

Fix workflow file

1ff2229

Fix bigint conversions

6ff9a98

Fix lint

0e5b29c

Exclude deleted expenses from group balance

c26dc4a

FriesischScott mentioned this pull request Jun 10, 2025

New group members can't be added to existing expenses. #216

Closed

FriesischScott added 3 commits June 18, 2025 13:34

Update seed

06eea78

1000 users in 101 groups with 10k expenses each

Add userid index to ExpenseParticipant

bca6c9d

Type Hash for faster lookup with =

Merge branch 'main' into sql-balances

4dc3d38

FriesischScott marked this pull request as ready for review June 18, 2025 12:29

Fix prettier warnings

1f330a5

krokosik requested changes Jul 7, 2025

View reviewed changes

krokosik changed the title ~~Compute balances in database~~ WIP: Compute balances in database Sep 5, 2025

krokosik marked this pull request as draft September 5, 2025 11:55

krokosik mentioned this pull request Oct 7, 2025

160 recurring expense #420

Merged

FriesischScott added 3 commits October 9, 2025 23:24

Merge remote-tracking branch 'upstream/main' into sql-balances

3270d55

Format SQL queries

510df2e

Switch service container to ossapps/postgres

391f161

krokosik mentioned this pull request Oct 19, 2025

Currency handling improvements #437

Merged

5 tasks

Uh oh!

WIP: Compute balances in database #247

Are you sure you want to change the base?

WIP: Compute balances in database #247

Uh oh!

Conversation

FriesischScott commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

guenzd commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 9, 2025

Uh oh!

guenzd commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krokosik commented Jun 18, 2025

Uh oh!

FriesischScott commented Jun 19, 2025

Uh oh!

krokosik left a comment

Choose a reason for hiding this comment

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

FriesischScott Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

FriesischScott Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik commented Jul 7, 2025

Uh oh!

FriesischScott commented Jul 9, 2025

Uh oh!

alexanderwassbjer commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FriesischScott commented Oct 9, 2025

Uh oh!

krokosik commented Oct 9, 2025

Uh oh!

FriesischScott commented Oct 11, 2025

Uh oh!

FriesischScott commented Oct 11, 2025

Uh oh!

alexanderwassbjer commented Oct 11, 2025

Uh oh!

krokosik commented Oct 14, 2025

Uh oh!

FriesischScott commented Oct 14, 2025

Uh oh!

krokosik commented Oct 14, 2025

Uh oh!

krokosik commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FriesischScott commented Jun 18, 2025 •

edited

Loading

alexanderwassbjer commented Aug 5, 2025 •

edited

Loading