Workaround hasura bugs, tweaks to get upserting working #698

ajvpot · 2022-06-01T20:46:25Z

Database

Migration

Alter tables in package schema to use unique constraints instead of just unique indexes.
- ref: On-conflict support for partial unique indexes hasura/graphql-engine#3666 (comment)
Migrate enums into public schema to work around Hasura doesn't find custom type declared in a schema other than public hasura/graphql-engine#4630

Hasura Metadata

Fix relationships for release dependencies and dependents, they were swapped.

`genqlient`

Enable pointers for optional fields to support nullability for custom types like google/uuid returning 00000-000... for MarshalText for nil uuids which is incompatible with hasura upsert.
Change npm enums to be correct case
Replace genqlient with a patched version that always uses omitempty until Global decorators Khan/genqlient#190 is aresolved.

Workflows

Split out github workflow into more steps so logs are less spammy

Misc

Cleaned up dead code
Merges npm and npm2 packages into npm, removes dead code.

Sample

{
  "data": {
    "package": [
      {
        "id": "cebd943c-f841-4ab5-9144-c0241da0c0b0",
        "name": "svelte",
        "package_maintainers": [
          {
            "maintainer": {
              "name": "paulbgd"
            }
          },
          {
            "maintainer": {
              "name": "rich_harris"
            }
          },
          {
            "maintainer": {
              "name": "conduitry"
            }
          }
        ],
        "releases": [
          {
            "id": "be1f97a4-d679-4f75-95b9-23b7ddef79bf",
            "version": "1.13.6",
            "release_dependencies": [
              {
                "package_name": "babel-core",
                "package_version_query": "^6.23.1",
                "is_dev": false
              },...
           "upstream_data": {
              "_id": "[email protected]",
              "nyc": {
                "exclude": [
                  "src/**/__test__.js"
                ],
                "include": [
                  "src/**/*.js"
                ]
              },
              "bugs": {
                "url": "https://github.com/sveltejs/svelte/issues"
              },
              "dist": {
                "shasum": "716a509f80a6786393883415d79eccff87464881",
                "tarball": "https://registry.npmjs.org/svelte/-/svelte-1.13.6.tgz"
              },
              "main": "compiler/svelte.js",
              "name": "svelte",
              "_from": ".",
              "babel": {...

lunatrace-by-lunasec

Build Snapshot Complete

View Full Report

Security Scan Findings

Showing 12 results.

Package Name	Versions	Severity	Locations
immer	8.0.1	Critical	1 location	Dismiss
merge	1.2.1	Critical	1 location	Dismiss
docs	0.0.0-use.local	Critical	1 location	Dismiss
pac-resolver	4.2.0	Critical	1 location	Dismiss
vm2	3.9.3	Critical	1 location	Dismiss
jsonpointer	5.0.0	Critical	1 location	Dismiss
through	2.3.8	Critical	1 location	Dismiss
ramda	0.24.1	Critical	1 location	Dismiss
shell-quote	1.7.2	Critical	1 location	Dismiss
ejs	3.1.6	Critical	1 location	Dismiss
eventsource	1.1.0	Critical	1 location	Dismiss
ejs	2.7.4	Critical	1 location	Dismiss

github-actions · 2022-06-01T22:14:47Z

Hasura Semantic Diff

Hasura config files have changed. This comment shows which fields have changed ignoring formatting.

Click to expand!

insert_permissions.0.permission.columns
  + one list entry added:
    - fetched_time

select_permissions.0.permission.columns
  + one list entry added:
    - fetched_time

update_permissions.0.permission.columns
  + one list entry added:
    - fetched_time


array_relationships.release_dependencies.using.foreign_key_constraint_on.column
  ± value change
    - dependency_release_id
    + release_id

array_relationships.release_dependents.using.foreign_key_constraint_on.column
  ± value change
    - release_id
    + dependency_release_id

insert_permissions.0.permission.columns
  + one list entry added:
    - fetched_time

select_permissions.0.permission.columns
  + one list entry added:
    - fetched_time

update_permissions.0.permission.columns
  + one list entry added:
    - fetched_time


diff --git a/lunatrace/bsl/hasura/migrations/lunatrace/1654107375714_move_enums/down.sql b/lunatrace/bsl/hasura/migrations/lunatrace/1654107375714_move_enums/down.sql
new file mode 100644
index 00000000..980c1a94
--- /dev/null
+++ b/lunatrace/bsl/hasura/migrations/lunatrace/1654107375714_move_enums/down.sql
@@ -0,0 +1,52 @@
+CREATE TYPE package.license_source AS ENUM ('manual', 'scan_repo', 'scan_binary', 'api_npm');
+
+CREATE TYPE package.package_manager AS ENUM ('npm');
+
+ALTER TABLE package.release_license
+    ALTER COLUMN source TYPE package.license_source USING source::TEXT::package.license_source;
+
+ALTER TABLE package.package
+    ALTER COLUMN package_manager TYPE package.package_manager USING package_manager::TEXT::package.package_manager;
+
+ALTER TABLE package.maintainer
+    ALTER COLUMN package_manager TYPE package.package_manager USING package_manager::TEXT::package.package_manager;
+
+DROP TYPE public.license_source;
+
+DROP TYPE public.package_manager;
+
+CREATE UNIQUE INDEX ON package.maintainer (package_manager, email);
+ALTER TABLE package.maintainer
+    DROP CONSTRAINT maintainer_package_manager_email_idx;
+
+CREATE UNIQUE INDEX ON package.package_maintainer (package_id, maintainer_id);
+ALTER TABLE package.package_maintainer
+    DROP CONSTRAINT package_maintainer_package_id_maintainer_id_idx;
+
+CREATE UNIQUE INDEX ON package.package (package_manager, custom_registry, name);
+ALTER TABLE package.package
+    DROP CONSTRAINT package_package_manager_custom_registry_name_idx;
+
+CREATE UNIQUE INDEX ON package.release (package_id, version);
+ALTER TABLE package.release
+    DROP CONSTRAINT release_package_id_version_idx;
+
+CREATE UNIQUE INDEX ON package.release_dependency (release_id, package_name, package_version_query);
+ALTER TABLE package.release_dependency
+    DROP CONSTRAINT release_dependency_release_id_package_name_package_version__idx;
+
+CREATE UNIQUE INDEX ON package.license (name);
+ALTER TABLE package.license
+    DROP CONSTRAINT license_name_idx;
+
+ALTER TABLE package.package
+    DROP COLUMN fetched_time;
+
+ALTER TABLE package.release
+    DROP COLUMN fetched_time;
+
+ALTER TABLE package.package
+    ALTER COLUMN custom_registry DROP NOT NULL;
+
+ALTER TABLE package.package
+    ALTER COLUMN custom_registry DROP DEFAULT;
diff --git a/lunatrace/bsl/hasura/migrations/lunatrace/1654107375714_move_enums/up.sql b/lunatrace/bsl/hasura/migrations/lunatrace/1654107375714_move_enums/up.sql
new file mode 100644
index 00000000..950ef398
--- /dev/null
+++ b/lunatrace/bsl/hasura/migrations/lunatrace/1654107375714_move_enums/up.sql
@@ -0,0 +1,52 @@
+CREATE TYPE public.license_source AS ENUM ('manual', 'scan_repo', 'scan_binary', 'api_npm');
+
+CREATE TYPE public.package_manager AS ENUM ('npm');
+
+ALTER TABLE package.release_license
+    ALTER COLUMN source TYPE public.license_source USING source::TEXT::public.license_source;
+
+ALTER TABLE package.package
+    ALTER COLUMN package_manager TYPE public.package_manager USING package_manager::TEXT::public.package_manager;
+
+ALTER TABLE package.maintainer
+    ALTER COLUMN package_manager TYPE public.package_manager USING package_manager::TEXT::public.package_manager;
+
+DROP TYPE package.license_source;
+
+DROP TYPE package.package_manager;
+
+DROP INDEX IF EXISTS package.release_dependency_release_id_package_name_package_version__idx;
+ALTER TABLE package.release_dependency
+    ADD CONSTRAINT release_dependency_release_id_package_name_package_version__idx UNIQUE (release_id, package_name, package_version_query);
+
+DROP INDEX IF EXISTS package.release_package_id_version_idx;
+ALTER TABLE package.release
+    ADD CONSTRAINT release_package_id_version_idx UNIQUE (package_id, version);
+
+DROP INDEX IF EXISTS package.package_maintainer_package_id_maintainer_id_idx;
+ALTER TABLE package.package_maintainer
+    ADD CONSTRAINT package_maintainer_package_id_maintainer_id_idx UNIQUE (package_id, maintainer_id);
+
+DROP INDEX IF EXISTS package.package_package_manager_custom_registry_name_idx;
+ALTER TABLE package.package
+    ADD CONSTRAINT package_package_manager_custom_registry_name_idx UNIQUE (package_manager, custom_registry, name);
+
+DROP INDEX IF EXISTS package.maintainer_package_manager_email_idx;
+ALTER TABLE package.maintainer
+    ADD CONSTRAINT maintainer_package_manager_email_idx UNIQUE (package_manager, email);
+
+DROP INDEX IF EXISTS package.license_name_idx;
+ALTER TABLE package.license
+    ADD CONSTRAINT license_name_idx UNIQUE (name);
+
+ALTER TABLE package.package
+    ADD fetched_time timestamptz;
+
+ALTER TABLE package.release
+    ADD fetched_time timestamptz;
+
+ALTER TABLE package.package
+    ALTER COLUMN custom_registry SET NOT NULL;
+
+ALTER TABLE package.package
+    ALTER COLUMN custom_registry SET DEFAULT '';

dev-lunatrace-by-lunasec

LUNATRACE IN DEV MODE

Build Snapshot Complete

View Full Report

Security Scan Findings

Showing 16 results.

Package Name	Versions	Severity	Locations
jackson-databind	2.6.4	Critical	1 location	Dismiss
redis	2.8.0	Critical	1 location	Dismiss
merge	1.2.1	Critical	1 location	Dismiss
docs	0.0.0-use.local	Critical	1 location	Dismiss
pac-resolver	4.2.0	Critical	1 location	Dismiss
immer	8.0.1	Critical	1 location	Dismiss
vm2	3.9.3	Critical	1 location	Dismiss
jsonpointer	5.0.0	Critical	1 location	Dismiss
through	2.3.8	Critical	1 location	Dismiss
ramda	0.24.1	Critical	1 location	Dismiss
shell-quote	1.7.2	Critical	1 location	Dismiss
minimist	0.0.8	Critical	1 location	Dismiss
minimist	1.2.5	Critical	2 locations	Dismiss
eventsource	1.1.0	Critical	1 location	Dismiss
ejs	3.1.6	Critical	1 location	Dismiss
ejs	2.7.4	Critical	1 location	Dismiss

factoidforrest

nice

Former-commit-id: 03a1637 Former-commit-id: f594c180ceed054cf8da5ce4ed9c28fb0331d0de

ajvpot added 2 commits June 1, 2022 13:44

enum migration

3e9ba01

regenerate

c45efaf

lunatrace-by-lunasec bot reviewed Jun 1, 2022

View reviewed changes

ajvpot added 6 commits June 1, 2022 14:07

fix unique constraints

e92983e

format migration

ff31f0d

go mod tidy

e78d2d6

replace genqlient

d14135a

patch genqlient

06ec802

it work

6604620

ajvpot added 4 commits June 1, 2022 15:14

echo the diff

14c79c8

split out workflow

3261494

split out workflow

a485212

go mod

3357578

ajvpot changed the title ~~Move enums to public schema~~ Workaround hasura bugs, tweaks to get upserting working Jun 1, 2022

ajvpot requested a review from breadchris June 1, 2022 22:32

ajvpot added 6 commits June 1, 2022 15:47

rm dead code

e2a2df6

merge npm2 and npm packages

e8e8017

change licensesource enum to string

1690247

regen

dd552e2

use fetched time

222beed

change gql query result

141b5af

dev-lunatrace-by-lunasec bot reviewed Jun 2, 2022

View reviewed changes

ajvpot added 7 commits June 1, 2022 18:08

create check constraint for null custom registries

d167947

fix wrong constraint

ce92e37

just make custom registry not nullable

5667e28

idempotency fix

cc5f41b

idempotency fix

bbfec8e

Merge branch 'master' into fix_enums_2

1bc16b5

enable nullability

90f6935

ajvpot added 5 commits June 1, 2022 19:53

switch back to google uuid

7e8ea3e

make it a crawler

afef899

todo

dcaea4d

add check for package fetch time for queue handler

652e994

optimize gql query

d30e024

factoidforrest approved these changes Jun 2, 2022

View reviewed changes

ajvpot added 3 commits June 2, 2022 15:10

code style settings

c0af693

don't specify tabs for python

f6598be

merge and regen

e1715bb

ajvpot merged commit 8a0ca43 into master Jun 2, 2022

freeqaz deleted the fix_enums_2 branch September 7, 2022 23:28

breadchris pushed a commit that referenced this pull request Dec 21, 2022

Workaround hasura bugs, tweaks to get upserting working (#698)

ea8dfa0

Former-commit-id: 03a1637 Former-commit-id: f594c180ceed054cf8da5ce4ed9c28fb0331d0de

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workaround hasura bugs, tweaks to get upserting working #698

Workaround hasura bugs, tweaks to get upserting working #698

ajvpot commented Jun 1, 2022 •

edited

Loading

lunatrace-by-lunasec bot left a comment •

edited

Loading

github-actions bot commented Jun 1, 2022 •

edited

Loading

dev-lunatrace-by-lunasec bot left a comment

factoidforrest left a comment

Workaround hasura bugs, tweaks to get upserting working #698

Workaround hasura bugs, tweaks to get upserting working #698

Conversation

ajvpot commented Jun 1, 2022 • edited Loading

Database

Migration

Hasura Metadata

genqlient

Workflows

Misc

Sample

lunatrace-by-lunasec bot left a comment • edited Loading

Choose a reason for hiding this comment

Build Snapshot Complete

Security Scan Findings

github-actions bot commented Jun 1, 2022 • edited Loading

Hasura Semantic Diff

dev-lunatrace-by-lunasec bot left a comment

Choose a reason for hiding this comment

Build Snapshot Complete

Security Scan Findings

factoidforrest left a comment

Choose a reason for hiding this comment

ajvpot commented Jun 1, 2022 •

edited

Loading

`genqlient`

lunatrace-by-lunasec bot left a comment •

edited

Loading

github-actions bot commented Jun 1, 2022 •

edited

Loading