Skip to content

Refine Inspect queries and add CSVQ support for rule parsing #3595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: develop
Choose a base branch
from
Open

Refine Inspect queries and add CSVQ support for rule parsing #3595

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
0b2c5c8
add simple rules for parsing report output
encima Apr 25, 2025
c2136bd
added csvq driver and normalised column names across queries
encima Apr 25, 2025
9249429
modify to use table output
encima Apr 25, 2025
5edbff2
modified seq_scans output to remove reserved words, added some rules
encima Apr 29, 2025
86ba7c6
remove more keywords from queries and improve rule output
encima Apr 30, 2025
c370a61
use null string for empty responses
encima Apr 30, 2025
bd064bf
update bloat query
encima May 6, 2025
7538537
combined table and index queries to index_stats and table_stats
encima May 6, 2025
d891175
combine report queries to creats rollup for tables, indexes, roles an…
encima May 7, 2025
baad4a1
cleaned up bloat query output
encima May 9, 2025
2831cf9
rebasing new sql commands
encima May 12, 2025
0ba7cb0
update rules for new inspect commands
encima May 12, 2025
4c88244
update tests for all queries in inspect dir
encima May 12, 2025
e266458
updated tests to ignore internal schemas and removed unused sql queries
encima May 26, 2025
9f4a035
comments addressed: fix in rules for field names and formatted. error…
encima May 29, 2025
42ac8ff
address review comments, re-add deprecated commands and add notice
encima Jun 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
222 changes: 106 additions & 116 deletions cmd/inspect.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,36 @@
package cmd

import (
"database/sql"
"fmt"
"os"
"os/signal"
"path/filepath"
"time"

"github.com/spf13/afero"
"github.com/spf13/cobra"
"github.com/supabase/cli/internal/inspect/bloat"
"github.com/supabase/cli/internal/inspect/blocking"
"github.com/supabase/cli/internal/inspect/cache"
"github.com/supabase/cli/internal/inspect/role_stats"

"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/internal/utils/flags"

_ "github.com/mithrandie/csvq-driver"
"github.com/pelletier/go-toml/v2"
"github.com/supabase/cli/internal/inspect"
"github.com/supabase/cli/internal/inspect/calls"
"github.com/supabase/cli/internal/inspect/index_sizes"
"github.com/supabase/cli/internal/inspect/index_usage"
"github.com/supabase/cli/internal/inspect/db_stats"
"github.com/supabase/cli/internal/inspect/index_stats"
"github.com/supabase/cli/internal/inspect/locks"
"github.com/supabase/cli/internal/inspect/long_running_queries"
"github.com/supabase/cli/internal/inspect/outliers"
"github.com/supabase/cli/internal/inspect/replication_slots"
"github.com/supabase/cli/internal/inspect/role_configs"
"github.com/supabase/cli/internal/inspect/role_connections"
"github.com/supabase/cli/internal/inspect/seq_scans"
"github.com/supabase/cli/internal/inspect/table_index_sizes"
"github.com/supabase/cli/internal/inspect/table_record_counts"
"github.com/supabase/cli/internal/inspect/table_sizes"
"github.com/supabase/cli/internal/inspect/total_index_size"
"github.com/supabase/cli/internal/inspect/total_table_sizes"
"github.com/supabase/cli/internal/inspect/unused_indexes"
"github.com/supabase/cli/internal/inspect/table_stats"

"github.com/supabase/cli/internal/inspect/vacuum_stats"
"github.com/supabase/cli/internal/migration/list"
)

var (
Expand All @@ -51,11 +50,11 @@ var (
},
}

inspectCacheHitCmd = &cobra.Command{
Use: "cache-hit",
inspectDBStatsCmd = &cobra.Command{
Use: "db-stats",
Short: "Show cache hit rates for tables and indices",
RunE: func(cmd *cobra.Command, args []string) error {
return cache.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
return db_stats.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

Expand All @@ -67,14 +66,6 @@ var (
},
}

inspectIndexUsageCmd = &cobra.Command{
Use: "index-usage",
Short: "Show information about the efficiency of indexes",
RunE: func(cmd *cobra.Command, args []string) error {
return index_usage.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectLocksCmd = &cobra.Command{
Use: "locks",
Short: "Show queries which have taken out an exclusive lock on a relation",
Expand Down Expand Up @@ -107,59 +98,11 @@ var (
},
}

inspectTotalIndexSizeCmd = &cobra.Command{
Use: "total-index-size",
Short: "Show total size of all indexes",
RunE: func(cmd *cobra.Command, args []string) error {
return total_index_size.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better hide deprecated commands instead so it doesn't break existing users.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we hide these typically? Do you return a recommendation of the new command or call the new command and print a deprecation warning?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cli/cmd/db.go

Line 171 in b2ac658

Deprecated: "use \"db diff --use-migra --linked\" instead.\n",

Depends on the situation. I think in this case, it is fine to invoke new command and print warning.

If there's no close substitute, then we need to keep the old code around as well.


inspectIndexSizesCmd = &cobra.Command{
Use: "index-sizes",
Short: "Show index sizes of individual indexes",
RunE: func(cmd *cobra.Command, args []string) error {
return index_sizes.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectTableSizesCmd = &cobra.Command{
Use: "table-sizes",
Short: "Show table sizes of individual tables without their index sizes",
RunE: func(cmd *cobra.Command, args []string) error {
return table_sizes.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectTableIndexSizesCmd = &cobra.Command{
Use: "table-index-sizes",
Short: "Show index sizes of individual tables",
RunE: func(cmd *cobra.Command, args []string) error {
return table_index_sizes.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectTotalTableSizesCmd = &cobra.Command{
Use: "total-table-sizes",
Short: "Show total table sizes, including table index sizes",
RunE: func(cmd *cobra.Command, args []string) error {
return total_table_sizes.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectUnusedIndexesCmd = &cobra.Command{
Use: "unused-indexes",
Short: "Show indexes with low usage",
inspectIndexStatsCmd = &cobra.Command{
Use: "index-stats",
Short: "Show combined index size, usage percent, scan counts, and unused status",
RunE: func(cmd *cobra.Command, args []string) error {
return unused_indexes.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectSeqScansCmd = &cobra.Command{
Use: "seq-scans",
Short: "Show number of sequential scans recorded against all tables",
RunE: func(cmd *cobra.Command, args []string) error {
return seq_scans.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
return index_stats.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

Expand All @@ -171,14 +114,6 @@ var (
},
}

inspectTableRecordCountsCmd = &cobra.Command{
Use: "table-record-counts",
Short: "Show estimated number of rows per table",
RunE: func(cmd *cobra.Command, args []string) error {
return table_record_counts.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectBloatCmd = &cobra.Command{
Use: "bloat",
Short: "Estimates space allocated to a relation that is full of dead tuples",
Expand All @@ -187,27 +122,27 @@ var (
},
}

inspectVacuumStatsCmd = &cobra.Command{
Use: "vacuum-stats",
Short: "Show statistics related to vacuum operations per table",
inspectRoleStatsCmd = &cobra.Command{
Use: "role-stats",
Short: "Show information about roles on the database",
RunE: func(cmd *cobra.Command, args []string) error {
return vacuum_stats.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
return role_stats.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectRoleConfigsCmd = &cobra.Command{
Use: "role-configs",
Short: "Show configuration settings for database roles when they have been modified",
inspectVacuumStatsCmd = &cobra.Command{
Use: "vacuum-stats",
Short: "Show statistics related to vacuum operations per table",
RunE: func(cmd *cobra.Command, args []string) error {
return role_configs.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
return vacuum_stats.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

inspectRoleConnectionsCmd = &cobra.Command{
Use: "role-connections",
Short: "Show number of active connections for all database roles",
inspectTableStatsCmd = &cobra.Command{
Use: "table-stats",
Short: "Show combined table size, index size, and estimated row count",
RunE: func(cmd *cobra.Command, args []string) error {
return role_connections.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
return table_stats.Run(cmd.Context(), flags.DbConfig, afero.NewOsFs())
},
}

Expand All @@ -218,48 +153,103 @@ var (
Short: "Generate a CSV output for all inspect commands",
RunE: func(cmd *cobra.Command, args []string) error {
ctx := cmd.Context()
if len(outputDir) == 0 {
defaultPath := filepath.Join(utils.CurrentDirAbs, "report")
title := fmt.Sprintf("Enter a directory to save output files (or leave blank to use %s): ", utils.Bold(defaultPath))
if dir, err := utils.NewConsole().PromptText(ctx, title); err != nil {
return err
} else if len(dir) == 0 {
outputDir = defaultPath
}
if err := inspect.Report(ctx, outputDir, flags.DbConfig, afero.NewOsFs()); err != nil {
return err
}
return inspect.Report(ctx, outputDir, flags.DbConfig, afero.NewOsFs())
return printReportSummary(outputDir)
},
}
)

// Load rules file at runtime (tools/inspect_rules.toml)

// Rule defines a validation rule for a CSV file
type Rule struct {
Query string `toml:"query"`
Pass string `toml:"pass"`
Fail string `toml:"fail"`
Name string `toml:"name"`
}

// Config holds all rules
type Config struct {
Rules []Rule `toml:"rule"`
}

func init() {
inspectFlags := inspectCmd.PersistentFlags()
inspectFlags.String("db-url", "", "Inspect the database specified by the connection string (must be percent-encoded).")
inspectFlags.Bool("linked", true, "Inspect the linked project.")
inspectFlags.Bool("local", false, "Inspect the local database.")
inspectCmd.MarkFlagsMutuallyExclusive("db-url", "linked", "local")
inspectDBCmd.AddCommand(inspectCacheHitCmd)
inspectDBCmd.AddCommand(inspectReplicationSlotsCmd)
inspectDBCmd.AddCommand(inspectIndexUsageCmd)
inspectDBCmd.AddCommand(inspectIndexStatsCmd)
inspectDBCmd.AddCommand(inspectLocksCmd)
inspectDBCmd.AddCommand(inspectBlockingCmd)
inspectDBCmd.AddCommand(inspectOutliersCmd)
inspectDBCmd.AddCommand(inspectCallsCmd)
inspectDBCmd.AddCommand(inspectTotalIndexSizeCmd)
inspectDBCmd.AddCommand(inspectIndexSizesCmd)
inspectDBCmd.AddCommand(inspectTableSizesCmd)
inspectDBCmd.AddCommand(inspectTableIndexSizesCmd)
inspectDBCmd.AddCommand(inspectTotalTableSizesCmd)
inspectDBCmd.AddCommand(inspectUnusedIndexesCmd)
inspectDBCmd.AddCommand(inspectSeqScansCmd)
inspectDBCmd.AddCommand(inspectLongRunningQueriesCmd)
inspectDBCmd.AddCommand(inspectTableRecordCountsCmd)
inspectDBCmd.AddCommand(inspectBloatCmd)
inspectDBCmd.AddCommand(inspectVacuumStatsCmd)
inspectDBCmd.AddCommand(inspectRoleConfigsCmd)
inspectDBCmd.AddCommand(inspectRoleConnectionsCmd)
inspectDBCmd.AddCommand(inspectTableStatsCmd)
inspectDBCmd.AddCommand(inspectRoleStatsCmd)
inspectDBCmd.AddCommand(inspectDBStatsCmd)
inspectCmd.AddCommand(inspectDBCmd)
reportCmd.Flags().StringVar(&outputDir, "output-dir", "", "Path to save CSV files in")
inspectCmd.AddCommand(reportCmd)
rootCmd.AddCommand(inspectCmd)
}

func printReportSummary(outDir string) error {
// point to the date-based subdirectory
date := time.Now().Format("2006-01-02")
outDir = filepath.Join(outDir, date)
// Load rules from tools/inspect_rules.toml
data, err := os.ReadFile(filepath.Join(utils.CurrentDirAbs, "tools", "inspect_rules.toml"))
if err != nil {
return err
}
var cfg Config
if err := toml.Unmarshal(data, &cfg); err != nil {
return err
}
// Open csvq database rooted at the output directory
db, err := sql.Open("csvq", outDir)
if err != nil {
return err
}
defer db.Close()

// Build report summary table
table := "RULE|STATUS|MATCHES\n|-|-|-|\n"

// find matching rule
var status string
for i := range cfg.Rules {
r := &cfg.Rules[i]
name := r.Name

row := db.QueryRow(r.Query)
var match sql.NullString

if err := row.Scan(&match); err != nil {
if err == sql.ErrNoRows {
status = r.Pass
} else {
status = err.Error()
}
} else {
if !match.Valid || match.String == "" {
status = r.Pass
} else {
status = r.Fail
}
}
matchStr := "-"
if match.Valid {
matchStr = match.String
}
table += fmt.Sprintf("|`%s`|`%s`|`%s`|\n", name, status, matchStr)
}
return list.RenderTable(table)
}
7 changes: 6 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,11 @@ require (
github.com/jackc/pgtype v1.14.4
github.com/jackc/pgx/v4 v4.18.3
github.com/joho/godotenv v1.5.1
github.com/mithrandie/csvq-driver v1.7.0
github.com/muesli/reflow v0.3.0
github.com/oapi-codegen/oapi-codegen/v2 v2.4.1
github.com/oapi-codegen/runtime v1.1.1
github.com/pelletier/go-toml/v2 v2.2.4
github.com/slack-go/slack v0.16.0
github.com/spf13/afero v1.14.0
github.com/spf13/cobra v1.9.1
Expand Down Expand Up @@ -222,6 +224,10 @@ require (
github.com/microcosm-cc/bluemonday v1.0.27 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/hashstructure/v2 v2.0.2 // indirect
github.com/mithrandie/csvq v1.18.1 // indirect
github.com/mithrandie/go-file/v2 v2.1.0 // indirect
github.com/mithrandie/go-text v1.6.0 // indirect
github.com/mithrandie/ternary v1.1.1 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
Expand All @@ -239,7 +245,6 @@ require (
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pjbgf/sha1cd v0.3.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
Expand Down
10 changes: 10 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,16 @@ github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrk
github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4=
github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE=
github.com/mitchellh/mapstructure v0.0.0-20150613213606-2caf8efc9366/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mithrandie/csvq v1.18.1 h1:f7NB2scbb7xx2ffPduJ2VtZ85RpWXfvanYskAkGlCBU=
github.com/mithrandie/csvq v1.18.1/go.mod h1:MRJj7AtcXfk7jhNGxLuJGP3LORmh4lpiPWxQ7VyCRn8=
github.com/mithrandie/csvq-driver v1.7.0 h1:ejiavXNWwTPMyr3fJFnhcqd1L1cYudA0foQy9cZrqhw=
github.com/mithrandie/csvq-driver v1.7.0/go.mod h1:HcN3xL9UCJnBYA/AIQOOB/KlyfXAiYr5yxDmiwrGk5o=
github.com/mithrandie/go-file/v2 v2.1.0 h1:XA5Tl+73GXMDvgwSE3Sg0uC5FkLr3hnXs8SpUas0hyg=
github.com/mithrandie/go-file/v2 v2.1.0/go.mod h1:9YtTF3Xo59GqC1Pxw6KyGVcM/qubAMlxVsqI/u9r++c=
github.com/mithrandie/go-text v1.6.0 h1:8gOXTMPbMY8DJbKMTv8kHhADcJlDWXqS/YQH4SyWO6s=
github.com/mithrandie/go-text v1.6.0/go.mod h1:xCgj1xiNbI/d4xA9sLVvXkjh5B2tNx2ZT2/3rpmh8to=
github.com/mithrandie/ternary v1.1.1 h1:k/joD6UGVYxHixYmSR8EGgDFNONBMqyD373xT4QRdC4=
github.com/mithrandie/ternary v1.1.1/go.mod h1:0D9Ba3+09K2TdSZO7/bFCC0GjSXetCvYuYq0u8FY/1g=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/sys/atomicwriter v0.1.0 h1:kw5D/EqkBwsBFi0ss9v1VG3wIkVhzGvLklJ+w3A14Sw=
Expand Down
13 changes: 6 additions & 7 deletions internal/inspect/bloat/bloat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,10 @@ import (
var BloatQuery string

type Result struct {
Type string
Schemaname string
Object_name string
Bloat string
Waste string
Type string
Name string
Bloat string
Waste string
}

func Run(ctx context.Context, config pgconn.Config, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error {
Expand All @@ -41,9 +40,9 @@ func Run(ctx context.Context, config pgconn.Config, fsys afero.Fs, options ...fu
return err
}

table := "|Type|Schema name|Object name|Bloat|Waste\n|-|-|-|-|-|\n"
table := "|Type|Name|Bloat|Waste\n|-|-|-|-|\n"
for _, r := range result {
table += fmt.Sprintf("|`%s`|`%s`|`%s`|`%s`|`%s`|\n", r.Type, r.Schemaname, r.Object_name, r.Bloat, r.Waste)
table += fmt.Sprintf("|`%s`|`%s`|`%s`|`%s`|\n", r.Type, r.Name, r.Bloat, r.Waste)
}
return list.RenderTable(table)
}
Loading