Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanction check base #803

Open
wants to merge 58 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
4aa8f15
wip: sanction check configs.
Jan 15, 2025
c404824
Added initial migration and dbmodels for sanction checks.
Jan 16, 2025
e55c0da
wip: sanction check execution.
Jan 16, 2025
354b6db
Set up server-level configuration for OpenSanctions provider.
Jan 17, 2025
108baf0
Set up plumbing for org-level configuration.
Jan 17, 2025
0e72b13
Implementing dummy sanctions check modifiers.
Jan 17, 2025
8232761
Implement organization config columns for sanction checking
Jan 17, 2025
4c842a8
Start implementing scenario config for Sanction Check (outcome and sc…
Jan 17, 2025
fd707b6
Continue implementing API for scenario config around Sanction Check.
Jan 17, 2025
3175f0c
Sanction Check config: added validations, improved SQL query building…
Jan 19, 2025
a85f44e
Return basic sanction check result as part of the decision DTO.
Jan 20, 2025
b5e53e8
Clean up and refactor.
Jan 20, 2025
cf1d5b7
Bump migration version after merge.
Jan 20, 2025
5b760c0
Start writing results and matches to database.
Jan 20, 2025
6013b30
Added tentative API endpoint to retrieve a sanction check.
Jan 21, 2025
0a4c533
Remove manual unmarshalling of DB JSON.
Jan 22, 2025
34834be
Implement review match API.
Jan 22, 2025
dfd684b
Added missing fields and some optimizations.
Jan 22, 2025
1ad6f15
Added endpoint to post a sanction check comment and added comment_cou…
Jan 22, 2025
2dd0299
Added API endpoint to list comments on a match.
Jan 22, 2025
2342156
Added example simple usecase test, mocking enforcers and repositories.
Jan 22, 2025
7cc9999
Small refactor.
Jan 23, 2025
02079ff
We should only consider non-archived sanction check when listing.
Jan 23, 2025
47152a9
Updated route to be consistent.
Jan 23, 2025
a55faa4
Add freshness checker for the datasets.
Jan 23, 2025
b18270d
Fixed test.
Jan 23, 2025
57fb096
Continue outdated dataset check.
Jan 23, 2025
a90b76d
Store and process the trigger rule on the sanction check.
Jan 23, 2025
b90e87d
Add another example test.
Jan 24, 2025
07f2f6e
Fixed issues with data model.
Jan 24, 2025
e7a51a4
Refactored sanction check evaluation and HTTP client injection. Added…
Jan 26, 2025
e9592bf
Some refactoring.
Jan 27, 2025
479cbbf
Added refine endpoint (still with static query).
Jan 27, 2025
c9f54aa
Added "no_hit" option for sanction check status.
Jan 27, 2025
7a3383d
Refactor: getting ready to merge WIP.
Jan 28, 2025
4ac73f6
Added logs the right way, and refactored enforcer for cases.
Jan 29, 2025
89d35fb
Remove mentions of OpenSanctions from the configuration.
Jan 29, 2025
a454044
cascade changes on sanction check, matches when one match is reviewed
Pascal-Delange Jan 30, 2025
aca17dc
fix test
Pascal-Delange Jan 30, 2025
b562afa
create case event when a sanction check is finished reviewing
Pascal-Delange Jan 30, 2025
dd64534
fix migration
Pascal-Delange Jan 30, 2025
2a72914
implement repo method
Pascal-Delange Jan 30, 2025
03b3d7a
Added Basic and Bearer authentication when self-hosting Open Sanctions.
Jan 31, 2025
d907966
implement: cannot approve a decision if it has a sanction check attac…
Pascal-Delange Jan 30, 2025
d96c7de
refactor sanction check repo methods
Pascal-Delange Jan 31, 2025
2242652
refactor continued
Pascal-Delange Jan 31, 2025
38194b5
rewrite model a bit
Pascal-Delange Jan 31, 2025
bcbfabd
fix test
Pascal-Delange Jan 31, 2025
eaa2a47
fix permission implementation on sanction check list
Pascal-Delange Jan 31, 2025
57bc04f
Implement query AST, with a new function, StringConcat.
Jan 29, 2025
4bd9b79
Added a few unit tests for the evaluate sanction check function.
Jan 30, 2025
295ed51
Add on delete cascade between sanction check configs and scenario ite…
Jan 30, 2025
238c9fa
Mark the trigger rule of a sanction check as non-null, only its updat…
Jan 30, 2025
3f377de
Edit scenario validation to include sanction check config.
Jan 30, 2025
c93d909
StringConcat node: skip nil values.
Jan 30, 2025
4bc215f
StringConcat node: error out when all values are nil.
Jan 30, 2025
69c20db
Bump migration version.
Jan 31, 2025
ef5b69b
Fixed tests from feature branch changes.
Jan 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions api/handle_organization.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ func handlePatchOrganization(uc usecases.Usecases) func(c *gin.Context) {
organization, err := usecase.UpdateOrganization(ctx, models.UpdateOrganizationInput{
Id: organizationID,
DefaultScenarioTimezone: data.DefaultScenarioTimezone,
SanctionCheckConfig: models.OrganizationOpenSanctionsConfig{
MatchThreshold: data.SanctionsThreshold,
MatchLimit: data.SanctionsLimit,
},
})

if presentError(ctx, c, err) {
Expand Down
164 changes: 164 additions & 0 deletions api/handle_sanction_checks.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
package api

import (
"net/http"

"github.com/checkmarble/marble-backend/dto"
"github.com/checkmarble/marble-backend/models"
"github.com/checkmarble/marble-backend/pure_utils"
"github.com/checkmarble/marble-backend/usecases"
"github.com/checkmarble/marble-backend/utils"
"github.com/gin-gonic/gin"
)

func handleSanctionCheckDataset(uc usecases.Usecases) func(c *gin.Context) {
return func(c *gin.Context) {
ctx := c.Request.Context()
uc := usecasesWithCreds(ctx, uc).NewSanctionCheckUsecase()

dataset, err := uc.CheckDataset(ctx)

if presentError(ctx, c, err) {
return
}

c.JSON(http.StatusOK, dto.AdaptSanctionCheckDataset(dataset))
}
}

func handleListSanctionChecks(uc usecases.Usecases) func(c *gin.Context) {
return func(c *gin.Context) {
ctx := c.Request.Context()
decisionId := c.Query("decision_id")

if decisionId == "" {
c.Status(http.StatusBadRequest)
return
}

uc := usecasesWithCreds(ctx, uc).NewSanctionCheckUsecase()
sanctionChecks, err := uc.ListSanctionChecks(ctx, decisionId)

if presentError(ctx, c, err) {
return
}

c.JSON(http.StatusOK, pure_utils.Map(sanctionChecks, dto.AdaptSanctionCheckDto))
}
}

func handleUpdateSanctionCheckMatchStatus(uc usecases.Usecases) func(c *gin.Context) {
return func(c *gin.Context) {
ctx := c.Request.Context()
matchId := c.Param("id")

var payload dto.SanctionCheckMatchUpdateDto

if presentError(ctx, c, c.ShouldBindJSON(&payload)) {
return
}

creds, ok := utils.CredentialsFromCtx(ctx)

if !ok {
presentError(ctx, c, models.ErrUnknownUser)
return
}

update, err := dto.AdaptSanctionCheckMatchUpdateInputDto(matchId, creds.ActorIdentity.UserId, payload)

if presentError(ctx, c, err) {
return
}

uc := usecasesWithCreds(ctx, uc).NewSanctionCheckUsecase()

match, err := uc.UpdateMatchStatus(ctx, update)

if presentError(ctx, c, err) {
return
}

c.JSON(http.StatusOK, dto.AdaptSanctionCheckMatchDto(match))
}
}

func handleListSanctionCheckMatchComments(uc usecases.Usecases) func(c *gin.Context) {
return func(c *gin.Context) {
ctx := c.Request.Context()
matchId := c.Param("id")

uc := usecasesWithCreds(ctx, uc).NewSanctionCheckUsecase()

comments, err := uc.MatchListComments(ctx, matchId)

if presentError(ctx, c, err) {
return
}

c.JSON(http.StatusOK, pure_utils.Map(comments, dto.AdaptSanctionCheckMatchCommentDto))
}
}

func handleCreateSanctionCheckMatchComment(uc usecases.Usecases) func(c *gin.Context) {
return func(c *gin.Context) {
ctx := c.Request.Context()
matchId := c.Param("id")

var payload dto.SanctionCheckMatchCommentDto

if presentError(ctx, c, c.ShouldBindJSON(&payload)) {
return
}

creds, ok := utils.CredentialsFromCtx(ctx)

if !ok {
presentError(ctx, c, models.ErrUnknownUser)
return
}

uc := usecasesWithCreds(ctx, uc).NewSanctionCheckUsecase()
comment, err := dto.AdaptSanctionCheckMatchCommentInputDto(matchId, creds.ActorIdentity.UserId, payload)

if presentError(ctx, c, err) {
return
}

comment, err = uc.MatchAddComment(ctx, matchId, comment)

if presentError(ctx, c, err) {
return
}

c.JSON(http.StatusCreated, dto.AdaptSanctionCheckMatchCommentDto(comment))
}
}

func handleRefineSanctionCheck(uc usecases.Usecases) func(c *gin.Context) {
return func(c *gin.Context) {
ctx := c.Request.Context()

var payload dto.SanctionCheckRefineDto

if presentError(ctx, c, c.ShouldBindJSON(&payload)) {
return
}

creds, ok := utils.CredentialsFromCtx(ctx)

if !ok {
presentError(ctx, c, models.ErrUnknownUser)
return
}

uc := usecasesWithCreds(ctx, uc).NewSanctionCheckUsecase()
sanctionCheck, err := uc.Refine(ctx, dto.AdaptSanctionCheckRefineDto(payload), creds.ActorIdentity.UserId)

if presentError(ctx, c, err) {
return
}

c.JSON(http.StatusOK, dto.AdaptSanctionCheckDto(sanctionCheck))
}
}
9 changes: 9 additions & 0 deletions api/routes.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,15 @@ func addRoutes(r *gin.Engine, conf Configuration, uc usecases.Usecases, auth Aut
router.PATCH("/scenario-iteration-rules/:rule_id", tom, handleUpdateRule(uc))
router.DELETE("/scenario-iteration-rules/:rule_id", tom, handleDeleteRule(uc))

router.GET("/sanction-checks/dataset", tom, handleSanctionCheckDataset(uc))
router.GET("/sanction-checks", tom, handleListSanctionChecks(uc))
router.POST("/sanction-checks/refine", tom, handleRefineSanctionCheck(uc))
router.PATCH("/sanction-checks/matches/:id", tom, handleUpdateSanctionCheckMatchStatus(uc))
router.POST("/sanction-checks/matches/:id/comments", tom,
handleCreateSanctionCheckMatchComment(uc))
router.GET("/sanction-checks/matches/:id/comments", tom,
handleListSanctionCheckMatchComments(uc))

router.GET("/scenario-publications", tom, handleListScenarioPublications(uc))
router.POST("/scenario-publications", tom, handleCreateScenarioPublication(uc))
router.GET("/scenario-publications/preparation", tom,
Expand Down
8 changes: 8 additions & 0 deletions cmd/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,13 @@ func RunServer() error {
RateLimit: utils.GetEnv("CONVOY_RATE_LIMIT", 50),
}

openSanctionsConfig := infra.InitializeOpenSanctions(
http.DefaultClient,
utils.GetEnv("OPENSANCTIONS_API_HOST", ""),
utils.GetEnv("OPENSANCTIONS_AUTH_METHOD", ""),
utils.GetEnv("OPENSANCTIONS_API_KEY", ""),
)

seedOrgConfig := models.SeedOrgConfiguration{
CreateGlobalAdminEmail: utils.GetEnv("CREATE_GLOBAL_ADMIN_EMAIL", ""),
CreateOrgName: utils.GetEnv("CREATE_ORG_NAME", ""),
Expand Down Expand Up @@ -133,6 +140,7 @@ func RunServer() error {
infra.InitializeConvoyRessources(convoyConfiguration),
convoyConfiguration.RateLimit,
),
repositories.WithOpenSanctions(openSanctionsConfig),
repositories.WithClientDbConfig(clientDbConfig),
repositories.WithTracerProvider(telemetryRessources.TracerProvider),
)
Expand Down
15 changes: 14 additions & 1 deletion dto/decision_dto.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ type DecisionRule struct {
RuleEvaluation *ast.NodeEvaluationDto `json:"rule_evaluation,omitempty"`
}

type DecisionSanctionCheck struct {
Partial bool `json:"partial"`
Count int `json:"count"`
}

type ErrorDto struct {
Code int `json:"code"`
Message string `json:"message"`
Expand Down Expand Up @@ -128,7 +133,8 @@ type Decision struct {

type DecisionWithRules struct {
Decision
Rules []DecisionRule `json:"rules"`
Rules []DecisionRule `json:"rules"`
SanctionCheck *DecisionSanctionCheck `json:"sanction_check,omitempty"`
}

func NewDecisionDto(decision models.Decision, marbleAppHost string) Decision {
Expand Down Expand Up @@ -190,6 +196,13 @@ func NewDecisionWithRuleDto(decision models.DecisionWithRuleExecutions, marbleAp
decisionDto.Rules[i] = NewDecisionRuleDto(ruleExecution, withRuleExecution)
}

if decision.SanctionCheckExecution != nil {
decisionDto.SanctionCheck = &DecisionSanctionCheck{
Partial: decision.SanctionCheckExecution.Partial,
Count: decision.SanctionCheckExecution.Count,
}
}

return decisionDto
}

Expand Down
6 changes: 6 additions & 0 deletions dto/organization_dto.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,17 @@ type APIOrganization struct {
Id string `json:"id"`
Name string `json:"name"`
DefaultScenarioTimezone *string `json:"default_scenario_timezone"`
SanctionsThreshold *int `json:"sanctions_threshold"`
SanctionsLimit *int `json:"sanctions_limit"`
}

func AdaptOrganizationDto(org models.Organization) APIOrganization {
return APIOrganization{
Id: org.Id,
Name: org.Name,
DefaultScenarioTimezone: org.DefaultScenarioTimezone,
SanctionsThreshold: org.OpenSanctionsConfig.MatchThreshold,
SanctionsLimit: org.OpenSanctionsConfig.MatchLimit,
}
}

Expand All @@ -23,4 +27,6 @@ type CreateOrganizationBodyDto struct {

type UpdateOrganizationBodyDto struct {
DefaultScenarioTimezone *string `json:"default_scenario_timezone,omitempty"`
SanctionsThreshold *int `json:"sanctions_threshold,omitempty"`
SanctionsLimit *int `json:"sanctions_limit,omitempty"`
}
31 changes: 31 additions & 0 deletions dto/sanction_check_dataset_dto.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package dto

import (
"time"

"github.com/checkmarble/marble-backend/models"
)

type OpenSanctionsDataset struct {
Upstream OpenSanctionsUpstreamDataset `json:"upstream"`
Version string `json:"version"`
UpToDate bool `json:"up_to_date"`
}

type OpenSanctionsUpstreamDataset struct {
Version string `json:"version"`
Name string `json:"name"`
LastExport time.Time `json:"last_export"`
}

func AdaptSanctionCheckDataset(model models.OpenSanctionsDataset) OpenSanctionsDataset {
return OpenSanctionsDataset{
Upstream: OpenSanctionsUpstreamDataset{
Version: model.Upstream.Version,
Name: model.Upstream.Name,
LastExport: model.Upstream.LastExport,
},
Version: model.Version,
UpToDate: model.UpToDate,
}
}
Loading