feat: ALS-based i2i similarity experiment#325
Open
em3s wants to merge 2 commits into
Open
Conversation
dosubot
Bot
added
size:L
6 tasks
Dogfoods the Step DSL (#313) on a real recommendation workload: V2 edge JSON dumps treated as implicit feedback, ALS-fit, then top-K cosine similarity over the item factors written as Parquet. - AlsTransform: wraps spark.ml.recommendation.ALS as a Transform. Default hyperparams target implicit feedback (implicitPrefs=true, coldStartStrategy="drop"); seed pinned for reproducibility. - TopKSimilarityTransform: cosine similarity via self-cross-join + Window row_number(). Output schema (item_id long, similar_item_id long, score double, rank int). Returns 0.0 for zero-vector pairs instead of NaN. - Adds Spark MLlib dependency (compileOnly + testImplementation). - pipeline/conf/als-i2i-experiment.yaml: production-shape workflow YAML driving FileSource → SqlTransform prep → AlsTransform → TopK → FileSink. - AlsI2iWorkflowTest: end-to-end on synthetic two-cluster edges, verifying ALS recovers the structure and TopK returns the correct sibling for each item. Depends on feat/step-dsl-313 (Step DSL itself). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Step DSL spec was refactored to Source/Flow/Merge/Split/Sink (commit e147235 on feat/step-dsl-313). This commit reclassifies the experiment's built-ins onto the new traits: - AlsTransform → AlsFlow (Flow, 1→1; emits item factors only) - TopKSimilarityTransform → TopKSimilarityFlow (Flow, 1→1) - YAML and end-to-end test references updated (SqlTransform → SqlMerge, AlsTransform → AlsFlow, TopKSimilarityTransform → TopKSimilarityFlow) CacheTransform multi-input require-check is dropped since Flow's signature guarantees a single input by construction. AlsFlow doc notes that dual-factor (item + user) belongs to a future AlsFactorSplit (1→2 Split) once Split built-ins land. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
em3s
force-pushed
the
feat/als-i2i-experiment
branch
from
b6784e2 to
0befd43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ALS i2i similarity experiment dogfooding the Step DSL on V2 edge JSON. First real workload that drove the Source/Flow/Merge/Split/Sink spec revision in #313.
Closes none — sits on top of #314 (Step DSL).
Changes
AlsFlow(Flow 1→1): wrapsspark.ml.recommendation.ALS; emits item factors. Implicit-feedback defaults.TopKSimilarityFlow(Flow 1→1): cosine similarity self-cross-join + Window top-K. Output schema(item_id long, similar_item_id long, score double, rank int).pipeline/conf/als-i2i-experiment.yaml: production-shape workflow —FileSource(json) → SqlMerge(prep) → AlsFlow → TopKSimilarityFlow → FileSink(parquet).compileOnly+testImplementation).How to Test
./gradlew :pipeline:test—AlsFlowTest(2),TopKSimilarityFlowTest(5),AlsI2iWorkflowTest(end-to-end on synthetic two-cluster edges; verifies each item's top-1 neighbor is its cluster sibling).path:in the YAML and run viaspark-submit --class com.kakao.actionbase.pipeline.jobs.StepsRunnerJob.AI Assistance