-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathpfm-deep.log
More file actions
110 lines (104 loc) · 8.17 KB
/
Copy pathpfm-deep.log
File metadata and controls
110 lines (104 loc) · 8.17 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
──────────────────────────────────────────────────────────────────────────────
PrivateFoundationModels deep verification
──────────────────────────────────────────────────────────────────────────────
model: lfm2.5-350m
date: 2026-05-13 03:24:46 +0000
• Fetching mlboydaisuke/lfm2.5-350m-coreml → /Users/majimadaisuke/Library/Application Support/PrivateFoundationModels/lfm2.5-350m-coreml
• [1/12] .gitattributes (1.5 KB) — cached
• [2/12] README.md (6.6 KB) — cached
• [3/12] hf_model/config.json (1.3 KB) — cached
• [4/12] hf_model/generation_config.json (134 B) — cached
• [5/12] hf_model/tokenizer.json (4.5 MB) — cached
• [6/12] hf_model/tokenizer_config.json (573 B) — cached
• [7/12] model.mlmodelc/analytics/coremldata.bin (243 B) — cached
• [8/12] model.mlmodelc/coremldata.bin (581 B) — cached
• [9/12] model.mlmodelc/metadata.json (4.3 KB) — cached
• [10/12] model.mlmodelc/model.mil (387.3 KB) — cached
• [11/12] model.mlmodelc/weights/weight.bin (805.1 MB) — cached
• [12/12] model_config.json (454 B) — cached
• Loading model…
• Reading config...
• Loading tokenizer...
• Loading model...
[Load] LFM2 conv_state_in shape: [10, 1024, 3]
• Ready
──────────────────────────────────────────────────────────────────────────────
Generable scenarios (structured output)
──────────────────────────────────────────────────────────────────────────────
• G1. simple-object (3 strings) — prompt: Pick one famous landmark and return its city and country.
✓ G1. simple-object (3 strings) → city=Eiffel Tower country=France (1966 ms)
• G2. mixed-primitives (string + int? no — number + bool) — prompt: Invent a sensor reading. Pick a short name, a numeric value, and an active boolean.
✓ G2. mixed-primitives (string + int? no — number + bool) → name=temperature value=25.0 active=true (2618 ms)
• G3. array-of-strings (3-5 items) — prompt: Make a shopping list with a name and 3 items.
✓ G3. array-of-strings (3-5 items) → name=Organic Fresh Produce items=["organic apples", "fresh berries", "organic spinach"] (2508 ms)
• G4. nested-object (2 levels) — prompt: Invent a person profile with a name, age, and address (city, country).
✓ G4. nested-object (2 levels) → Liam Carter, age 30, New York, USA (3320 ms)
• G5. optional-fields (absent OK) — prompt: Make up a one-line article. Title only, no summary.
⚠ G5. optional-fields (absent OK) — model emitted unparseable output (model-quality, not framework). raw="Renewable energy sources are becoming increasingly cost-effective."
• G6. streaming-generable (Profile) — prompt: Invent a person profile. Name, age, city, country.
⚠ G6. streaming-generable (Profile) — model emitted no parseable snapshot (model-quality)
──────────────────────────────────────────────────────────────────────────────
Tool calling scenarios
──────────────────────────────────────────────────────────────────────────────
• T1. single-tool (add) — prompt: What is 17 plus 25? You MUST use the add tool.
✓ T1. single-tool (add) → tool used (add) (6417 ms)
• last tool output: 42
• final assistant: 17 plus 25 equals 42.
• T2. multi-tool, picks add — prompt: Use a tool to compute 7 + 3.
✓ T2. multi-tool, picks add → tool used (add) (8010 ms)
• last tool output: 10
• final assistant: 7 + 3 = 10
• T3. multi-tool, picks multiply — prompt: Use a tool to compute 6 times 7.
⚠ T3. multi-tool, picks multiply — model answered without invoking the expected tool (model-quality)
• T4. complex-arguments (lookup topic+limit) — prompt: Use the lookup tool to get 3 facts about Swift concurrency.
⚠ T4. complex-arguments (lookup topic+limit) — model answered without invoking the expected tool (model-quality)
• T5. throwing-tool surfaces error — prompt: Use the boom tool with key=foo.
✓ T5. throwing-tool surfaces error → caught Boom from tool via GenerationError.backend
──────────────────────────────────────────────────────────────────────────────
Multimodal + builder scenarios
──────────────────────────────────────────────────────────────────────────────
• M1. respond(to:image:) — prompt: Describe what you see.
✓ M1. respond(to:image:) → "I see a quiet forest at dawn, with trees swaying gently in the soft morning light." (816 ms)
• M2. streamResponse(to:image:)
✓ M2. streamResponse(to:image:) → 36 snapshots, final "The image depicts a serene landscape with a few trees in the foreground, surrounded by a soft, golden light. The sky is clear and calm, with…" (1004 ms)
• M3. respond { PromptBuilder }
✓ M3. PromptBuilder → "Bonne matinée." (644 ms)
──────────────────────────────────────────────────────────────────────────────
Summary
──────────────────────────────────────────────────────────────────────────────
PASS (API works + content correct): 10
MODEL (API works, content model-limited): 4
FAIL (framework / backend regression): 0
PASS G1. simple-object (3 strings)
city=Eiffel Tower country=France
PASS G2. mixed-primitives (string + int? no — number + bool)
name=temperature value=25.0 active=true
PASS G3. array-of-strings (3-5 items)
name=Organic Fresh Produce items=["organic apples", "fresh berries", "organic spinach"]
PASS G4. nested-object (2 levels)
Liam Carter, age 30, New York, USA
MODEL G5. optional-fields (absent OK)
decodingFailure on "Renewable energy sources are becoming increasingly cost-effective."
MODEL G6. streaming-generable (Profile)
no parseable snapshot
PASS T1. single-tool (add)
calls=add final=17 plus 25 equals 42.
PASS T2. multi-tool, picks add
calls=add final=7 + 3 = 10
MODEL T3. multi-tool, picks multiply
kinds=[PrivateFoundationModels.Transcript.Entry.Kind.instructions, PrivateFoundationModels.Transcript.Entry.Kind.prompt, PrivateFoundationModels.Transcript.Entry.Kind.response] final=TOOL_CALL: multiply
SINGLE-LINE-JSON-ARG: "6*7"
MODEL T4. complex-arguments (lookup topic+limit)
kinds=[PrivateFoundationModels.Transcript.Entry.Kind.instructions, PrivateFoundationModels.Transcript.Entry.Kind.prompt, PrivateFoundationModels.Transcript.Entry.Kind.response] final=```json
{
"TOOL_CALL": "lookup",
"single-line_json_arguments": {
"topic"
PASS T5. throwing-tool surfaces error
Boom surfaced
PASS M1. respond(to:image:)
"I see a quiet forest at dawn, with trees swaying gently in the soft morning light."
PASS M2. streamResponse(to:image:)
36 snapshots
PASS M3. respond { PromptBuilder }
"Bonne matinée."