Skip to content

Commit 14b85b2

Browse files
miamsclaude
miams
and
claude
committed
feat: fix 1850 census loading and add note citation harvesting project
Census batch processing: - Fix slave schedule loading for 1850 census (use EventType 18) - Add mortality schedule support (EventType 2) - Change GROUP BY from EventID to CitationID to handle multiple sources per event Validators: - Accept plural page patterns ("pages X-Y", "pp. X-Y") for pre-1880 censuses - Add check for missing period after FamilySearch in bibliography Documentation: - Add NOTE_CITATION_HARVESTING.md project documentation - Initial discovery findings: 7,066 notes with URLs, 3,701 events without citations - Document 5 common note patterns and proposed architecture 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent ddb171d commit 14b85b2

4 files changed

Lines changed: 278 additions & 9 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,7 @@ browser = await playwright.chromium.connect_over_cdp("http://localhost:9222")
290290
| [docs/reference/CENSUS_FORM_RENDERING.md](docs/reference/CENSUS_FORM_RENDERING.md) | Census form rendering architecture |
291291
| [docs/architecture/CENSUS_BATCH_PROCESSING_ARCHITECTURE.md](docs/architecture/CENSUS_BATCH_PROCESSING_ARCHITECTURE.md) | Census processing workflow |
292292
| [docs/architecture/BATCH_PROCESSING_ARCHITECTURE.md](docs/architecture/BATCH_PROCESSING_ARCHITECTURE.md) | Find a Grave processing |
293+
| [docs/projects/NOTE_CITATION_HARVESTING.md](docs/projects/NOTE_CITATION_HARVESTING.md) | Note citation harvesting project |
293294

294295
## Common Tasks
295296

‎docs/projects/NOTE_CITATION_HARVESTING.md‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
# Note Citation Harvesting Project
2+
3+
## Objective
4+
Harvest citation information from notes fields in the RootsMagic database and generate proper Evidence Explained citations with free-form style.
5+
6+
## Status
7+
**Phase**: Discovery / Requirements Gathering
8+
9+
## Overview
10+
Many genealogical sources are currently stored as informal notes rather than proper citations. This project aims to:
11+
1. Identify notes containing source information (URLs, source references)
12+
2. Categorize by source type (FamilySearch, Ancestry, state vital records, etc.)
13+
3. Create citation templates for common source types
14+
4. Generate proper Evidence Explained citations
15+
5. Migrate data from notes to proper citation structure
16+
17+
## Target Note Locations
18+
- Person notes (PersonTable.Note)
19+
- Birth event notes
20+
- Death event notes
21+
- Other event notes with URLs
22+
23+
## Source Types to Support
24+
- FamilySearch collections
25+
- Ancestry.com collections
26+
- Newspapers.com
27+
- State vital records (e.g., "California Births")
28+
- Other online sources
29+
30+
---
31+
32+
## Discovery Findings
33+
34+
### Database Analysis (2026-01-01)
35+
36+
#### Volume Summary
37+
| Location | Count |
38+
|----------|-------|
39+
| Person notes with URLs | 888 |
40+
| Event notes with URLs | 6,178 |
41+
| **Total notes to process** | **7,066** |
42+
43+
#### URL Domain Distribution
44+
| Domain | Count |
45+
|--------|-------|
46+
| FamilySearch | 3,367 |
47+
| Newspapers.com | 2,220 |
48+
| Ancestry.com | 677 |
49+
| Other | ~800 |
50+
51+
#### Event Types Without Citations (Highest Priority)
52+
Events that have URL notes but NO linked citations - prime candidates for harvesting:
53+
54+
| Event Type | Count |
55+
|------------|-------|
56+
| Marriage | 1,374 |
57+
| Death | 887 |
58+
| Birth | 689 |
59+
| Obituary | 268 |
60+
| Divorce | 96 |
61+
| War Veteran | 59 |
62+
| News | 54 |
63+
| WWII Draft | 51 |
64+
| WWI Draft | 20 |
65+
| **Total** | **3,701** |
66+
67+
#### Citation Status
68+
- Events with URL notes that HAVE citations: 2,477
69+
- Events with URL notes WITHOUT citations: 3,701 (target)
70+
71+
### FamilySearch Collection Analysis
72+
73+
Most frequently referenced FamilySearch collections in notes:
74+
75+
| Collection | Count |
76+
|------------|-------|
77+
| United States, Social Security Numerical Identification Files (NUMIDENT) | 166 |
78+
| United States Social Security Death Index | 108 |
79+
| United States Public Records, 1970-2009 | 101 |
80+
| Ohio, County Marriages, 1789-1994 | 98 |
81+
| Pennsylvania, County Marriages, 1885-1950 | 87 |
82+
| United States World War II Draft Registration Cards, 1942 | 55 |
83+
| Ohio, Deaths, 1908-1953 | 49 |
84+
| North Carolina, County Marriages, 1762-1979 | 49 |
85+
| Iowa, County Marriages, 1838-1934 | 46 |
86+
| Texas Birth Index, 1903-1997 | 42 |
87+
| Ohio, Marriages, 1800-1958 | 40 |
88+
| Ohio, County Births, 1841-2003 | 39 |
89+
| California Death Index, 1940-1997 | 38 |
90+
| California Birth Index, 1905-1995 | 38 |
91+
92+
### Common Note Patterns
93+
94+
#### Pattern 1: Well-Formatted FamilySearch Citation
95+
```
96+
"Pennsylvania, County Marriages, 1885-1950," index and images, <i>FamilySearch</i>
97+
(https://familysearch.org/pal:/MM9.1.1/VFQ9-J71 : accessed 06 Oct 2013),
98+
James B Iams and Ruth Jones, 1941.
99+
```
100+
- Already in Evidence Explained format
101+
- Contains: collection name, record type, publisher, URL, access date, subject details
102+
103+
#### Pattern 2: FamilySearch Citation with Citing Info
104+
```
105+
"California Birth Index, 1905-1995," database, <i>FamilySearch</i>
106+
(https://familysearch.org/ark:/61903/1:1:VLDB-6KS : 27 November 2014),
107+
Robert Ijams, 07 Mar 1920; citing Los Angeles, California, United States,
108+
Department of Health Services, Vital Statistics Department, Sacramento.
109+
```
110+
- Includes original source citation
111+
112+
#### Pattern 3: Ancestry Source Information
113+
```
114+
Source Information
115+
Ancestry.com. <i>Pennsylvania, U.S., Marriages, 1852-1968</i> [database on-line].
116+
Lehi, UT, USA: Ancestry.com Operations, Inc., 2016.
117+
118+
Original data: <i>Marriage Records</i>. <i>Pennsylvania Marriages.</i>
119+
Various County Register of Wills Offices, Pennsylvania.
120+
121+
https://www.ancestry.com/discoveryui-content/view/902116152%3A61381
122+
```
123+
- Structured with source info, original data, and URL
124+
125+
#### Pattern 4: Newspapers.com Citation
126+
```
127+
Source Citation
128+
The Bristol Daily Courier; Publication Date: 15/ Apr/ 1964;
129+
Publication Place: Bristol, Pennsylvania, USA;
130+
URL: https://www.newspapers.com/image/53107874/...
131+
```
132+
- Contains publication name, date, place, and URL
133+
134+
#### Pattern 5: Raw Note with URL
135+
```
136+
http://wc.rootsweb.ancestry.com/cgi-bin/igm.cgi?op=GET&db=raykohler&id=I29652
137+
DEATH NOTICE:
138+
HOWARD IMES
139+
[transcription of death notice]
140+
(Juniata Sentinel & Republican - August 22, 1917)
141+
```
142+
- URL plus transcribed content with source reference
143+
144+
---
145+
146+
## Technical Approach
147+
148+
### Proposed Architecture
149+
150+
#### Phase 1: Parser Development
151+
Create parsers for each source pattern:
152+
1. **FamilySearchParser** - Parse FamilySearch-formatted citations
153+
2. **AncestryParser** - Parse Ancestry source information blocks
154+
3. **NewspapersComParser** - Parse Newspapers.com citations
155+
4. **GenericURLParser** - Extract URLs and surrounding context
156+
157+
#### Phase 2: Source Matching/Creation
158+
1. Check if source already exists (by collection name + year)
159+
2. Create new free-form source if needed
160+
3. Use TemplateID=0 (free-form) for all generated sources
161+
162+
#### Phase 3: Citation Generation
163+
1. Generate Evidence Explained format citations
164+
2. Create Footnote, Short Footnote, Bibliography forms
165+
3. Link citations to events (CitationLinkTable)
166+
167+
#### Phase 4: Note Cleanup
168+
1. Optionally preserve original note as research note
169+
2. Remove citation text from event notes after migration
170+
3. Keep person notes as-is (reference only)
171+
172+
### Database Operations
173+
174+
#### Source Creation
175+
```sql
176+
-- Free-form source (TemplateID=0)
177+
INSERT INTO SourceTable (Name, TemplateID, Fields)
178+
VALUES ('Birth Records: California Birth Index, 1905-1995 - FamilySearch', 0, [BLOB])
179+
```
180+
181+
#### Citation Linking
182+
```sql
183+
-- Link citation to event
184+
INSERT INTO CitationLinkTable (OwnerType, OwnerID, CitationID, Quality, ...)
185+
VALUES (2, [EventID], [CitationID], 'PDO', ...)
186+
```
187+
188+
---
189+
190+
## Questions to Resolve
191+
192+
### Already Answered
193+
1. **What URL patterns exist in notes?**
194+
- FamilySearch ark URLs (2,388), Newspapers.com URLs (2,220), Ancestry URLs (677)
195+
196+
2. **What are the most common source types?**
197+
- FamilySearch vital records (births, deaths, marriages)
198+
- Social Security files (SSDI, NUMIDENT)
199+
- Newspapers.com clippings
200+
- Ancestry record collections
201+
202+
### Still Need to Decide
203+
1. **Should we create one source per collection, or per record?**
204+
- Recommendation: One source per collection, multiple citations
205+
206+
2. **How to handle notes that already have citations?**
207+
- Skip? Verify? Flag for review?
208+
209+
3. **Should the original note text be preserved?**
210+
- Options: Keep as research note, append to citation, discard
211+
212+
4. **Priority order for processing?**
213+
- Suggestion: Start with events WITHOUT citations (3,701 records)
214+
215+
5. **Should we auto-process or require user review per batch?**
216+
- Consider: Review first 10 of each type, then batch process
217+
218+
---
219+
220+
## Progress Log
221+
- **2026-01-01**: Project initiated, beginning discovery phase
222+
- **2026-01-01**: Completed initial database analysis:
223+
- Identified 7,066 total notes with URLs
224+
- Found 3,701 events without citations (priority targets)
225+
- Analyzed top FamilySearch collections (50+ unique collections)
226+
- Documented 5 common note patterns
227+
- Proposed technical architecture
228+
229+
---
230+
231+
## Next Steps
232+
1. Build prototype parser for FamilySearch formatted citations (most common, well-structured)
233+
2. Test extraction on sample of 10-20 notes
234+
3. Define Evidence Explained citation templates for each source type
235+
4. Decide on user review workflow

‎scripts/census_quality/validators.py‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -231,9 +231,10 @@ def check_footnote(
231231
# Match sheet with optional hyphen: "sheet 6B" or "sheet 6-B"
232232
has_sheet = bool(re.search(r"sheet\s+\d+-?[AB]?", footnote, re.IGNORECASE))
233233
has_stamp = bool(re.search(r"stamp\s+\d+", footnote, re.IGNORECASE))
234-
has_page = bool(re.search(r"page\s+\d+", footnote, re.IGNORECASE))
234+
# Match "page X" or "pages X-Y" (for records spanning multiple pages)
235+
has_page = bool(re.search(r"pages?\s+\d+", footnote, re.IGNORECASE))
235236
has_page_stamped = bool(
236-
re.search(r"page\s+\d+\s*\(stamped\)", footnote, re.IGNORECASE)
237+
re.search(r"pages?\s+\d+(?:-\d+)?\s*\(stamped\)", footnote, re.IGNORECASE)
237238
)
238239

239240
# 1850-1870 use "page X", 1880 uses "page X (stamped)"
@@ -447,10 +448,10 @@ def check_short_footnote(
447448
# Match sheet with optional hyphen: "sheet 6B" or "sheet 6-B"
448449
has_sheet = bool(re.search(r"sheet\s+\d+-?[AB]?", short_footnote, re.IGNORECASE))
449450
has_stamp = bool(re.search(r"stamp\s+\d+", short_footnote, re.IGNORECASE))
450-
# Match "page X" or "p. X" for pre-1880 censuses
451-
has_page = bool(re.search(r"(?:page|p\.)\s+\d+", short_footnote, re.IGNORECASE))
451+
# Match "page X", "pages X-Y", "p. X", or "pp. X-Y" for pre-1880 censuses
452+
has_page = bool(re.search(r"(?:pages?|pp?\.)\s+\d+", short_footnote, re.IGNORECASE))
452453
has_page_stamped = bool(
453-
re.search(r"(?:page|p\.)\s+\d+\s*\(stamped\)", short_footnote, re.IGNORECASE)
454+
re.search(r"(?:pages?|pp?\.)\s+\d+(?:-\d+)?\s*\(stamped\)", short_footnote, re.IGNORECASE)
454455
)
455456

456457
# 1850-1870 use "page X" or "p. X", 1880 uses "p. X (stamped)"
@@ -670,6 +671,30 @@ def check_bibliography(
670671
)
671672
)
672673

674+
# Check for period after FamilySearch before URL
675+
# Correct: "</i>. https://" or "&lt;/i&gt;. https://"
676+
# Wrong: "</i> https://" or "&lt;/i&gt; https://"
677+
if has_italics:
678+
missing_period = bool(
679+
re.search(r"</i>\s+https://|&lt;/i&gt;\s+https://", bibliography)
680+
)
681+
has_period = bool(
682+
re.search(r"</i>\.\s+https://|&lt;/i&gt;\.\s+https://", bibliography)
683+
)
684+
if missing_period and not has_period:
685+
issues.append(
686+
Issue(
687+
source_id=source_id,
688+
issue_type="bibliography_missing_period_after_publisher",
689+
severity="error",
690+
message="Missing period after <i>FamilySearch</i> before URL",
691+
field="bibliography",
692+
current_value="</i> https://",
693+
expected_value="</i>. https://",
694+
category="format",
695+
)
696+
)
697+
673698
return issues
674699

675700

‎scripts/process_census_batch.py‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -294,10 +294,17 @@ def find_census_citations(
294294
JOIN CitationLinkTable cl ON e.EventID = cl.OwnerID AND cl.OwnerType = 2
295295
JOIN CitationTable c ON cl.CitationID = c.CitationID
296296
JOIN SourceTable s ON c.SourceID = s.SourceID
297-
WHERE e.EventType = 18
298-
AND e.Date LIKE ?
297+
WHERE (
298+
-- Census events (EventType 18) for population and slave schedules
299+
(e.EventType = 18 AND (
300+
s.Name LIKE 'Fed Census: ' || ? || ',%'
301+
OR s.Name LIKE 'Fed Census Slave Schedule: ' || ? || ',%'
302+
))
303+
-- Mortality schedule events (EventType 2 = Death)
304+
OR (e.EventType = 2 AND s.Name LIKE 'Fed Census Mortality Schedule: ' || ? || ',%')
305+
)
299306
AND s.TemplateID = 0
300-
GROUP BY e.EventID
307+
GROUP BY c.CitationID
301308
ORDER BY n.Surname COLLATE RMNOCASE, n.Given COLLATE RMNOCASE
302309
LIMIT ? OFFSET ?
303310
"""
@@ -323,7 +330,8 @@ def find_census_citations(
323330
max_iterations = 20 # Safety limit to prevent infinite loops
324331

325332
for iteration in range(max_iterations):
326-
cursor.execute(query, (f'%{census_year}%', batch_size, current_offset))
333+
# Pass census_year 3 times (for population, slave, mortality patterns) + limit + offset
334+
cursor.execute(query, (str(census_year), str(census_year), str(census_year), batch_size, current_offset))
327335
rows = cursor.fetchall()
328336

329337
if not rows:

0 commit comments

Comments
 (0)