Skip to content

Add offline map tile prefetch tooling#18

Open
brothercorvo wants to merge 7 commits into
torlando-tech:mainfrom
brothercorvo:codex/map-prefetch-offline-20260316
Open

Add offline map tile prefetch tooling#18
brothercorvo wants to merge 7 commits into
torlando-tech:mainfrom
brothercorvo:codex/map-prefetch-offline-20260316

Conversation

@brothercorvo

Copy link
Copy Markdown

Summary

  • add ools/prefetch_map_tiles.py to download Pyxis-compatible XYZ PNG tiles into iles/{z}/{x}/{y}.png
  • add ools/README.md describing the defaults, radius-based area selection, and SD card layout
  • include the additional staged repo updates requested during branch preparation

Notes

  • the script defaults to the coordinates shown in the provided screenshot and a 200 km radius
  • output matches the firmware map loader format directly

Testing

  • not run

@greptile-apps

greptile-apps Bot commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

  • Adds tools/prefetch_map_tiles.py, a developer utility that downloads XYZ raster tiles into the tiles/{z}/{x}/{y}.png layout expected by the Pyxis firmware SD card loader, along with a tools/README.md documenting usage and tile-provider requirements.
  • All issues flagged in the previous review round have been addressed: the OSM tile-policy warning is now prominent in both the README and --help, --url-template is required for non-dry-run invocations and validated for {z}/{x}/{y} placeholders, write_bytes is inside the try block with OSError caught, and antimeridian bounding boxes are split correctly via tile_x_ranges + the shifted-longitude loop in tile_intersects_radius.
  • One remaining P2: the zoom-level validation has no upper bound, so a user passing --max-zoom 20+ will trigger an in-memory tile enumeration that can exhaust RAM or run indefinitely before a single download begins.

Confidence Score: 5/5

Safe to merge; all previously flagged P1s are resolved and only a minor zoom-cap guard remains.

Every P1/P0 issue from the prior review round (OSM policy, antimeridian distance, write-error cleanup, URL template validation) is addressed in the current HEAD. The single new finding is a P2 missing upper-bound guard on --max-zoom, which does not cause wrong data or incorrect downloads — only a potential performance/resource problem for unusually high user-supplied values. P2-only → 5/5 per severity anchoring.

tools/prefetch_map_tiles.py — zoom-level upper-bound guard at line 278.

Important Files Changed

Filename Overview
tools/prefetch_map_tiles.py New tile prefetch script. Antimeridian handling, write-error cleanup, and URL-template validation are all addressed. One P2: no upper bound enforced on --max-zoom, which can cause OOM or indefinite runtime for high zoom values.
tools/README.md New documentation for the prefetch script. Now correctly warns against OSM's public tile service, documents required --url-template, and notes antimeridian splitting support.
.gitignore Adds /tiles/ output directory and both log file names to .gitignore. Clean, addresses the previously flagged committed log files.
deps/microReticulum Submodule bumped from ca355e5 to efdc7a1; routine dependency update, no logic changes in this repo.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([main]) --> B{validate args}
    B -- invalid --> Z1([exit 2])
    B -- valid --> C[enumerate tiles per zoom level\nbounding_box → tile_x_ranges\n+ tile_intersects_radius filter]
    C --> D{dry-run?}
    D -- yes --> Z2([print counts, exit 0])
    D -- no --> E[for each tile]
    E --> F{dest exists\n& no overwrite?}
    F -- skip --> G[skipped++]
    F -- download --> H[build_url\ndownload_tile]
    H --> I{HTTP success?}
    I -- yes --> J[write_bytes to\ntiles/z/x/y.png]
    J --> K[downloaded++]
    I -- fail/retry --> L{attempts\nexhausted?}
    L -- no --> H
    L -- yes --> M[unlink partial\nFAIL log]
    M --> N[failed++]
    G & K & N --> O{more tiles?}
    O -- yes --> E
    O -- no --> P([print summary\nexit 0 or 1])
Loading
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
tools/prefetch_map_tiles.py:278-280
No upper bound is enforced on `--max-zoom`. Tile counts grow as 4× per zoom level: the default 200 km radius produces ~55 k tiles at zoom 14, but ~3.5 M tiles at zoom 19 and billions above that. Because `enumerate_tiles` builds the complete tile list in memory before any downloading begins, an overly high value can exhaust RAM or lock the script into days of computation with no warning.

```suggestion
    MAX_ZOOM_LIMIT = 19
    if args.min_zoom < 0 or args.max_zoom < 0 or args.min_zoom > args.max_zoom:
        print("invalid zoom range", file=sys.stderr)
        return 2
    if args.max_zoom > MAX_ZOOM_LIMIT:
        print(f"--max-zoom {args.max_zoom} exceeds {MAX_ZOOM_LIMIT}; tile counts grow exponentially and most servers cap at z19.", file=sys.stderr)
        return 2
```

Reviews (5): Last reviewed commit: "Harden prefetch tile write failures" | Re-trigger Greptile

Comment thread tools/prefetch_map_tiles.py Outdated
Comment thread prefetch_map_tiles.stderr.log Outdated
Comment on lines +1 to +16
FAIL https://tile.openstreetmap.org/13/2552/2984.png -> tiles\13\2552\2984.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2985.png -> tiles\13\2552\2985.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2986.png -> tiles\13\2552\2986.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2987.png -> tiles\13\2552\2987.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2988.png -> tiles\13\2552\2988.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2989.png -> tiles\13\2552\2989.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2990.png -> tiles\13\2552\2990.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2552/2991.png -> tiles\13\2552\2991.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2984.png -> tiles\13\2553\2984.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2985.png -> tiles\13\2553\2985.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2986.png -> tiles\13\2553\2986.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2987.png -> tiles\13\2553\2987.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2988.png -> tiles\13\2553\2988.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2989.png -> tiles\13\2553\2989.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2990.png -> tiles\13\2553\2990.png (HTTP Error 404: Not Found)
FAIL https://tile.openstreetmap.org/13/2553/2991.png -> tiles\13\2553\2991.png (HTTP Error 404: Not Found)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log files should not be committed

These log files (prefetch_map_tiles.stderr.log and prefetch_map_tiles.stdout.log) are runtime output artifacts from a test run and should not be tracked in the repository. Consider removing them from the commit and adding *.log or these specific filenames to .gitignore.

Prompt To Fix With AI
This is a comment left during a code review.
Path: prefetch_map_tiles.stderr.log
Line: 1-16

Comment:
**Log files should not be committed**

These log files (`prefetch_map_tiles.stderr.log` and `prefetch_map_tiles.stdout.log`) are runtime output artifacts from a test run and should not be tracked in the repository. Consider removing them from the commit and adding `*.log` or these specific filenames to `.gitignore`.

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brothercorvo please remove the log files from your PR

Comment thread tools/prefetch_map_tiles.py
Comment thread tools/README.md Outdated
Comment on lines +53 to +54
- The script defaults to the same OpenStreetMap tile URL currently used by the firmware.
- If you use a different tile provider, pass `--url-template`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing OSM tile usage policy warning

The README should note that the default tile server (tile.openstreetmap.org) prohibits bulk downloading and offline prefetching per the OSM Tile Usage Policy. Users should be directed to use a self-hosted tile server or a provider that explicitly permits offline/bulk downloads.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tools/README.md
Line: 53-54

Comment:
**Missing OSM tile usage policy warning**

The README should note that the default tile server (`tile.openstreetmap.org`) prohibits bulk downloading and offline prefetching per the [OSM Tile Usage Policy](https://operations.osmfoundation.org/policies/tiles/). Users should be directed to use a self-hosted tile server or a provider that explicitly permits offline/bulk downloads.

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brothercorvo Greptile is right, here is a screenshot from the usage policy. This is against the OSM Tile Usage Policy linked above

Image

Comment thread tools/prefetch_map_tiles.py
Comment thread tools/prefetch_map_tiles.py Outdated
Comment thread tools/prefetch_map_tiles.py Outdated
@giu-Expedibox

Copy link
Copy Markdown

@torlando-tech this is also ready for review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants