Raster tiles not generated due to 404 on sources

[utagawal]

Using version 5.5.0.pre-5

With maptehorn configured as a source for raster-dem using http endpoint with webp tiles, the endpoint returns 404 when tiles are not available at certain zoom levels.

The challenge is that some areas goes to Zoom 17 while others to zoom 12, meaning you get 404 if you query with http at more important zoom level in these areas.

When such situation happens a lot it hungs the creation of raster tiles.

For the record, tilserver-gl is configured with "maxRendererPoolSizes": [6] (running with 8 vCPUs)

and the source used is defined as :

"Mapterhorn": {
  "type": "raster-dem",
  "url": "https://tiles.mapterhorn.com/tilejson.json"
}

Here is the diag thanks to a claude.ai script :

============================================
HUNG RENDERER DIAGNOSTICS

1. Incomplete/Hung Tile Requests:

---

Looking for tiles with no completion (- - - - ms)...
[sudo] password for lilian:
Found 10250 incomplete tile requests

Last 20 hung requests:
GET /styles/utagawavtt/15/15926/10083.png - - - - ms
GET /styles/utagawavtt/15/15927/10083.png - - - - ms
GET /styles/utagawavtt/15/15924/10082.png - - - - ms
GET /styles/utagawavtt/15/15924/10079.png - - - - ms
GET /styles/utagawavtt/15/15928/10082.png - - - - ms
GET /styles/utagawavtt/15/15930/10082.png - - - - ms
GET /styles/utagawavtt/15/15929/10082.png - - - - ms
GET /styles/utagawavtt/15/15931/10082.png - - - - ms
GET /styles/utagawavtt/15/15926/10078.png - - - - ms
GET /styles/utagawavtt/15/15927/10082.png - - - - ms
GET /styles/utagawavtt/15/15929/10078.png - - - - ms
GET /styles/utagawavtt/15/15928/10078.png - - - - ms
GET /styles/utagawavtt/15/15931/10078.png - - - - ms
GET /styles/utagawavtt/15/15930/10078.png - - - - ms
GET /styles/utagawavtt/15/15927/10078.png - - - - ms
GET /styles/utagawavtt/15/15924/10083.png - - - - ms
GET /styles/utagawavtt/15/15929/10083.png - - - - ms
GET /styles/utagawavtt/15/15928/10083.png - - - - ms
GET /styles/utagawavtt/15/15930/10083.png - - - - ms
GET /styles/utagawavtt/15/15931/10083.png - - - - ms

2. Node Process Status:

---

node 1 0.0 0.0 1008 4 ? Ss 21:29 0:00 /sbin/docker-init -- /usr/src/app/docker-entrypoint.sh
node 7 211 26.5 23495688 8166592 ? Sl 21:29 55:18 node /usr/src/app/
node 13 0.0 0.2 235704 76336 ? Sl 21:29 0:00 Xvfb :99 -nolisten unix
node 5545 0.0 0.0 29272 11608 ? Rsl 21:55 0:00 ps aux

3. Checking for Zombie Processes:

---

✓ No zombie processes

4. Process CPU Usage (indicates hung renderers):

---

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
node 7 211 26.4 23495688 8159796 ? Sl 21:29 55:20 node /usr/src/app/
node 1 0.0 0.0 1008 4 ? Ss 21:29 0:00 /sbin/docker-init -- /usr/src/app/docker-entrypoint.sh
node 13 0.0 0.2 235704 76336 ? Sl 21:29 0:00 Xvfb :99 -nolisten unix
node 5560 0.0 0.0 29272 11592 ? Rsl 21:55 0:00 ps aux --sort=-%cpu

5. Renderer Timeout Errors:

---

Found 85 renderer timeout errors

Recent timeout errors:
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer
Renderer timeout - destroying hung renderer

6. Renderer Pool Errors:

---

Pool not found for scale 1, mode tile
Pool not found for scale 1, mode tile
Pool not found for scale 1, mode tile
Pool not found for scale 1, mode tile
Pool not found for scale 1, mode tile
Pool not found for scale 1, mode tile

7. Current Network Connections:

---

0
Active connections: 0

8. Memory State:

---

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
b1419abe07dd tileserver-gl 548.63% 7.391GiB / 16GiB 46.20% 614MB / 89.4MB 88.7MB / 4.1kB 55

9. Testing Renderer Response:

---

Requesting a simple tile (zoom 3)...
✓ Response received in 6 seconds
File size: 149962 bytes
File type: /tmp/test_renderer.png: PNG image data, 256 x 256, 8-bit/color RGBA, non-interlaced

10. Last Successful Renders:

---

Last 10 successful PNG renders (status 200):
GET /styles/utagawavtt/17/66156/47585.png 200 89644 - 2275.372 ms
GET /styles/utagawavtt/17/66157/47585.png 200 82842 - 1977.783 ms
GET /styles/utagawavtt/17/66158/47585.png 200 96339 - 3915.344 ms
GET /styles/utagawavtt/17/66159/47585.png 200 145833 - 4104.110 ms
GET /styles/utagawavtt/17/66160/47585.png 200 125072 - 2389.927 ms
GET /styles/utagawavtt/17/66161/47585.png 200 132971 - 3186.034 ms
GET /styles/utagawavtt/17/66156/47586.png 200 91042 - 1734.396 ms
GET /styles/utagawavtt/17/66157/47586.png 200 112940 - 1602.282 ms
GET /styles/utagawavtt/17/66158/47586.png 200 123019 - 1520.554 ms
GET /styles/utagawavtt/3/4/2.png 200 149962 - 3555.896 ms

11. Error Pattern Analysis:

---

External fetch errors (404):
Count: 13691

External fetch timeouts:
Count: 179

Header errors:
Count: 0

12. Xvfb Display Server Status:

---

✓ Xvfb is running
node 13 0.0 0.2 235704 76336 ? Sl 21:29 0:00 Xvfb :99 -nolisten unix

============================================
DIAGNOSTIC SUMMARY

⚠ CRITICAL: 10250 hung tile requests detected

Likely causes:

1. External tile source (tiles.mapterhorn.com) timing out
2. Renderer pool exhausted
3. Memory pressure causing slowdowns

Recommended fixes:

1. Disable or fix external tile source
2. Restart container to clear hung renderers
3. Reduce maxRendererPoolSizes in config

⚠ PROBLEM: 13691 external tile 404 errors

The tiles.mapterhorn.com source is failing!
This is causing rendering delays and hangs.

Quick fix: Remove or comment out mapterhorn source in style JSON

⚠ PROBLEM: 179 external fetch timeouts

External tile sources are too slow/unreliable
This blocks the renderer pool

To fix hung renderers immediately:
sudo docker restart tileserver-gl

To prevent future hangs:

1. Fix external tile sources (see below)
2. Reduce renderer pool size
3. Add request timeouts
   ============================================

[utagawal]

A wild guess... would ther be a way to "sparse" a raster source to handle gently the 404 missing tiles ?

[acalcutt]

Any of the data sources specified in the config file should be able to use the "sparse" option, which would send a 410 cone error instead of a 404.

The code is supposed to handle that when rendering with https://github.com/maptiler/tileserver-gl/blob/master/src/serve_rendered.js#L1290-L1299
and
https://github.com/maptiler/tileserver-gl/blob/master/src/serve_rendered.js#L1356-L1365

Unless you mean allowing the rendered raster endpoints to have a sparse flag.... right now that isn't supported... but if all your sources have a sparse flage it should be able to render i would think.

[acalcutt]

Since it is the renderer failing it could be a missing resource, a maplibre-native bug, or an issue fetching a resource in this section of code ( https://github.com/maptiler/tileserver-gl/blob/master/src/serve_rendered.js#L1233-L1462 )

I have seen things where fetching a font or sprite resource fails in that section of code and it doesn't return what maplibre-native is asking for and locks it up. that section basically handles all resource request maplibre-native makes.

[acalcutt]

How are you specifying your maptehorn source? in the style or as a data source in the config? what does your config/style source look like?

[utagawal]

In the style.json file

[utagawal]

I have now a lot of processing time to generate raster tiles and plenty of hung rendered destroyed with raster tiles not produced :

Error fetching remote URL https://maps.utagawavtt.com/contours/planet-contours/12/2133/1414.pbf: This operation was aborted
GET /maintenance.html 404 155 - 16.107 ms
GET /maintenance.html 404 155 - 16.062 ms
GET /maintenance.html 404 155 - 6.631 ms
GET /maintenance.html - - - - ms
GET /maintenance.html 404 155 - 5.771 ms
GET /maintenance.html 404 155 - 3.630 ms
GET /data/planet/12/2118/1495.pbf - - - - ms
Renderer timeout - destroying hung renderer
GET /data/planet/12/2118/1494.pbf - - - - ms
GET /data/planet/14/8533/5888.pbf - - - - ms
FETCH TIMEOUT for https://tiles.mapterhorn.com/15/17064/11321.webp. The request took longer than 10 seconds to complete.
Error fetching remote URL https://tiles.mapterhorn.com/15/17064/11321.webp: This operation was aborted
FETCH TIMEOUT for https://tiles.mapterhorn.com/15/17064/11322.webp. The request took longer than 10 seconds to complete.
Error fetching remote URL https://tiles.mapterhorn.com/15/17064/11322.webp: This operation was aborted
GET /data/planet/14/8534/5888.pbf - - - - ms
GET /styles/utagawavtt/512/9/269/191.png - - - - ms
GET /styles/utagawavtt/512/9/270/191.png - - - - ms
GET /styles/utagawavtt/512/9/267/191.png - - - - ms
GET /styles/utagawavtt/512/9/268/191.png - - - - ms
FETCH TIMEOUT for https://tiles.mapterhorn.com/15/17065/11322.webp. The request took longer than 10 seconds to complete.
Error fetching remote URL https://tiles.mapterhorn.com/15/17065/11322.webp: This operation was aborted
GET /styles/utagawavtt/15/16929/11615.png - - - - ms
GET /styles/utagawavtt/15/16928/11615.png - - - - ms
GET /styles/utagawavtt/15/16930/11615.png - - - - ms
GET /styles/utagawavtt/15/16929/11614.png - - - - ms
GET /styles/utagawavtt/15/16930/11614.png - - - - ms
Renderer timeout - destroying hung renderer
GET /data/planet/13/4237/2990.pbf - - - - ms
GET /data/planet/13/4237/2989.pbf - - - - ms
GET /data/planet/13/4238/2989.pbf - - - - ms
Renderer timeout - destroying hung renderer
GET /data/planet/13/4237/2988.pbf - - - - ms
GET /data/planet/13/4238/2988.pbf - - - - ms
GET /data/planet/14/8475/5979.pbf - - - - ms
GET /data/planet/14/8475/5978.pbf - - - - ms
GET /data/planet/14/8476/5978.pbf - - - - ms
GET /data/planet/14/8476/5979.pbf - - - - ms
FETCH TIMEOUT for https://tiles.mapterhorn.com/15/17062/11321.webp. The request took longer than 10 seconds to complete.
Error fetching remote URL https://tiles.mapterhorn.com/15/17062/11321.webp: This operation was aborted
FETCH TIMEOUT for https://tiles.mapterhorn.com/15/17062/11320.webp. The request took longer than 10 seconds to complete.
Error fetching remote URL https://tiles.mapterhorn.com/15/17062/11320.webp: This operation was aborted
FETCH TIMEOUT for https://maps.utagawavtt.com/contours/planet-contours/12/2133/1415.pbf. The request took longer than 10 seconds to complete.
Error fetching remote URL https://maps.utagawavtt.com/contours/planet-contours/12/2133/1415.pbf: This operation was aborted
GET /data/planet/14/8475/5980.pbf - - - - ms
GET /data/planet/14/8476/5980.pbf - - - - ms
FETCH TIMEOUT for https://tiles.mapterhorn.com/15/17064/11320.webp. The request took longer than 10 seconds to complete.
Error fetching remote URL https://tiles.mapterhorn.com/15/17064/11320.webp: This operation was aborted
GET /data/planet/14/8535/5887.pbf - - - - ms
GET /styles/utagawavtt/15/16928/11613.png - - - - ms
GET /styles/utagawavtt/15/16931/11613.png - - - - ms
GET /styles/utagawavtt/15/16930/11613.png - - - - ms
GET /styles/utagawavtt/15/16928/11614.png - - - - ms
GET /styles/utagawavtt/15/16929/11613.png - - - - ms
GET /data/planet/14/8475/5981.pbf - - - - ms
GET /data/planet/14/8476/5981.pbf - - - - ms
GET /data/planet/14/8535/5886.pbf - - - - ms

[utagawal]

@acalcutt Have you been able to reproduce the issue ? Any lead on how to fix ?

[acalcutt]

No i haven't looked at it much, right now tileserver-gl requires a 410 error for sparse tiles to work, which would be needed for the variable depths of mapterhorn right now.

I did however find an issue when working on #1855 where using the same source multiple times was causing the remote http server to reject some connections. Based on suggestions from AI I tried to add in a cachekey so it would only open one connection to each source, which seemed to help with the issue I was seeing trying to use a terrain file as multiple sources.

[utagawal]

Any chance you could publish a pre container with the fix to see if that helps with the FETCH TIMEOUTs ?

I could rewrite 404 to 410 with a NGINX cache to see if that helps though.

[acalcutt]

I changed the latest version (v5.5.0-pre.6) so the sparse option now defaults to send a 404, which does seem to work in maplibre-gl-js like @wipfli mentioned. I also re-worked the rendering fetching so it will allow overzooming with a by default on errors (like 404) on non-vector sources and only create a blank tile when a 204 status code is received

I have also added a global 'sparse' option and a per data source sparse option which overrides it. When sparse is false a 204 error is sent for missing tiles and a blank tile gets generated (the old default). when it is true it sends a 404 and overzooming is allowed.

[acalcutt]

I was a bit conflicted on how to handle 204 vs 404, since it seems like 204 support was originally made to suppress browser console errors in #278 , which is why i left it a config option