CGIMAP container

[Rub21]

This is the container for serving the openstreetmap-cgimap service; these changes include the Helm configuration and the Docker file.

cc. @batpad

[Rub21]

I think this is not going to work, since we implemented cgimap in the same container like :OpenHistoricalMap/ohm-deploy#228

[batpad]

@Rub21 this is looking really good to me.

But don't we need to serve CGIMap on the same hostname? And then handle URL re-writing, or can it work at like cgimap.openhistoricalmap.org? If we need to serve it on the same hostname, we will need to handle the URL re-writing at the Ingress level, and handle hitting the correct container based on the URL Regexes. Am not fully sure if my understanding of how this is supposed to work is correct though, but I'd be happy to pair and figure out next steps here.

[Rub21]

But don't we need to serve CGIMap on the same hostname? And then handle URL re-writing, or can it work at like cgimap.openhistoricalmap.org?

I think we need to serve in URL like : cgimap.openhistoricalmap.org it is going to be created by ingress it works as load balancer as well and then in the background we can run many pods as we want.

[spwoodcock]

For my temp work in hotosm/osm-sandbox (planned to merge all work there into osm-seed), I made a compose config based off the official Dockerfile in the openstreetmap-cgi repo:

https://github.com/hotosm/osm-sandbox/blob/main/docker-compose.yml#L120-L149

  osm-cgi:
    image: "ghcr.io/hotosm/osm-sandbox/cgimap:${CGIMAP_VERSION:-v2.0.1}"
    build:
      context: https://github.com/zerebubuth/openstreetmap-cgimap.git#${CGIMAP_VERSION:-v2.0.1}
      dockerfile: docker/debian/Dockerfile_bookworm
    depends_on:
      osm-db:
        condition: service_healthy
    environment:
      CGIMAP_HOST: osm-db
      CGIMAP_DBNAME: openstreetmap
      CGIMAP_USERNAME: openstreetmap
      CGIMAP_PASSWORD: openstreetmap
      CGIMAP_MEMCACHE: memcached
      CGIMAP_RATELIMIT: 204800
      CGIMAP_MAXDEBT: 250
      CGIMAP_MODERATOR_RATELIMIT: 1048576
      CGIMAP_MODERATOR_MAXDEBT: 1024
      CGIMAP_PORT: 8000
      CGIMAP_INSTANCES: 3
    networks:
      - osm-net
    restart: unless-stopped
    # Override from https://github.com/zerebubuth/openstreetmap-cgimap/blob/master/docker/debian/Dockerfile_bookworm
    # defaults: --max-payload 50000000L (50MB) --max-changeset-elements 10000
    # --map-nodes 50000 --map-area 0.25 (square degrees)
    # Note the Ruby server max-changeset-elements is hardcoded (but this config should override it):
    # https://github.com/openstreetmap/openstreetmap-website/blob/cee9818dfc9ac3f6eae01cdb51df9093ae5d1322/app/models/changeset.rb#L58
    entrypoint: >
      sh -c "/usr/bin/openstreetmap-cgimap --pidfile /tmp/cgimap.pid \
      --logfile=/proc/1/fd/1 --daemon \
      --max-payload 500000000 --max-changeset-elements 1000000 \
      --map-nodes 10000000 --map-area 10 && \
      tail --pid=\$(cat /tmp/cgimap.pid) -f /dev/null"

Then I use an Nginx proxy config to route between Rails / CGI:

https://github.com/hotosm/osm-sandbox/blob/main/nginx/templates/osm.conf.template#L1-L78

upstream openstreetmap {
    server osm:3000 max_fails=1 fail_timeout=2s;
    keepalive 32;
}

upstream cgimap {
    server osm-cgi:8000;
    keepalive 32;
}

server {
    # Redirect all HTTP traffic to HTTPS
    listen 80;
    server_name ${DOMAIN};
    return 301 https://$host$request_uri;
}

server {
    # Default handler for port 443 (HTTPS)
    listen 443 ssl reuseport;
    server_name ${DOMAIN};
    
    ssl_certificate /etc/letsencrypt/live/${DOMAIN}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/${DOMAIN}/privkey.pem;
    include /etc/nginx/options-ssl-nginx.conf;
    include /etc/nginx/options-security.conf;

    client_max_body_size 10M;

    add_header 'Content-Security-Policy' 'upgrade-insecure-requests';

    # Route specific paths to cgimap
    location ~ ^/api/0\.6/map$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_pass cgimap;
    }

    location ~ ^/api/0\.6/(nodes|ways|relations)$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_pass cgimap;
    }

    location ~ ^/api/0\.6/(way|relation)/([^/]+)/full$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_pass cgimap;
    }

    location ~ ^/api/0\.6/(node|way|relation)/([^/]+)$ {
        include /etc/nginx/fastcgi_params;
        if ($request_method ~ ^(GET|HEAD)$) {
            fastcgi_pass cgimap;
        }
    }

    location / {
        # Request headers
        proxy_set_header Host                $http_host;
        proxy_set_header X-Real-IP           $remote_addr;
        proxy_set_header X-Forwarded-Proto   $scheme;
        proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host    $http_host;
        proxy_set_header X-Forwarded-Server  $http_host;
        proxy_set_header X-Forwarded-Port    $server_port;

        # Disable buffer to temp files, tweak buffer for memory
        proxy_max_temp_file_size 0;
        proxy_buffer_size 64k;
        proxy_buffers 8 64k;
        proxy_busy_buffers_size 64k;

        proxy_pass http://openstreetmap;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Let me know your thoughts on this approach / if one is preferred over the other!

[spwoodcock]

Any thoughts on this approach?

It would affect the main OSM web container too though.

Replacing lighttpd and passenger with an nginx container for compose and then simply the nginx ingress for kubernetes (routing rules for osm-web and osm-cgimap) 😄

[mmd-osm]

--max-changeset-elements 1000000

I’m wondering why you’re bumping this value (which is not recommended in any way). Curiously, the nginx config doesn’t include an endpoint where this setting is relevant. As it stands this change has no impact.

[Rub21]

Thank you everyone for your input here.

These improvements aim to split the Rails and CGIMAP services. I’ve been testing this setup in a Kubernetes environment and got it working — it could also work in a Docker-based environment.

The goal is to run Rails and CGIMAP in separate containers, each with its own autoscaling, to better allocate resources when there are many users in the app and to optimize costs as well.

I’m also trying to use a configuration somewhat similar to what’s used in OSM Chef, using Apache but without relying on Ruby-based configuration.

It also seems that CGIMAP and Rails run on the same server in OSM, but those servers are quite large with enough resources, compared to the smaller-scale projects that use osm-seed.

[Rub21]

I don't see the additional build flags we're using in https://github.com/zerebubuth/openstreetmap-cgimap/blob/v2.0.1/docker/debian/Dockerfile_bookworm#L23
Production binaries should be compiled in "Release" mode...

@mmd-osm You are right. I think it would be better to use your Dockerfile version. Are there any official CGIMAP Docker images available somewhere, or does it need to be compiled from the Dockerfile?

[mmd-osm]

I'm afraid, the only "official" shipment channel for osm.org is the Debian repo over at https://apt.openstreetmap.org/pool/main/o/openstreetmap-cgimap/ ... Besides, the Dockerfiles need to be built like you said. They're not published in any Docker repo at the moment.

[Rub21]

@mmd-osm I am using the code that you generated in the repo to build the docker image, with some small fixed for cloning and for starting the process under de docker container. https://github.com/developmentseed/osm-seed/blob/cgmap/images/cgimap/Dockerfile

ENV CGIMAP_GITSHA=8ea707e10aeab5698e6859856111816d75354592

the gitsha refers to the version v2.0.1.

[Rub21]

This CGImap docker image should be stored in Docker Hub or in another registry we decide on, as discussed here: #363
cc. @batpad

[spwoodcock]

@mmd-osm would you consider it if I made a PR for an image building Github workflow in cgimap --> publish in GHCR?

[mmd-osm]

We discussed this topic in zerebubuth/openstreetmap-cgimap#332 already. Unfortunately, I cannot manage anything on GHCR as collaborator on the openstreetmap-cgimap repo, so the answer is no at the moment.

[spwoodcock]

Gotcha, thanks for the info😄

As a workaround, it would be pretty easy to do the builds in this repo for now, similar to:
https://github.com/baosystems/docker-postgis/blob/multiarch/.github/workflows/multiarch.yml

(until we can discuss further / get repo permissions for the workflow upstream)

Also happy to contribute this, if it helps 👍