OIDC keycloak forward auth what am I missing?

[Astrohelo]

Hi there I have been trying to perfect the forward auth for some months now with little to no luck so here we go. I have tried many forward auth images.
So basically I login with
https://keycloak.dev.mydomain.uk/realms/myrealm/protocol/openid-connect/token
I get the access token and add it to bearer token in
https://api.dev.mydomain.uk/api/events/findOneEvent?id=6626d6330666a1f4ad7149c6
this is the message in docker logs:
2024-07-29 12:39:07 time="2024-07-29T10:39:07Z" level=debug msg="Authenticating request" cookies="[]" handler=Auth host=api.dev.mydomain.uk method=GET proto=https rule=default source_ip=172.25.0.1 uri="/api/events/findOneEvent?id=6626d6330666a1f4ad7149c6"
2024-07-29 12:39:07 time="2024-07-29T10:39:07Z" level=debug msg="Set CSRF cookie and redirected to provider login url" csrf_cookie="_forward_auth_csrf_f0450c=f0450c6d398dc4e57ac88d5532970162; Path=/; Domain=api.dev.mydomain.uk; Expires=Mon, 29 Jul 2024 11:39:07 GMT; HttpOnly" handler=Auth host=api.dev.mydomain.uk login_url="https://keycloak.dev.mydomain.uk/realms/myrealm/protocol/openid-connect/auth?client_id=flutter-client&redirect_uri=https%3A%2F%2Fapi.dev.mydomain.uk%2F_oauth&response_type=code&scope=openid+profile+email&state=f0450c6d398dc4e57ac88d5532970162%3Aoidc%3Ahttps%3A%2F%2Fapi.dev.mydomain.uk%2Fapi%2Fevents%2FfindOneEvent" method=GET proto=https rule=default source_ip=172.25.0.1 uri="/api/events/findOneEvent?id=6626d6330666a1f4ad7149c6"
and the redirect returns me to the keycloak login page I am not sure where the problem lies anymore

traefik?
forward auth?
keycloak?
cloudflare?
postman?
Tried to look into all of them but spent like 200 hours on this already...
previously paste.dev.mydomain.uk was working with forward auth (i guess using cookies) perfectly with the mesosphere but now if i login here i get Not authorized

version: '3.4'
secrets:
  cloudflare-token:
    file: "./secrets/cloudflare-token.secret"
  cloudflare-email:
    file: "./secrets/cloudflare-email.secret"    
   
services:
  traefik:
    image: "traefik:v2.11"
    restart: always
    container_name: "traefik"
    networks:
      - mynetwork
    command:
      - "--log.level=INFO"
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --api.dashboard=true
      # Set up LetsEncrypt certificate resolver
      - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
      - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
      - --certificatesResolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,1.0.0.1:53
      - --certificatesResolvers.letsencrypt.acme.dnschallenge.delayBeforeCheck=20
      - --certificatesresolvers.letsencrypt.acme.email=axelvarai2k@gmail.com
      - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
      - --certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
      # Set up an insecure listener that redirects all traffic to TLS
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.web.http.redirections.entrypoint.to=websecure
      - --entrypoints.web.http.redirections.entrypoint.scheme=https
      # Set up the TLS configuration for our websecure listener
      - --entrypoints.websecure.http.tls=true
      - --entrypoints.websecure.http.tls.certResolver=letsencrypt
      - --entrypoints.websecure.http.tls.domains[0].main=mydomain.uk
      - --entrypoints.websecure.http.tls.domains[0].sans=*.mydomain.uk
    secrets:
      - "cloudflare-token"
      - "cloudflare-email"
    environment:
      - "CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-token"
      - "CF_API_EMAIL_FILE=/run/secrets/cloudflare-email"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "./certs:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.dev.mydomain.uk`)"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"      
      - "traefik.http.routers.traefik.middlewares=traefik-auth"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:{SHA}encryptedpassword"

  paste-bin:
    container_name: paste-bin
    image: ghcr.io/enchant97/hasty-paste:latest
    restart: unless-stopped
    depends_on:
      - traefik-forward-auth
    networks:
      - mynetwork
    labels:
      - "traefik.enable=true"
      - 'traefik.http.routers.paste.rule=Host(`paste.dev.mydomain.uk`)'
      - "traefik.http.routers.paste.entrypoints=websecure"
      - "traefik.http.routers.paste.tls.certresolver=letsencrypt"   
      - "traefik.http.routers.paste.middlewares=traefik-forward-auth"

  traefik-forward-auth:
    image: thomseddon/traefik-forward-auth:latest
    restart: on-failure
    environment:
      - DEFAULT_PROVIDER=oidc
      - PROVIDERS_OIDC_ISSUER_URL=https://keycloak.dev.mydomain.uk/realms/myrealm
      - PROVIDERS_OIDC_CLIENT_ID=flutter-client
      - PROVIDERS_OIDC_CLIENT_SECRET=secretfromkeycloak
      - SECRET=generatedsecret
      - INSECURE_COOKIE=true
      - COOKIE_DOMAIN=dev.mydomain.uk
      - URL_PATH=/_oauth
      - LOG_LEVEL=trace
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.traefik-forward-auth.forwardauth.address=http://traefik-forward-auth:4181"
      - "traefik.http.middlewares.traefik-forward-auth.forwardauth.authResponseHeaders=X-Forwarded-User,X-Auth-User,Authorization"
      - "traefik.http.middlewares.traefik-forward-auth.forwardauth.authRequestHeaders=Authorization"
      - "traefik.http.services.traefik-forward-auth.loadbalancer.server.port=4181"
      - "traefik.http.routers.traefik-forward-auth.middlewares=traefik-forward-auth"
      - "traefik.http.middlewares.traefik-forward-auth.forwardauth.trustForwardHeader=true"
    networks:
      - mynetwork

  keycloakdb:
    image: postgres:16.2-alpine
    container_name: keycloakdb
    environment:
        - POSTGRES_DB=keycloak
        - POSTGRES_USER=keycloak
        - POSTGRES_PASSWORD=password
        - POSTGRES_ROOT_PASSWORD=password
    networks:
      - mynetwork
    ports:
      - "5432:5432"
    volumes:
      - keycloakdata:/var/lib/postgresql/data
    labels:
      - "traefik.enable=false"

  keycloak:
    image: quay.io/keycloak/keycloak:24.0
    container_name: keycloak
    hostname: keycloak
    environment:
      - KC_HOSTNAME_STRICT=false
      - KC_DB=postgres
      - KC_DB_URL=jdbc:postgresql://keycloakdb/keycloak
      - KC_DB_URL_PORT=5432
      - KC_DB_USERNAME=keycloak
      - KC_DB_PASSWORD=password
      - KC_DB_SCHEMA=public
      - KC_LOG_LEVEL=info
      - KC_FEATURES=docker
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=password
      - KC_PROXY=edge
    networks:
      - mynetwork
    depends_on:
      - traefik
      - keycloakdb
    labels:
        - "traefik.enable=true"
        - "traefik.http.routers.keycloak.rule=Host(`keycloak.dev.mydomain.uk`)"
        - "traefik.http.routers.keycloak.entrypoints=websecure"
        - "traefik.http.routers.keycloak.tls.certresolver=letsencrypt"
    entrypoint: ["/opt/keycloak/bin/kc.sh", "start-dev"]

  mongodb:
    image: mongo:latest
    container_name: mongodb
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  events:
    image: ${DOCKER_REGISTRY-}eventsapi
    build:
      context: .
      dockerfile: Events/Events.Api/Dockerfile
    ports:
      - "8080"
    environment:
      - "EventBusSettings:HostAddress=amqp://guest:guest@rabbitmq:5672"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.events.rule=Host(`api.dev.mydomain.uk`) && PathPrefix(`/api/events`)"
      - "traefik.http.routers.events.entrypoints=websecure"
      - "traefik.http.routers.events.tls.certresolver=letsencrypt"   # i know this shouldnt be must
      - "traefik.http.routers.events.middlewares=traefik-forward-auth@docker" # i am not sure the @docker needed
    networks:
      - mynetwork
    depends_on:
      - traefik
      - mongodb
      - rabbitmq
  
  rabbitmq:
    image: rabbitmq:3.13.0-management
    restart: always
    ports:
      - "5672:5672"
      - "15672:15672"
    depends_on:
      - traefik
    networks:
      - mynetwork

networks:
  mynetwork:
    driver: bridge

volumes:
  mongo-data:
  keycloakdata:  

[Astrohelo]

Also keep in mind I don't use any other traefik conf file just docker-compose

[ryanshaut]

I get a 'Not Authorized' error sometimes too. If I remove the url slug (https://codespaces.example.com/_oauth?... > https://codesapces.example.com), I'm able to open the appliction correctly.

My docker setup is similar to yours and the forward auth logs look similar.