de.siegmar.logbackgelf.GelfHttpAppender gives errors without message

[gbarsan]

We are using this settings:

<appender name="GELFNEW" class="de.siegmar.logbackgelf.GelfHttpAppender">
<uri>https://logs.example.com/gelf</uri>
<encoder class="de.siegmar.logbackgelf.GelfEncoder">
<staticField>application:${application}</staticField>
</encoder>
</appender>
<appender name="ASYNC_GELF" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="GELFNEW"/>
<queueSize>500</queueSize>
<neverBlock>true</neverBlock>
</appender>
<root level="DEBUG">
<appender-ref ref="ASYNC_GELF"/>
</root>

But it leads to this error on STDOUT:

12:08:02,466 |-ERROR in de.siegmar.logbackgelf.GelfHttpAppender[GELFNEW] - Invalid response from https://logs.example.com/gelf. Code: 413; Message:

Does anyone know how to handle this? How can we solve it?

Many thanks for any help!

[osiegmar]

HTTP error 413 resolves to "Content Too Large". Graylog (or the underlaying Elasticsearch) typically throws this error if the message is over ~32 KiB. Maybe that helps? What does the Graylog server log tell you?

[gbarsan]

We're trying to figure out how to find the according entries and see what Graylog "says". At least on NGINX (which we use as reverse proxy/for SSL termination) we don't see any errors. We will also try to implement a custom message filter and limit the size.

[shurkanTwo]

In case this is still useful (or for anyone else hitting it): the empty Message: is the main clue. It points at Graylog's GELF HTTP input, not at NGINX.

GelfHttpAppender sends one log event per POST, GZIP-compressed by default (buildPackage). Graylog's HTTP inputs decompress the body first and then aggregate it with a size limit of Max. HTTP chunk size, which defaults to 64 KiB (AbstractHttpTransport, default). A single event whose uncompressed GELF JSON is larger than that, typically a big stack trace or a huge DEBUG payload, is rejected by Netty's HttpObjectAggregator with a 413 and an empty body. The appender then logs exactly Code: 413; Message: (source).

That also explains why NGINX shows nothing: if NGINX rejected the request itself (client_max_body_size, default 1m, which applies to the compressed body), its error log would contain client intended to send too large body and the response would be its HTML error page, not an empty body.

Two ways to fix it:

1. Raise the limit on the Graylog side: System → Inputs → (your GELF HTTP input) → Edit → Max. HTTP chunk size, e.g. 1048576 for 1 MiB. If you go well above 1 MiB, raise client_max_body_size in NGINX too.

2. Cap the event size on the client side with the encoder layouts (same syntax as the project's http-config.xml), for example by truncating the message and limiting the stack trace depth:

   <encoder class="de.siegmar.logbackgelf.GelfEncoder">
       <shortMessageLayout class="ch.qos.logback.classic.PatternLayout">
           <pattern>%.-1000m%nopex</pattern>
       </shortMessageLayout>
       <fullMessageLayout class="ch.qos.logback.classic.PatternLayout">
           <pattern>%.-20000m%n%ex{50}</pattern>
       </fullMessageLayout>
       <staticField>application:${application}</staticField>
   </encoder>

   %.-N truncates the message after N characters (both short_message and full_message carry it by default) and %ex{50} keeps the first 50 stack trace lines. Note the root logger is at DEBUG, which tends to produce the occasional very large message.