✨ add feature parse content-type application/x-ndjson

[huykn]

Description

add feature parse content-type application/x-ndjson
ex input body json

{"name":"john"}
{"name":"doe"}

expect return array via call method BodyParser

type Demo struct {
    Name string `json:"name" xml:"name" form:"name" query:"name"`
}
c.Request().Header.SetContentType(MIMEApplicationXNDJSON)
c.Request().SetBody([]byte("{\"name\":\"john\"}\n{\"name\":\"doe\"}\n"))
c.Request().Header.SetContentLength(len(c.Body()))
var out []Demo
utils.AssertEqual(t, nil, c.BodyParser(&out))
utils.AssertEqual(t, 2, len(out))
utils.AssertEqual(t, "john", out[0].Name)
utils.AssertEqual(t, "doe", out[1].Name)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• For new functionalities I follow the inspiration of the express js framework and built them similar in usage
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation - /docs/ directory for https://docs.gofiber.io/
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• If new dependencies exist, I have checked that they are really necessary and agreed with the maintainers/community (we want to have as few dependencies as possible)
• I tried to make my code as fast as possible with as few allocations as possible
• For new code I have written benchmarks so that they can be analyzed and improved

Commit formatting:

Use emojis on commit messages so it provides an easy way of identifying the purpose or intention of a commit. Check out the emoji cheatsheet here: https://gitmoji.carloscuesta.me/

[ReneWerner87]

thanks for the feature

but i'm not sure if we should support any kind of body parsing as a framework or if it's better to use extension packages for it and focus on the general ones

@efectn @gaby whats your opinion

[gaby]

@ReneWerner87 I feel like this should be something in fasthttp instead of GoFiber. It doesn't make sense to add all this custom logic to the framework ndjson is very useful, specially when dealing with processing logs/metrics. Is there a library that could help with this?

[efectn]

thanks for the feature

but i'm not sure if we should support any kind of body parsing as a framework or if it's better to use extension packages for it and focus on the general ones

@efectn @gaby whats your opinion

I do agree. If we try to implement every JSON implementations, the core will get really bloated.

[ReneWerner87]

cool would be a plugin system

where we could declare a parser for the content type in a map, then you could add these parsers separately and only the core parsers are included

pseudocode
type Parser = (&ctx, &schemaItem) => {}

app.bodyParser["application/x-www-form-urlencoded"] = formDataParserClosure
app.bodyParser["multipart/form-data"] = multiformDataParserClosure
app.bodyParser["application/json"] = jsonParserClosure
app.bodyParser["application/xml>"] = xmlParserClosure
app.bodyParser["text/xml"] = xmlParserClosure
....

like express with the bodyParser

[gaby]

@huykn @ReneWerner87 @efectn According to this article you can already parse ndjson using native Go.

https://abhinavg.net/2020/07/02/parse-ndjson/

decoder := json.NewDecoder(src)
for decoder.More() {
    var order Order
    if err := decoder.Decode(&order); err != nil {
        return fmt.Errorf("parse order: %w", err)
    }

    fmt.Println(order)
}

[huykn]

@huykn @ReneWerner87 @efectn According to this article you can already parse ndjson using native Go.

https://abhinavg.net/2020/07/02/parse-ndjson/

decoder := json.NewDecoder(src)
for decoder.More() {
    var order Order
    if err := decoder.Decode(&order); err != nil {
        return fmt.Errorf("parse order: %w", err)
    }

    fmt.Println(order)
}

@gaby i think we have custom config apply for use 3rd json at: https://github.com/gofiber/fiber/blob/master/docs/guide/faster-fiber.md
if we use only json from encoding/json i think logic is not correct. i'm not sure any json 3rd support ndjson.

[ReneWerner87]

I tested this and it was not working

[huykn]

I tested this and it was not working

for feature in this merge request or smt?

[ReneWerner87]

Feature is working
I mean i tested the parsing of the njson content with the default parser(without extra logic) and that's not working

[huykn]

Feature is working I mean i tested the parsing of the njson content with the default parser(without extra logic) and that's not working

do you think this feature should be merged or not? or do we need to change to add parsers separately and only the core parsers are included?

[ReneWerner87]

Good question, we will discuss it and then decide (unfortunately not everyone is here right now)

[ReneWerner87]

sorry for the late reply
we have decided not to adopt this feature request

because we can't possibly have parsers in the core for all content types that exist
we currently only want to provide the most common ones in the core

in v3 we can try to find a way to add parser from outside and in the code we can have a default parser for this in a separate folder or through additional packages from the contrib package

or we add functionality externally that makes it easier to write custom parsing externally

you are welcome to support us in v3 with a proposal