transform-middleware for webrequests

[coderofsalvation]

This is an idea from the slackchat:

me: btw. just curious, have you considered adding middleware for jasonette web-requests?
My case above is obviously not a good example for that.
But i could imagine transforming incoming json into something else.
Let's say you want to search thru all github AND gitlab repositories using 2 api-calls, which both have a slightly different array of results.
It would simply things a lot if the jasonette-app could use one internal json-format.
This would be possible, if you could just write 2 middleware functions (gitlab + github) which transforms the incoming result into the internal json-format..and render it


ethan: that's an interesting idea. i guess the challenge is how to actually implement it
if you have good ideas, feel free to share more details on github as an issue

A simple middleware pattern could allow all jasonette-clients (web/ios/android) to aggregate or transform incoming data and/or outgoing data.

Usecases

• to aggregate multiple api's into a single internal format
• to patch incoming data (e.g. missing json-property due to bug or version-change of api)
• to patch outgoing data (e.g. add specific required headers like RESTFUL pagination)

Example middleware in jasonette json

This is just thinking out loud, and can probably be improved a LOT

{
  "$jason:{
    head:{
      webrequest:{
        "POST":[
           "url":".*gitlab\.com",
           "transform":{
               "headers":{
                   "X-API-TOKEN":"fe34fe3fe43",
                   "X-Total-Count": "{{mysettings.max_items}}"
               },
               "payload":{
                   "page":"{{mysettings.page}}",
               },
               "result":{
                   error: "{{this.gitlab_repositories.length ? true : false}}",
                   repos: "{{this.gitlab_repositories}}"
               }
           }
        ]
      }
    }
  }
}

Basic idea

whenever jasonette does a webrequest 
  ╰￫ check if the url + method matches with the `webrequest` section in `head`
    ╰￫ call selecttransform on the payload+header
       ╰￫ forward the result of `process(request, doActualRequest )
          ╰￫ call selecttransform on the `result`
             ╰￫ forward the result of `process(result, renderTemplate )` 

Considerations

Im curious about your feedback, and I'm definately not sure whether we should pursue this path.

reasons to not support it

Sometimes deciding not to write a feature can be also be a huge blessing.
All the logic above could also be written as a proxy-api server-implementation, instead of a client implementation..

reasons to support it

These days all focus is rather on the frontend, that's where the money is.
Therefore flexible transformation-features of i/o can be very attractive, all power to the frontend-dev.
CRUD Server-development seems to be a necessary evil these days (imho), preferrably abstracted away by phrases like 'serverless','apaas' etc.

[coderofsalvation]

https://www.npmjs.com/package/xhook <-- Im going to experiment to see if i can inject this into an agent.

[gliechtenstein]

There are a couple of important details that may decide the scope of this problem.

There are currently 3 types of network requests:

1. The main markup URL request that loads when the app opens for the first time
2. All subsequent URLs that open a new view through href
3. $network.request action

I am guessing this is about the 3rd item only. In this case I think the spec should somehow tie into $network.action markup and not the global markup because it's only relevant in the $network.request context.

If this is the case (if it's about multiplexed fetch + processing), couldn't we write a custom action to do something like this? For example we can create a custom action like

{
  "$jason": {
    "head": {
      "actions": {
        "fetch": {
          "type": "$network.request",
          "options": {
             "url": "https://gitlab.com/.....",
             "options": {
                "headers":{
                   "X-API-TOKEN":"fe34fe3fe43",
                   "X-Total-Count": "{{$jason.max_items}}"
                },
                "payload":{
                   "page":"{{$jason.page}}",
                }
              }
            },
            "success": {
               "type": "$return.success",
               "options": {
                   error: "{{$jason.gitlab_repositories.length ? true : false}}",
                   repos: "{{$jason.gitlab_repositories}}"
               }
            }
         ...

Above fetch custom action would take a JSON argument that has max_items and page as attributes, make a $network.request, and then return a value through $return.success action http://docs.jasonette.com/actions/#returnsuccess

One could call this in the following manner using trigger http://docs.jasonette.com/actions/#triggering-actions :

{
  "$jason": {
    "head": {
      "actions": {
        "fetch": { ... },
        "$load": {
          "trigger": "fetch",
          "options": {
            "max_items": 20,
            "page": 2
          },
          "success": {
            "type": "$util.alert",
            "options": {
              "title": "response",
              "description": "Error: {{$jason.error.toString()}} Fetched Length: {{$jason.repos.length}}"
            }
...

This approach wouldn't require any additional code on Jasonette side, but you can implement the entire logic as just a JSON markup and reuse it everywhere (maybe even turn it into mixins)

What I thought was interesting about the idea you shared in the chatroom was the possibility of fetching multiple sources and processing them into a single result.

But upon thinking about this more, I think we might be able to take the same approach as above and use $require instead of $network.request to solve the same problem.

Are there any other cases I missed out? Also please let me know what you think.

[coderofsalvation]

Hm this seems to be a better approach.
i agree that $network.request could be a better place for it (and mixins allow flexible reuse).

aggregation of data

Jasonette seems to already support aggregation of data using $require and mixins.
I'm not really sure about how to aggregate data, i can imagine different ways to support this:

type	javascript way	transformation of request i/o
parallel	Promise.all	1, after all calls are finished
serial	each request, Promise.then(request1).then(request2).then(..)	before/after each call

Not really sure how to bundle/pipeline several requests into one.
Questions:

• how could we somehow 'pipe' a custom action's output into another one?