From 38d12f0b9c55276490892d86030ad23673dbeb4b Mon Sep 17 00:00:00 2001 From: arkilic Date: Thu, 16 Mar 2017 11:37:23 -0400 Subject: [PATCH 1/3] First cut at databroker web service design document --- design/dbws-design.rst | 43 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 design/dbws-design.rst diff --git a/design/dbws-design.rst b/design/dbws-design.rst new file mode 100644 index 0000000..d7c6e4e --- /dev/null +++ b/design/dbws-design.rst @@ -0,0 +1,43 @@ +================================= +Data Broker Service Layer Design +================================= + +Motivation +========== + +Databroker has been actively developed and widely used over the past 2 years. +It has become more powerful and stable thanks to Dan Allan's efforts over the +past year. Its stability and wide acceptance as the de facto way access data +acquired in our beamlines. A new wave of requirements are reported as a result +of our community outreach. One of these requirements is a set of databroker +server implementations that will allow remote access to experimental data over +network. The service layer will also eliminate any issues that may occur as a +result of providing direct/library-level access to databases, allowing +us to implement data access policies. + +Challenges +=========== + +NSLS-II and other facilities have a wide variety of experimental setups that +generate data at different rates and structure. This poses a major challenge +for a single-backend implementation. Another challenge is the lack of sys admin +and dev-ops support. Making performant and reliable asynchronous servers with +low-latency is that are maintainable in the long term that are easy to deploy +is especially difficult with the amount of resources at hand. + + +Roadmap +=========== + +- Identify the data rates required to be transferred in NSLS-II databases. This +does not require a major effort, a simple investigation of existing metadatastore +and filestore will suffice + +- Identify the technologies to be used for transmitting ND arrays over the wire: + msgpack, protobuffe, h5serv, etc. This includes performance measurements + +- Have a first cut on existing databroker using .. _ujson: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-json-libraries/ + +- Prioritize the backends to be implemented, define timelines, and implement. + +- Add an authentication layer From 5bb113fe38c124beaa8c650fd63a4857d217b078 Mon Sep 17 00:00:00 2001 From: arkilic Date: Thu, 16 Mar 2017 14:55:03 -0400 Subject: [PATCH 2/3] Some server side information --- design/dbws-design.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/design/dbws-design.rst b/design/dbws-design.rst index d7c6e4e..44f71e6 100644 --- a/design/dbws-design.rst +++ b/design/dbws-design.rst @@ -41,3 +41,34 @@ and filestore will suffice - Prioritize the backends to be implemented, define timelines, and implement. - Add an authentication layer + + +Technology +============= + +Given there are plans for central storage and data management, the implementation +of the server layer must be scalable. This means a micro-service oriented +infrastructure that can eventually be placed behind load balancers is a good +intuitive choice. A fully RESTful approach is achiveable yet limiting given the +flexibility provided by the stack. This layer needs some discussion. We could +definitely adopt a RESTful approach (ugh, dreaded approach word) could be +adopted rather than handler per document type sort of approach of metadataservice +and amostra. + +Server Side Design +=================== + +HeaderHandler +------------ + +**get:** Decode jsonified query and return a serialize a list of dicts +If invalid request, return appropriate http error code. + + +EventSourceHandler +------------------ + +**get:** Using the uid of the header, obtains an event generator. +By paginating on this generator, returns the payload to the client side. +In the future, this bit will sort out the source of the data fill source +and redirect to the appropriate handler. From c85c9a58b5b7db5987e5c209f251694b1328d82e Mon Sep 17 00:00:00 2001 From: arkilic Date: Thu, 16 Mar 2017 15:37:09 -0400 Subject: [PATCH 3/3] Clarification --- design/dbws-design.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/design/dbws-design.rst b/design/dbws-design.rst index 44f71e6..a2b1f76 100644 --- a/design/dbws-design.rst +++ b/design/dbws-design.rst @@ -68,7 +68,9 @@ If invalid request, return appropriate http error code. EventSourceHandler ------------------ -**get:** Using the uid of the header, obtains an event generator. +**get:** Using the uid of the descriptor(s), obtains an event generator. By paginating on this generator, returns the payload to the client side. In the future, this bit will sort out the source of the data fill source -and redirect to the appropriate handler. +and redirect to the appropriate handler. In this context, handler refers to +the eventsource provided specific choice of technology given the data structure +and rate.