diff --git a/design/dbws-design.rst b/design/dbws-design.rst new file mode 100644 index 0000000..a2b1f76 --- /dev/null +++ b/design/dbws-design.rst @@ -0,0 +1,76 @@ +================================= +Data Broker Service Layer Design +================================= + +Motivation +========== + +Databroker has been actively developed and widely used over the past 2 years. +It has become more powerful and stable thanks to Dan Allan's efforts over the +past year. Its stability and wide acceptance as the de facto way access data +acquired in our beamlines. A new wave of requirements are reported as a result +of our community outreach. One of these requirements is a set of databroker +server implementations that will allow remote access to experimental data over +network. The service layer will also eliminate any issues that may occur as a +result of providing direct/library-level access to databases, allowing +us to implement data access policies. + +Challenges +=========== + +NSLS-II and other facilities have a wide variety of experimental setups that +generate data at different rates and structure. This poses a major challenge +for a single-backend implementation. Another challenge is the lack of sys admin +and dev-ops support. Making performant and reliable asynchronous servers with +low-latency is that are maintainable in the long term that are easy to deploy +is especially difficult with the amount of resources at hand. + + +Roadmap +=========== + +- Identify the data rates required to be transferred in NSLS-II databases. This +does not require a major effort, a simple investigation of existing metadatastore +and filestore will suffice + +- Identify the technologies to be used for transmitting ND arrays over the wire: + msgpack, protobuffe, h5serv, etc. This includes performance measurements + +- Have a first cut on existing databroker using .. _ujson: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-json-libraries/ + +- Prioritize the backends to be implemented, define timelines, and implement. + +- Add an authentication layer + + +Technology +============= + +Given there are plans for central storage and data management, the implementation +of the server layer must be scalable. This means a micro-service oriented +infrastructure that can eventually be placed behind load balancers is a good +intuitive choice. A fully RESTful approach is achiveable yet limiting given the +flexibility provided by the stack. This layer needs some discussion. We could +definitely adopt a RESTful approach (ugh, dreaded approach word) could be +adopted rather than handler per document type sort of approach of metadataservice +and amostra. + +Server Side Design +=================== + +HeaderHandler +------------ + +**get:** Decode jsonified query and return a serialize a list of dicts +If invalid request, return appropriate http error code. + + +EventSourceHandler +------------------ + +**get:** Using the uid of the descriptor(s), obtains an event generator. +By paginating on this generator, returns the payload to the client side. +In the future, this bit will sort out the source of the data fill source +and redirect to the appropriate handler. In this context, handler refers to +the eventsource provided specific choice of technology given the data structure +and rate.