doc: update The_Configuration_File.md

README.md

@@ -8,6 +8,7 @@ workflows (interdependent steps/programs using a lot of data, with many
 parameter variations, running on different *hosts* with various schedulers).
 - an engine taking care of orchestrating the run of those workflows, 
 and keeping track everything that succeeds, fails, or gets lost.
+Ketrew can be a standalone application, or use a client-server architecture.
 
 This documentation is available at <http://hammerlab.github.io/ketrew/>,
 the source is at <https://github.com/hammerlab/ketrew/>.
@@ -72,10 +73,11 @@ See the file [`src/test/cli.ml`](src/test/cli.ml) for more examples (*work-in-pr
 and the [documentation of the EDSL API](src/lib/ketrew_edsl.mli).
 
 
-### Ketrew
+### Standalone Ketrew
 
-Ketrew is driven from the command-line client.  See first:
-`ketrew --help`; then:
+The default for Ketrew is to run in “Standalone” mode.
+From the command-line client, one can both query and run the engine.  See
+first: `ketrew --help`; then:
 
 - To display the current status: `ketrew status`.
 - To run as many steps as possible until a “fix-point” is reached:
@@ -91,8 +93,14 @@ See also `ketrew interact` or `ketrew explore` for fun
 Ketrew can try to be clever about killing an archiving;
 see `ketrew autoclean --help`.
 
-There is an HTTP API (work in progress); with the commands
-`ketrew start-server` and `ketrew stop-server`.
+### Client-Server Mode
+
+In this mode, the Ketrew engine runs a proper server which is
+accessed over an HTTP API.
+
+See the commands `ketrew start-server` and `ketrew stop-server`.
+
+The client works in the same way as in “Standalone” mode.
 
 Where to Go Next
 ----------------
@@ -113,12 +121,12 @@ like [`Ketrew_lsf`](src/lib/ketrew_lsf.mli) or in the tests:
 - You may want to extend Ketrew, or preconfigure it, *without* configuration
 files or dynamically loaded libraries: just
 [create](src/doc/Alternative_CLI_Application.md) your own comand-line app.
-- You may want to try out the [HTTP API](src/doc/The_HTTP_API.md)
-(*work in progress*).
+- You may want to call out directly yo the [HTTP API](src/doc/The_HTTP_API.md)
+(i.e. without the client).
 - If you want to help or simply to understand Ketrew
 see the [development](src/doc/Developer_Documentation.md)
 documentation, and have a look at the modules
-like [`Ketrew_state`](src/lib/ketrew_state.mli).
+like [`Ketrew_engine`](src/lib/ketrew_engine.mli).
 
 License
 -------

src/doc/The_Configuration_File.md

@@ -11,6 +11,25 @@ Top-level options:
 
 - `debug-level`: integer specifying the amount of verbose messages: `0`: none,
 `1`: verbose, `2`: very verbose.
+
+Then the configuration is divided into “sections”:
+`engine`, `client`, `ui`, and `server`.
+The presence of the `client` or `server` sections, defines the running mode
+(if none of both are defined, Ketrew will be in standalone mode, if both are
+defined this will be an error).
+
+
+### The `engine` Section
+
+In standalone and server modes, the `engine` section configures how to run the
+workflows (ignored in `client` mode).
+
+- `database-path`: the path to the database directory; it's a
+currently a Git-repository that one can inspect with usual `git` commands (the
+default is `~/.ketrew/database/`).
+- `state-key`: the name of the key to the “root of the tree”; if more than one
+application are using the same database, this can be useful to avoid conflicts
+(“normal” users should *never* need to set this).
 - `turn-unix-ssh-failure-into-target-failure`: boolean;
 when an SSH or system call fails it may not mean that the command in your
 workflow is wrong (could be an SSH configuration or tunneling problem), by
@@ -19,89 +38,103 @@ fail. To change this behavior put the option to `true`.
 - `host-timeout-upper-bound`: float (seconds, default: `60.`); every
 connection/command time-out will be `≤ upper-bound`.
 
-### The `client` Section
+### The `ui` Section
 
-This optional table configures the client-side/command-line application: 
+This section configures the behavior of the User Interface.
 
 - `color`: boolean (default `true`); tell Ketrew to display *f-ANSI* colors.
 
-### The `database` Section
+### The `plugins` Section
+
+This optional section asks Ketrew to dynamically load plugins:
 
-Configure Ketrew's persistence (mandatory section):
+- `ocamlfind`: a package name or a list of package names to find with `Findlib`
+and to load.
+- `compiled`: a path or a list of paths to load.
 
-- `path`: the path to the database file; it's a
-[Dbmᵂ](http://en.wikipedia.org/wiki/Dbm) database for now, this parameter is
-mandatory.
-- `state-key`: the name of the key to the “root of the tree”; if more than one
-application are using the same database, this can be useful to avoid conflicts
-(“normal” users should need to set this).
+### The `client` Section
+
+This table configures Ketrew in client-mode: 
+
+- `connection`: URL to connect to server (e.g. `"https://example.com:8443"`).
+- `token`: API authentication token.
 
 ### The `server` Section
 
-This optional section configures the HTTP server:
+This section configures the HTTP server:
 
 - `certificate`: path to the SSL certificate (*mandatory*).
 - `private-key`: path to the SSL private-key (*mandatory*).
 - `port`: port to listen on (*mandatory*).
-- `authorized-tokens-path`: path to the file giving in an SSH `authorized_keys`
-style, to authentication tokens that the server accepts (*mandatory-ish* since
-authentication-less server-side is not implemented so far).
+- `authorized-tokens-path`: path to the file giving the authentication tokens
+that the server accepts (*mandatory-ish* since
+authentication-less server-side is not implemented so far). The format of the
+file is the SSH `authorized_keys` style.
 - `command-pipe-path`: if set this asks the server to listen on a named pipe
-for control commands (*recommended*).
+for control commands (*highly recommended*).
 - `daemonize`: if `true`, ask the server to detach from current terminal.
-- `log-path`: if set, ask the server to redirect logs to this path (if not set
-logs go to `/dev/null`).
+- `log-path`: if set together with `daemonize`, ask the server to redirect logs
+to this path (if not set logs go to `/dev/null`).
 - `return-error-messages`: if `true`, the server will return real error
 messages (in the *body* of the response) to the client; if `false` (the
 default), any kind of error will result in the same uninformative message.
 
-### The `plugins` Section
 
-This optional section asks Ketrew to dynamically load plugins:
-
-- `ocamlfind`: a package name or a list of package names to find with `Findlib`
-and to load.
-- `compiled`: a path or a list of paths to load.
+Examples
+--------
 
+Note the `please.sh` script can generate configuration files for testing
+(`please.sh test-env`).
 
-Example
--------
-
-Here is a somewhat complete example:
+### Standalone
 
 ```toml
 # How much noise do you want on your terminal:
 debug-level = 2
 
-turn-unix-ssh-failure-into-target-failure = false
+[plugins]
+  ocamlfind =["lwt.unix", "findlib"]
+  compiled = "/path/to/some_plugin.cmxs"
 
-# Unless explicitly told, all SSH/system calls are bounded by a timeout.
-host-timeout-upper-bound = 120
+[engine]
+  database-path = "/path/to/database-standalone"
+  host-timeout-upper-bound = 120
 
-[client]
-  # One can remove the colors in the output of the client with:
-  color = false
+[ui]
+  color = true
+```
 
+### Server
 
-[database]
-  # The path to the database (mandatory):
-  path = "/path/to/some/file"
-  # Normal users should not change that:
-  state-key = "some-string"
+```toml
+debug-level = 1
+
+[engine]
+  database-path = "/path/to/database-client-server"
+  turn-unix-ssh-failure-into-target-failure = false
+
+[ui]
+  color = true
 
 [server]
-  certificate = "/etc/ketrew/cert.pem"
-  private-key = "/etc/ketrew/key.pem"
   port = 8443
-  authorized-tokens-path = "/etc/ketrew/authorized-tokens"
+  certificate = "/path/to/test-cert.pem"
+  private-key = "/path/to/test-key.pem"
+  authorized-tokens-path = "/path/to/ketrew-authorized-tokens"
   return-error-messages = true
-  log-path = "/var/log/ketrew/server.log"
+  log-path = "/var/log/logs-of-ketrew-server.txt"
   daemonize = true
-  command-pipe-path = "/var/run/ketrew/command.pipe"
+  command-pipe-path = "/var/run/ketrew-command.pipe"
+```
 
-[plugins]
-  ocamlfind =["lwt.unix", "lwt.react"]
-  compiled = "_build/some_plugin/some_plugin.cmxs"
+### Client
+
+```toml
+debug-level = 2
+
+[client]
+  connection = "https://example.com:8443"
+  token = "aldskfjdalksjfiilfldksaj"
 ```
 
 Print The Configuration