Merge pull request #12 from arrdem/rolling

Generalized rolling

README.md

@@ -30,6 +30,11 @@ Then from within a git repo, run
 
 ```
 $ kat start-server
+Starting server ...
+Waiting for it to become responsive \
+Started server!
+  http port: 3636
+  nrepl port: 3637
 ```
 
 This will prompt Katamari to self-bootstrap, downloading the latest server standalone jar and creating a couple files in the root of your repository.
@@ -75,17 +80,19 @@ In Katamari, this dependency is explicit.
 One could write the following rollfile -
 
 ```clj
+(deftarget my/java-library
+  (java-library
+   :paths ["src/main/jvm"]))
+
 (deftarget my/simple-app
   (clojure-library
    :paths ["src/clj/main"]
    :deps {my/java-library nil
           org.clojure/clojure nil}))
-
-(deftarget my/java-library
-  (java-library
-   :paths ["src/main/jvm"]))
 ```
 
+The targets don't have to be dependency ordered, but it's convenient to write them as such.
+
 Katamari, like Leiningen, will compile your Java sources before doing anything with your Clojure target.
 However this dependency tree can go as deep as you would like it to.
 Your Java target could in turn depend on another Java target, or a Kotlin target, or what have you.
@@ -97,7 +104,77 @@ It's already "up to date".
 
 ### Tasks
 
-Katamari's tasks are extremely a work in progress, so more documentation here will have to come later.
+The Katamari CLI client implements several tasks - operations implemented either on the server side or as a hybrid of server and client actions.
+We can see what tasks Katamari is aware of by issuing the `list-tasks` (builtin) command.
+
+```
+$ ./kat list-tasks
+Commands:
+  compile
+  help
+  list-targets
+  list-tasks
+  meta
+  restart-server
+  show-request
+  start-server
+  stop-server
+```
+
+The most significant of these are probably `show-request`, which allows you to inspect both your build graph and configuration, `list-targets` which gives you a quick way to see what targets are visible in the project and `compile` which is used to compile targets and their dependencies.
+
+For instance in Katmari's own repository, the `example` tree is used to define examples of all of Katamari's available targets.
+That tree has a couple targets - `example/javac`, `example/clj`, `example/clj+jar` and `example/clj+uberjar`.
+
+Lets try building `example/clj+uberjar` - 
+
+```
+$ ./kat compile example/clj+uberjar
+{
+  "example/javac": {
+    "type": "katamari.roll.extensions.jvm/product",
+    "from": "example/javac",
+    "mvn/manifest": "roll",
+    "deps": null,
+    "paths": [
+      "/private/var/folders/z4/6b9f3h4x2dv6gvxbwyc55cwnwhsd5r/T/javac6828446538002526748"
+    ]
+  },
+  "example/clj": {
+    "type": "katamari.roll.extensions.clj/product",
+    "from": "example/clj",
+    "mvn/manifest": "roll",
+    "paths": [
+      "/Users/arrdem/katamari/example/src/main/clj"
+    ],
+    "deps": {
+      "example/javac": null,
+      "org.clojure/clojure": null
+    }
+  },
+  "example/clj+uberjar": {
+    "type": "katamari.roll.extensions.jar/product",
+    "from": "example/clj+uberjar",
+    "mvn/manifest": "roll",
+    "paths": [
+      "/Users/arrdem/katamari/target/clj-standalone.jar"
+    ],
+    "deps": {
+      "example/javac": null,
+      "example/clj": null
+    }
+  },
+  "intent": "json"
+}
+```
+
+This is a build product - or more specifically it's all the build products for all the artifacts in the compilation graph of `example/clj+uberjar`.
+`clj+uberjar` depends on the `clj` target, which depends on `javac`.
+When compiling `clj+uber`, first `javac` is compiled to produce a directory of `.class` files, visible in the `"paths"` array of the `javac` result.
+That result is consumed when producing the `clj` result, and all of those paths (and the depended jars) get packed into the file `target/clj-standalone.jar` if you care to unzip it and look.
+
+Note that unlike Leiningen and other tools, Katamari makes no attempt to automate generating manifests - at least not yet.
+Jar manifests are dependencies which must be built and included when packing a jar, just like any other dependency.
 
 ## Documentation
 

clojure-tools/Rollfile

@@ -8,4 +8,3 @@
    :deps
    {org.clojure/clojure nil
     org.clojure/tools.deps.alpha nil}))
-

docs/dictionary.md

@@ -0,0 +1,82 @@
+# Dictionary
+[⛓ README](/README.md)
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Dictionary](#dictionary)
+    - [Target](#target)
+    - [Rule manifests](#rule-manifests)
+    - [Build rule](#build-rule)
+    - [Build products](#build-products)
+    - [Rule inputs](#rule-inputs)
+    - [Rollfile](#rollfile)
+    - [Build graph](#build-graph)
+
+<!-- markdown-toc end -->
+
+## Target
+
+A target is a name, which may be associated with a [build rule](#build-rule) within a [build graph](#build-graph).
+
+For instance `me.arrdem/katamari` or `org.clojure/clojure` would be valid targets.
+
+## Rule manifests
+
+The interpretation of a [build rules](#build-rule) is defined by its `manifest`.
+The manifest itself does nothing, it simply serves to define dispatching constant used when handing a rule.
+
+## Build rule
+
+A build rule is a list of a symbol.
+The symbol is called the [rule manifest](#rule-manifest), and names the machinery which should be used to interpret the rule.
+The rest of a rule is `keys*` arguments, which define the manifest's behavior.
+
+For instance -
+
+```clj
+(clojure-library :deps {org.clojure/clojure nil} :paths ["src"])
+```
+
+would be a rule with the `clojure-library` manifest, parameterized with the `:paths` and `:deps`.
+The symbol `clojure-library` would be the dispatch constant used by the roll API when handling the rule.
+
+Rules are introduced by defining manifest types, and their parsing to rules.
+
+## Build products
+
+When building takes place, each build rule produces a product.
+The product value serves to explain whatever was built, and allow downstream targets which depend on that product to consume it.
+Typically this happens via [rule inputs](#rule-inputs), but products may also be transitively depended on.
+
+For instance when building an Uberjar, all dependencies and other built products are transitively depended on.
+
+## Rule inputs
+
+Rules may list other rules as build inputs, and use the `rule-inputs` method to enumerate their dependencies.
+When builds are executed, building takes place with both the inputs to the task, and all the existing build products.
+This allows build steps to refer to all other built products, as well as to the products which they directly depend on.
+
+For instance, the above `clojure-library` form would have no inputs - its only dependency is a Maven packaged artifact.
+
+## Rollfile
+
+Katamari uses files named `Rollfile` to define the [build graph](#buildgraph).
+A Katamari project may consist of many rollfiles.
+Each rollfile may define zero or more targets.
+
+At present, all targets have global scope and can be referred to from any rollfile.
+
+## Build graph
+
+Katamari maintains a build graph.
+The build graph maps [targets](#target) as defined by `deftarget` forms to the [build rule](#build-rule) which describes how to produce that target.
+
+For instance if there was only one `Rollfile` -
+
+```clj
+(deftarget demo/demo
+  (clojure-library :deps {} :paths ["src"]))
+```
+
+then the build graph would be a map of `demo/demo` to a [build rule](#build-rule) with the `clojure-library` [manifest](#manifest).

docs/manifesto.md

@@ -30,14 +30,14 @@ The tacit official recomendation for deploying `deps.edn` seems to be to `Makefi
 `deps.edn` is no help for these tasks.
 
 Katamari's design is less aescedic in its minimalism.
-The reality is that users' workflows between the REPL, running stand-alone tests and producing artifacts are related and benefit from integration as they depend fundamentally on the same dependency data.
+The reality is that users' workflows between the REPL, running stand-alone tests and producing artifacts are related and benefit from integration as they depend fundimentally on the same dependency data.
 Maintaining simplicity isn't easy when trying to solve these problems, but solutions are more generally useful and impose less upon their users to pick up the pieces.
 
 Also `deps` just leaves a lot on the table in terms of its caching strategy, and eschews the entire problem of multi-language builds.
 
 ## In contrast to Leiningen
 
-Lein does what I want really nicely for small projects, but its lack of caching and fundamental boot jvm, process, kill JVM architecture puts a 3-5s lower bound on any given lein operation.
+Lein does what I want really nicely for small projects, but its lack of caching and fundimental boot jvm, process, kill JVM architecture puts a 3-5s lower bound on any given lein operation.
 Checkouts don't really save you in a multi-module configuration because they still assume that you have artifacts of the same name installed and resolvable via Maven.
 This makes fully anonymous / source builds at best difficult if not impossible.
 
@@ -47,11 +47,11 @@ Boot is designed to solve the related problem of incremental file rebuilds also
 Unfortunately, Boot falls squarely into the single module at a time trap and there doesn't seem to be any recognizable prior art for multi-module boot setups.
 Multi-module is a hard design goal.
 
-## In contrast to Gradle
+## In contrast to Gradel
 
-Gradle is a more mature tool which supports programmable builds and has many existing extensions for languages including Clojure and Java.
+Gradel is a more mature tool which supports programmable builds and has many existing extensions for languages including Clojure and Java.
 If you're comfortable with Groovy, it's probably the best way to go here.
 Katamari was built in large part out of what Alex Miller characterized as a "fear of XML" and in this case of Groovy.
 
-I'm a Clojure developer, working on a Clojure team none of whom have any experience with the Gradle/Groovy toolchain.
-The barrier to entry for hacking on it or adapting it to our needs is fairly vertical, whereas the core algorithms and caching strategies behind build systems are fairly well understood and not hard to implement.
+I'm a Clojure developer, working on a Clojure team none of whom have any experience with the Grade/Groovy toolchain.
+The barrier to enntry for hacking on it or adapting it to our needs is fairly vertical, whereas the core algorithms and caching strategies behind build systems are fairly well understood and not hard to impleent.

docs/system-architecture.md

@@ -12,12 +12,101 @@ Furthermore in order to take advantage of graalvm or other Java optimizing tools
 This makes boot more tractable, but boot was designed and optimized for solving the problem of updating built resources such as `less` files and `cljs` to be served in a browser.
 Because in the context of a webapp all these resources must converge in the application, boot is not designed for a concept of targets or to support "partial" builds.
 
+## Kat client
+
 Katamari is designed as a small shell script - `kat`, which interacts with a persistent JVM server.
 The shell script serves as a shim, which starts a server instance if there isn't an active one, and then uses the server to perform any required actions.
 
+### Tasks
+
 Beyond this, Katamari's architecture is currently somewhat murky.
 As the Katamari server is intended to be durable, while it embeds a REPL (including CIDER), it's not really an appropriate vehicle from which to execute build steps.
 These would seem more naturally and more interactively placed close to or even in the user's shell, rather than having the remote server opaquely and non-interactively execute many commends.
 
 There would also seem to be cause to enable users to write their own shell scripts which interact with the Katamari server.
 For instance one could imagine a `kat status` task, implemented as a shell script which uses the normal `kat` machinery to inspect the server, and can avoid the default server starting behavior.
+
+### Intents
+
+At present, the Katamari client integrates with the server using an intents mechanism.
+The user may request an output format from the client - one of the following:
+
+- `-j` or `--json`, force JSON pretty printing based on `jq`
+- `-r` or `--raw`, directly display the response
+- `-m` or `--message`, format the `.message` of the response
+
+Unless the user specifically requests a response handling, the server can control it.
+This is done by responding with a JSON body, containing the `.intent` key.
+The intents `raw`, `json`, `msg`, `message`, `sh`, `subshell` and `exec` are supported.
+
+The `sh` and `subshell` intents will cause the `kat` client to execute the `.sh` response field as a BASH script.
+The `exec` intent will cause the client to well `exec` the `.exec` response field.
+These allow the server to express intents such as a remote restart, running a CLI REPL client or a test suite which are at best awkward from a fully remote server.
+
+## Katamari server
+
+The Katamari server is a full Ring application running embedded nREPL + CIDER for debugging and development support.
+The server provides only two routes at present - `/api/v0/ping` used only to establish the server's liveness, and `/api/v0/request` which is used to process requests from the `kat` script.
+
+When the user enters a command - say `./kat start-server` which shows server port information - this becomes essentially a
+
+```
+$ curl -XPOST localhost:3636/api/v0/ping --data @- <<EOF
+{"repo_root": "...",
+ "user_root": "...",
+ "config_file": "...",
+ "cwd": "$PWD",
+ "request": [$@]}
+EOF
+```
+
+allowing for some differences of implementation to correctly escape things.
+
+### Middlewares
+
+The Katamari server uses two middleware stacks to process requests.
+The first is the Ring side mdidleware stack, which behaves exactly as you'd expect.
+The second is the Katamari server extensions stack - used only to implement handling of tasks and wrapping of tasks on the server.
+These middleware stacks were separated from each other so that tasks could recurse and manipulate each other more finely.
+Whether that pans out is somewhat up in the air.
+
+Task middlewares come in two flavors `katamari.server.extensions/defhandler` and `defwrapper`.
+`handlers` are used to implement handing a single task - or passing along an unrecognized task request.
+`wrappers` don't handle tasks, but wrap up task handling with context.
+For instance the build graph it itself a `wrapper` around the handler which provides the `compile` task.
+
+The handler (and wrapper) contract is that they accept the `[handler, config, stack, request]` where the `handler` is really the next fn in the middleware stack, the `config` is the global configuration, the `stack` is the entire composed middleware as a fn in case you want to restart processing of a request, and the `request` is the `argv` as recieved from the client.
+
+The `show-request` task is pretty typical -
+
+`clj
+(defhandler show-request
+  "Show the request and config context as seen by the server (for debugging)"
+  [handler config stack request]
+  (->  {:intent :json
+        :config config
+        :request request}
+       (resp/response)
+       (resp/status 200)))
+```
+
+The advantage of these macros is that they tie into a dynamically re-computed middleware stack to better enable live development of handlers and wrappers without restarting the web server all the time.
+
+## Rolling
+
+Rolling - or more precisely the `katamari.roll.*` namespaces - are the core of Katamari's actual build machinery.
+The API to these namespaces is inspired heavily by `clojure.tools.deps(.alpha)`, and is intended to enable the separate development of additional targets.
+
+The roll API presents several multimethods for extension, visible in the `katamari.roll.extensions` namespace.
+The overall intent is that users leverage the `defmanifest` macro to create new [rule manifests](dictionary.md#rule-manifests) so that they can be parsed in Rollfiles by the reader.
+
+The rest of the API allows a manifest to define how it interacts with the build graph.
+Implementations of most methods are required for every manifest type.
+Only the `manifest-prep` and `rule-prep` methods are optional.
+
+The roll API leverages `clojure.tools.deps(.alpha)` to implement classpath building, by way of a custom `:roll` deps manifest type which allows for control over how deps behaves.
+
+## Build caching
+
+At present, Katamari only implements (correct!) static dependency order builds.
+Build product identification for minimal rebuilds is a work in progress.

example/Rollfile

@@ -0,0 +1,23 @@
+;; -*- mode: clojure; -*-
+
+(deftarget example/javac
+  (java-library
+   :paths ["src/main/java"]))
+
+(deftarget example/clj
+  (clojure-library
+   :paths ["src/main/clj"]
+   :deps {example/javac nil
+          org.clojure/clojure nil}))
+
+(deftarget example/clj+jar
+  (jar
+   :jar-name "clj.jar"
+   :deps {example/javac nil
+          example/clj nil}))
+
+(deftarget example/clj+uberjar
+  (uberjar
+   :jar-name "clj-standalone.jar"
+   :deps {example/javac nil
+          example/clj nil}))

example/src/main/clj/data_readers.edn

@@ -0,0 +1 @@
+{}

example/src/main/java/demo/Demo.java

@@ -0,0 +1,8 @@
+package demo;
+
+public class Demo {
+    public static int main(String[] args) {
+        System.out.println(String.format("Got %d args!", args.length()));
+        return 0;
+    }
+}