A first cut at reinstating sharing for the documentation (#901)

* A first cut. * fixing broken links * another broken link. * fix link * minor fixes * minor improvement. * a couple more fixes * Defining a variable that allows conditional text, oss. * Formatting fix. * pointing to the correct file. * This page wasn't showing in the output. * putting versions in a logical place * To fix a problem in the commercial docs. * Fixing an xref for the commercial doc. * Addressing review comments * removing unnecessary lines * fix the ci failure Co-authored-by: rstento <[email protected]>
lightbend · Nov 26, 2020 · 7aee74c · 7aee74c
1 parent d911eca
commit 7aee74c
Show file tree

Hide file tree

Showing 94 changed files with 902 additions and 1,979 deletions.
diff --git a/docs/docs-source/author-mode-site.yml b/docs/docs-source/author-mode-site.yml
@@ -24,6 +24,7 @@ asciidoc:
     review: ''
     todo: ''
     doc-title: 'Cloudflow Guide'
+    oss: ''
 output:
   dir: ./../target/staging
   clean: true
diff --git a/docs/docs-source/docs/modules/ROOT/nav.adoc b/docs/docs-source/docs/modules/ROOT/nav.adoc
@@ -11,14 +11,9 @@ include::partial$include.adoc[]
 *** xref:get-started:define-avro-schema.adoc[Define Avro schema]
 *** xref:get-started:develop-example-streamlets.adoc[Develop example streamlets]
 *** xref:get-started:create-example-blueprint.adoc[Create the example blueprint]
-** xref:get-started:run-in-sandbox.adoc[Run in a local sandbox]
-** xref:get-started:deploy-to-k8s-cluster.adoc[Deploy to a Kubernetes cluster]
-** xref:get-started:exercise-example.adoc[Exercise the example]
-
-* xref:run-cloudflow-gcp-marketplace:index.adoc[Running Cloudflow on GCP Marketplace]
-** xref:run-cloudflow-gcp-marketplace:find-cloudflow-on-gcp-marketplace.adoc[Finding Cloudflow on GCP Marketplace]
-** xref:run-cloudflow-gcp-marketplace:deploy-cloudflow.adoc[Deploying Cloudflow on GCP Marketplace]
-** xref:run-cloudflow-gcp-marketplace:run-sample-application.adoc[Running a Cloudflow application]
+*** xref:get-started:run-in-sandbox.adoc[Run in a local sandbox]
+*** xref:get-started:deploy-to-k8s-cluster.adoc[Deploy to a Kubernetes cluster]
+*** xref:get-started:exercise-example.adoc[Exercise the example]
 
 * xref:develop:index.adoc[Developing Applications with Cloudflow]
 ** xref:develop:project-structure.adoc[Project Structure]
@@ -54,23 +49,17 @@ include::partial$include.adoc[]
 ** xref:cli:cloudflow_update-docker-credentials.adoc[update-docker-credentials]
 ** xref:cli:cloudflow_version.adoc[version]
 
-* xref:administration:index.adoc[Cloudflow administration]
+* xref:administration:index.adoc[Installing and configuring components]
 ** xref:administration:installation-prerequisites.adoc[Installation Prerequisites]
+** xref:administration:versions.adoc[Supported component versions]
 ** xref:administration:installing-cloudflow.adoc[Installing Cloudflow]
 ** xref:administration:how-to-install-and-use-strimzi.adoc[Installing Kafka with Strimzi]
 ** xref:administration:installing-spark-operator.adoc[Adding Spark Support]
 ** xref:administration:installing-flink-operator.adoc[Adding Flink Support]
-** xref:administration:installing-enterprise.adoc[Installing Cloudflow Enterprise components]
-** xref:administration:upgrading-cloudflow.adoc[Upgrading Cloudflow]
 ** xref:administration:providing-external-access-to-cloudflow-services.adoc[Setting up external access]
 
-* xref:console:index.adoc[Observing Cloudflow applications]
-** xref:console:cloudflow-applications-page.adoc[Cloudflow applications page]
-** xref:console:application-monitoring-overview.adoc[Application monitoring overview]
-** xref:console:blueprint-graph.adoc[Blueprint Graph]
-** xref:console:application-monitoring-page.adoc[Application monitoring page elements]
-
-* xref:project-info:index.adoc[Project Information]
+* xref:project-info:index.adoc[Release notes and upgrading]
 ** xref:project-info:migration-1_3-2_0.adoc[Migration Guide 1.3.x to 2.0.x]
+** xref:administration:upgrading-cloudflow.adoc[Upgrading Cloudflow]
 
 * xref:api:index.adoc[Streamlet API Reference]
diff --git a/docs/docs-source/docs/modules/ROOT/pages/index.adoc b/docs/docs-source/docs/modules/ROOT/pages/index.adoc
@@ -50,7 +50,7 @@ image::apps-1.png[]
 
 == Feature set
 
-Cloudflow includes tools for developing and deploying streaming applications:
+Cloudflow dramatically accelerates your application development efforts, reducing the time required to create, package, and deploy an application from weeks--to hours. Cloudflow includes tools for developing and deploying streaming applications:
 
 * The Cloudflow application development toolkit includes:
 ** An SDK definition for `Streamlet`, the core abstraction in Cloudflow.
@@ -62,8 +62,11 @@ Cloudflow includes tools for developing and deploying streaming applications:
 
 * For deploying to Kubernetes clusters, the Cloudflow operator takes care of orchestrating the deployment of the different parts of a streaming pipeline as an end-to-end application.
 
-Cloudflow dramatically accelerates your application development efforts, reducing the time required to create, package, and deploy an application from weeks--to hours. 
-Lightbend also offers support with a https://www.lightbend.com/lightbend-platform-subscription[Lightbend Subscription].
+ifdef::todo[TODO: add the following back in once the commercial versions are ready. Text might need editing.]
+
+// Lightbend's commercial version of Cloudflow, https://developer.lightbend.com/docs/cloudflow/current/index.html[Akka Data Pipelines], includes support along with https://developer.lightbend.com/docs/telemetry/current/home.html[Telemetry] and https://developer.lightbend.com/docs/cloudflow/current/console/index.html[Lightbend Console], which provide visibility into your running Cloudflow apps.
+
+// You can also use https://developer.lightbend.com/docs/cloudflow/current/gcp/index.html[Akka Data Pipelines on Google Cloud Platform]. 
 
 == What's next
 

diff --git a/docs/docs-source/docs/modules/ROOT/partials/include.adoc b/docs/docs-source/docs/modules/ROOT/partials/include.adoc
@@ -1,16 +1,3 @@
-:page-partial:
-// global attributes
 
-:kubernetes-platform-name:     Kubernetes
-:kubernetes-platform-version:  1.12
-:kubectl-plugin-version:       KUBECTL_PLUGIN_VERSION
-
-:cli:                          kubectl
-:cli-plugin:                   kubectl
-
-:latest-openshift-platform-version:        3.11
-:minimum-kubernetes-on-openshift-version:  1.11
-:minimum-openshift-platform-version:       3.11
-:minimum-platform-version-major-dot-minor: 3.11
-:minimum-sbt-version:                      0.13.7
-:latest-sbt-version: 					   1.4.4	 
+// global attributes, if you update, make sure to update in shared-content-source ROOT:partials:include.adoc too
+// see the include file in shared-content:ROOT:partials:include.adoc
diff --git a/...cs-source/docs/modules/administration/pages/how-to-install-and-use-strimzi.adoc b/...cs-source/docs/modules/administration/pages/how-to-install-and-use-strimzi.adoc
@@ -4,136 +4,5 @@
 :toclevels: 2
 
 include::ROOT:partial$include.adoc[]
+include::{cloudflow-version}@shared-content:administration:page$how-to-install-and-use-strimzi.adoc[]
 
-This guide shows how to install Strimzi and create a Kafka cluster suitable for use with Cloudflow.
-
-Strimzi is a Kubernetes operator that simplifies the process of running Apache Kafka.
-
-== Prerequisites
-
-=== CLI Tools
-
-Make sure you have the following prerequisites installed before continuing:
-
-- Helm, version 3 or later
-- Kubectl 
-
-== Installing Strimzi
-
-In this guide, we will use Helm to install Strimzi.
-
-We are going to create Strimzi in the `cloudflow` namespace in this guide. Make sure that the namespace `cloudflow` exists before continuing. To create the `cloudflow` namespace, execute
-
-  kubectl create ns cloudflow
-
-Add the Strimzi Helm repository and update the local index.
-
-  helm repo add strimzi https://strimzi.io/charts/
-  helm repo update
-
-Install the latest version of Strimzi.
-
-  helm install strimzi strimzi/strimzi-kafka-operator --namespace cloudflow
-
-After the install complete, the Strimzi Kafka operator should be running in the `cloudflow` namespace.
-
-----
-$ kubectl get pods -n cloudflow
-NAME                                       READY   STATUS    RESTARTS   AGE
-strimzi-cluster-operator-9968fd8c9-fhqmj   1/1     Running   0          17s
-----
-
-=== Creating a Kafka cluster using Strimzi
-
-To create a Kafka cluster using Strimzi, we have to create a `CustomResource` of the kind `Kafka` in the namespace `cloudflow`. 
-
-NOTE: The Kafka cluster configuration shown here is meant for development and testing purposes only. Please consider your storage requirements and modify the custom resource accordingly for your intended usage.
-
-You can use the following command to create a cluster:
-
-[source,shell script]
-----
-kubectl apply -f - <<EOF
-apiVersion: kafka.strimzi.io/v1beta1
-kind: Kafka
-metadata:
-  name: cloudflow-strimzi
-  namespace: cloudflow
-spec:
-  kafka:
-    config:
-      auto.create.topics.enable: false
-      log.message.format.version: "2.3"
-      log.retention.bytes: 1073741824
-      log.retention.hours: 1
-      log.retention.check.interval.ms: 300000
-      offsets.topic.replication.factor: 3
-      transaction.state.log.min.isr: 2
-      transaction.state.log.replication.factor: 3
-    listeners:
-      plain: {}
-      tls: {}
-    livenessProbe:
-      initialDelaySeconds: 15
-      timeoutSeconds: 5
-    readinessProbe:
-      initialDelaySeconds: 15
-      timeoutSeconds: 5
-    replicas: 3
-    resources: {}
-    storage:
-      deleteClaim: false
-      size: 100Gi
-      type: persistent-claim
-    version: 2.5.1
-  zookeeper:
-    livenessProbe:
-      initialDelaySeconds: 15
-      timeoutSeconds: 5
-    readinessProbe:
-      initialDelaySeconds: 15
-      timeoutSeconds: 5
-    replicas: 3
-    resources: {}
-    storage:
-      deleteClaim: false
-      size: 10Gi
-      type: persistent-claim
-EOF
-----
-
-When the Strimzi Kafka operator has processed the custom resource, there should be six new pods in the `cloudflow` namespace. Three Kafka broker pods and three Zookeeper pods.
-
-----
-$ kubectl get pods -n cloudflow
-NAME                                       READY   STATUS    RESTARTS   AGE
-cloudflow-strimzi-kafka-0                  2/2     Running   0          1m
-cloudflow-strimzi-kafka-1                  2/2     Running   0          1m
-cloudflow-strimzi-kafka-2                  2/2     Running   0          1m
-cloudflow-strimzi-zookeeper-0              1/1     Running   0          2m
-cloudflow-strimzi-zookeeper-1              1/1     Running   0          2m
-cloudflow-strimzi-zookeeper-2              1/1     Running   0          2m
-strimzi-cluster-operator-9968fd8c9-fhqmj   1/1     Running   0          5m
-----
-
-TIP: If you want to change any parameters of the Kafka cluster, edit the `kafka-cluster.yaml` file and apply it again to the cluster, as shown above. The Strimzi Kafka cluster operator will make the necessary changes when detecting an update of the custom resource.
-
-The most important parameters here are:
-
-* Log retention policy - see https://www.cloudkarafka.com/blog/2018-05-08-what-is-kafka-retention-period.html#:~:text=If%20the%20log%20retention%20is,data%20is%20not%20a%20problem[this article] and
-https://medium.com/@sunny_81705/kafka-log-retention-and-cleanup-policies-c8d9cb7e09f8#:~:text=Apache%20Kafka%20provides%20two%20types,for%20Segments%20is%207%20days[this one]
-for log retention explanation. In our example we are using 1 hour retention, which is good enough for testing, with retention time check at 300000 ms.
-Alternatively you can use `log.retention.bytes` - a size-based retention policy (in bytes).
-* Kafka storage size - you need to make sure that this value is greater than the max log retention size in bytes,
-or calculated log size based on the log retention time policy and anticipated message rate.
-
-
-=== Removing Strimzi 
-
-If you want to remove Strimzi, you can do that with the following Helm command:
-
-  helm delete strimzi -n cloudflow
-
-You can find all Helm releases installed in the `cloudflow` namespace by executing the following command:
-
-  helm ls -n cloudflow
diff --git a/docs/docs-source/docs/modules/administration/pages/index.adoc b/docs/docs-source/docs/modules/administration/pages/index.adoc
@@ -1,23 +1,22 @@
-= Administration tasks
+= Installing and configuring components
 :toc:
 :toc-title: ON THIS PAGE
 :toclevels: 2
 
 include::ROOT:partial$include.adoc[]
 
-This chapter describes common Cloudflow and Kubernetes administration tasks.
+When you develop your Cloudflow application, you have the option to use Akka Streams, Spark, Flink and Kafka. This chapter contains the steps to install Cloudflow, using the `Helm` and `kubectl` command-line tools. It also shows how to install Kafka, Spark, and Flink operators that integrate with Cloudflow and how to expose your services to external access.
 
-It contains the steps to install Cloudflow, using the `Helm` and `kubectl` command-line tools.
-It also shows how to install Kafka, Spark, and Flink operators that integrate with Cloudflow.
 
-- Installation
-. xref:installation-prerequisites.adoc[]
-. xref:installing-cloudflow.adoc[]
-. xref:how-to-install-and-use-strimzi.adoc[]
-. xref:installing-spark-operator.adoc[]
-. xref:installing-flink-operator.adoc[]
-. xref:installing-enterprise.adoc[]
+* xref:installation-prerequisites.adoc[]
+* xref:versions.adoc[]
+* xref:installing-cloudflow.adoc[]
+* xref:how-to-install-and-use-strimzi.adoc[]
+* xref:installing-spark-operator.adoc[]
+* xref:installing-flink-operator.adoc[]
+* xref:providing-external-access-to-cloudflow-services.adoc[]
+
+For information on upgrading Cloudflow and migrating applications, see:
+* xref:upgrading-cloudflow.adoc[]
+* xref:project-info:index.adoc[]
 
-- Other administration
-** xref:upgrading-cloudflow.adoc[]
-** xref:providing-external-access-to-cloudflow-services.adoc[]
diff --git a/docs/docs-source/docs/modules/administration/pages/installation-prerequisites.adoc b/docs/docs-source/docs/modules/administration/pages/installation-prerequisites.adoc
@@ -4,81 +4,4 @@
 :toclevels: 2
 
 include::ROOT:partial$include.adoc[]
-
-This guide shows how to prepare your environment to install Cloudflow, using the `Helm` and `kubectl` command-line tools.
-It also shows how to install Kafka, Spark, and Flink operators that integrate with Cloudflow.
-
-== Prerequisites
-
-=== CLI Tools
-
-Make sure you have the following prerequisites installed before continuing:
-
-- Helm, version 3 or later (check with `helm version`)
-- Kubectl
-
-Before proceeding, make sure that `kubectl` is correctly installed and access the Kubernetes cluster to install Cloudflow. This can be done with the following command:
-
-    kubectl version
-
-=== Kafka
-
-Kafka is used by Cloudflow to connect streamlets together in a blueprint.
-If you intend to connect streamlets in this way then at least one Kafka cluster should be available before installation.
-Cloudflow may be used without Kafka (for example, when your application contains a single streamlet, or an xref:develop:clustering-akka-streamlet.adoc[Akka cluster]), but if your team intends to connect streamlets together and not include Kafka connection information in each topic they define then it's recommended to define a `default` Kafka cluster at install time.
-
-The Kafka broker bootstrap configuration string is a comma-separated list of host/port pairs used by Cloudflow to establish the connection to a Kafka cluster.
-The configuration string should have the following format:
-
-    broker-1-address:broker-1-port, broker-2-address;broker-2-port
-
-If you want to test Cloudflow and need a Kafka cluster, we recommend using Strimzi, a third-party Kafka operator that can create and manage Kafka clusters.
-
-See xref:how-to-install-and-use-strimzi.adoc[Installing Kafka with Strimzi] as a guide on how to configure and install a Kafka cluster using Strimzi.
-
-=== Storage requirements (for use with Spark or Flink)
-
-**If you plan to write Cloudflow applications using Spark or Flink**, the Kubernetes cluster will need to have a storage class of the `ReadWriteMany` type installed.
-
-For testing purposes, we suggest using the NFS Server Provisioner, which can be found here: https://github.com/helm/charts/tree/master/stable/nfs-server-provisioner[NFS Server Provisioner Helm chart]
-
-We'll install the nfs chart in the `cloudflow` namespace, if it does not exist yet, create the `cloudflow` namespace:
-
-  kubectl create ns cloudflow
-
-
-Add the `Stable` Helm repository and update the local index:
-
-  helm repo add stable https://charts.helm.sh/stable
-  helm repo update
-
-Install the NFS Server Provisioner using the following command:
-
-IMPORTANT: Depending on your Kubernetes configuration, you may want to adjust the values used during the install.
-Please see https://github.com/helm/charts/tree/master/stable/nfs-server-provisioner#configuration[NFS Server Provisioner configuration options].
-
-  helm install nfs-server-provisioner stable/nfs-server-provisioner \
-    --namespace cloudflow
-
-The result of the installation is shown below, the NFS Server provisioner pod is running and the new storage class exists.
-
-----
-$ kubectl get pods -n cloudflow
-NAME                       READY   STATUS    RESTARTS   AGE
-nfs-server-provisioner-0   1/1     Running   0          25s
-
-$ kubectl get sc
-NAME                 PROVISIONER            AGE
-nfs                  cloudflow-nfs          29s
-standard (default)   kubernetes.io/gce-pd   2m57s
-----
-
-NOTE:: The documented NFS storage class is very portable and has been verified to work on GKE, EKS, AKS and Openshift.
-
-== More Storage Options
-
-Continue with xref:more-storage-options.adoc[]
-
-== Next: Installing Cloudflow
-
-Continue with xref:installing-cloudflow.adoc[]
+include::{cloudflow-version}@shared-content:administration:page$installation-prerequisites.adoc[]
diff --git a/docs/docs-source/docs/modules/administration/pages/installing-cloudflow.adoc b/docs/docs-source/docs/modules/administration/pages/installing-cloudflow.adoc
@@ -4,6 +4,7 @@
 :toclevels: 2
 
 include::ROOT:partial$include.adoc[]
+include::{cloudflow-version}@shared-content:administration:page$installing-cloudflow.adoc[]
 
 This guide shows how to install Cloudflow, using the `Helm` and `kubectl` command-line tools.
 It also shows how to install Kafka, Spark, and Flink operators that integrate with Cloudflow.
@@ -137,6 +138,4 @@ If you plan to write applications that utilize Flink you will need to install th
 
 Continue with xref:installing-flink-operator.adoc[Adding Flink Support].
 
-== Next: Installing Enterprise components
 
-Continue with xref:installing-enterprise.adoc[]