Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add help for launching notebooks in any pyspark or sparkr application #3979

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/docs/ocean-spark/_sidebar.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@
- [Run Apps from Airflow](ocean-spark/tools-integrations/run-apps-from-airflow)
- [Spark Connect](ocean-spark/tools-integrations/spark-connect)
- [JDBC](ocean-spark/tools-integrations/jdbc)
- [JupyterLab](ocean-spark/tools-integrations/jupyter-lab)
- [VS Code](ocean-spark/tools-integrations/vs-code)
- [Hive Metastore](ocean-spark/tools-integrations/hive-metastore)
- [AWS Glue Data Catalog](ocean-spark/tools-integrations/aws-glue-catalog)
- [Docker Images Release Notes](ocean-spark/docker-images-release-notes/)
Expand Down
38 changes: 38 additions & 0 deletions src/docs/ocean-spark/tools-integrations/jupyter-lab.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Jupyter Lab

Install and launch JupyterLab server on any Spark application image with Python installed.

## Configuration

To start a Spark application with SparkConnect server, either run the mainClass SparkConnectServer or enable the SparkConnect plugin. Using the Spark Connect plugin, the application can run other tasks or services while enabling Spark Connect.

### Launch JupyterLab using the JupyterLab main class

```json
"mainClass": "com.netapp.spark.JupyterLab",
"deps": {
"packages": ["com.netapp.spark:jupyter-plugin:1.3.0"],
"repositories": ["https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters"]
},
```

### Launch JupyterLab using the spark plugin

```json
"sparkConf": {
"spark.plugins": "com.netapp.spark.JupyterPlugin",
"spark.jupyter.work.dir": "/opt/spark/work-dir"
},
"deps": {
"packages": ["com.netapp.spark:jupyter-plugin:1.3.0"],
"repositories": ["https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters"]
},
```

## Access

Access the JupyterLab server from a URL in the following format

```
https://console.spotinst.com/api/ocean/spark/cluster/osc-mycluster/app/spark-myapp/notebook/
```
38 changes: 38 additions & 0 deletions src/docs/ocean-spark/tools-integrations/vs-code.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# VS Code Server

Install and launch VS Code server on any Spark application image with Python installed.

## Configuration

To start a Spark application with SparkConnect server, either run the mainClass SparkConnectServer or enable the SparkConnect plugin. Using the Spark Connect plugin, the application can run other tasks or services while enabling Spark Connect.

### Launch VS Code Server using the VSCodeServer main class

```json
"mainClass": "com.netapp.spark.VSCodeServer",
"deps": {
"packages": ["com.netapp.spark:vscode-plugin:1.3.0"],
"repositories": ["https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters"]
},
```

### Launch VS Code Server using the spark plugin

```json
"sparkConf": {
"spark.plugins": "com.netapp.spark.SparkCodeServerPlugin",
"spark.jupyter.work.dir": "/opt/spark/work-dir"
},
"deps": {
"packages": ["com.netapp.spark:jupyter-plugin:1.3.0"],
"repositories": ["https://us-central1-maven.pkg.dev/ocean-spark/ocean-spark-adapters"]
},
```

## Access

Access the JupyterLab server from a URL in the following format

```
https://console.spotinst.com/api/ocean/spark/cluster/osc-mycluster/app/spark-myapp/code/
```
Loading