Feat: semi self-hosted LLMs using AzureML Online Endpoints

infra/README.md

@@ -1,24 +1,39 @@
 # Infra-as-Code setup for Docq.AI hosting
 
+- `app` folder - IaC for hosting the Docq.AI app on various cloud providers.
+- `inference` folder - IaC for hosting ML models used by the Docq app for inference.
+
+## Install Azure CLI
+
+- Install core Azure CLI `brew install azure-cli`
+- Install Azure ML CLI v2 (remove v1 if exists, then install v2)
+
+  ```terminal
+  az extension remove --name ml
+
+  az extension add --name ml --yes
+  ```
+
 ## Azure ARM Templates
 
 These docs are mainly for contributing and developing the various deployment methods available in the `/infra` folder. We recommend users start with installation instruction layed out in the user guide in the main docs site. But feel free if you want to get your hands dirty.
 
-The ARM template in `/infra/azure/arm` powers the whizard based one-click deploy method described in the main docs.
+The ARM template in `/infra/azure/arm` powers the wizard based one-click deploy method described in the main docs.
 
 ### Deploy and destroy scripts
 
-There two scripts combine several azure CLI commands to for convinience.
+There two scripts combine several azure CLI commands to for convenience.
 
-Running `./deploy.sh` is the easiest way to test when interating on the template.
+Running `./deploy.sh` is the easiest way to test when iterating on the template.
 
 - `./deploy.sh <NAME> <LOCATION> <RESOURCE_GROUP>` - args are optional. creates a resource group and deploys the ARM template based on several defaults. Inspect the script to discover the defaults and available parameters. Params can be overridden by passing argument values in order
-- `./destroy.sh <NAME> <LOCATION> <RESOURCE_GROUP>` - args are optional. Destroys the resource group and all resources within. Handles purging all Congnitive Services in the resource group that are deleted.
+- `./destroy.sh <NAME> <LOCATION> <RESOURCE_GROUP>` - args are optional. Destroys the resource group and all resources within. Handles purging all Cognitive Services in the resource group that are deleted.
 
 ### Useful CLI commands
 
 If using the scripts above you shouldn't need these but occasionally you they might help when troubleshooting.
 
+- authenticate `az login`
 - Create resource group CLI - `az group create --name docq-rg-westeurope --location westeurope`
 - Deploy template CLI - `az deployment group create --resource-group docq-rg-westeurope --name docq1 --template-file appservice.json`
 - Delete resources in resource group - `az group delete --name docq-rg-westeurope`
@@ -30,19 +45,19 @@ If using the scripts above you shouldn't need these but occasionally you they mi
 
 See the `models` tab in Azure AI Studio <https://oai.azure.com/portal> for models available to the specific Azure account along with version numbers"
 
-See API ref for detials on avail options <https://learn.microsoft.com/en-us/rest/api/cognitiveservices/azureopenaistable/deployments/create>
+See API ref for details on avail options <https://learn.microsoft.com/en-us/rest/api/cognitiveservices/azureopenaistable/deployments/create>
 
 Explanation about models <https://learn.microsoft.com/en-gb/azure/cognitive-services/openai/concepts/models>
 
 ### Testing the template
 
-- Run `./deploy.sh` to test the template deploys all resources sucessfully.
+- Run `./deploy.sh` to test the template deploys all resources successfully.
 - Navigate to the app URL for this instance. Verify the app is working as expect.
 - Test template deployment method aka one-click deploy. This is important as some times what works when deploying from the CLI doesn't work in template deployment.
   - push the template change to your branch (origin) such that it's publicly available.
   - Copy the 'raw' URL for the template file: Navigate to the file on Github.com. Click on the 'raw' button on the top right area.
   - URL encode the github raw URL.
-  - Trigger Azure template deployment by navigaiting to `https://portal.azure.com/#create/Microsoft.Template/uri/<encoded Github raw URL to the template file on your branch>`
+  - Trigger Azure template deployment by navigating to `https://portal.azure.com/#create/Microsoft.Template/uri/<encoded Github raw URL to the template file on your branch>`
   Example: The URL on main <https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdocqai%2Fdocq%2Fmain%2Finfra%2Fazure%2Farm%2Fappservice.json>
 
 ## AWS

infra/azure/arm/destroy.sh → infra/app/azure/arm/destroy.sh

@@ -4,8 +4,6 @@ LOCATION="${2:-"westeurope"}"
 
 RESOURCE_GROUP="${3:-${NAME}-rg-${LOCATION}}"
 
-
-
 read -p "This will delete all resources in resource group '${RESOURCE_GROUP}'. Are you sure? [y/n]" confirm
 
 if [ $confirm = "y" ] || [ $confirm = "Y" ] 

infra/inference/azure/arm/deploy.sh

@@ -0,0 +1,194 @@
+set -x
+
+NAME="${1:-"docq"}"
+
+LOCATION="${2:-"westeurope"}"
+
+RESOURCE_GROUP="${NAME}-ml-rg-${LOCATION}"
+
+
+res1=$(az group create --name $RESOURCE_GROUP --location $LOCATION)
+
+
+WORKSPACE="${NAME}-main-ws-${LOCATION}"
+
+az ml workspace create --name $WORKSPACE --resource-group $RESOURCE_GROUP --location $LOCATION
+wait
+
+#<get_access_token>
+TOKEN=$(az account get-access-token --query accessToken -o tsv)
+#</get_access_token>
+
+# <create_variables>
+SUBSCRIPTION_ID=$(az account show --query id -o tsv)
+#LOCATION=$(az ml workspace show --query location -o tsv)
+#RESOURCE_GROUP=$(az group show --query name -o tsv)
+#WORKSPACE=$(az configure -l --query "[?name=='workspace'].value" -o tsv)
+#</create_variables>
+
+# <set_endpoint_name>
+#export ENDPOINT_NAME=endpoint-`echo $RANDOM`
+export ENDPOINT_NAME="${NAME}-endpoint"
+# </set_endpoint_name>
+
+#<api_version>
+API_VERSION="2022-05-01"
+#</api_version>
+
+echo -e "Using:\nSUBSCRIPTION_ID=$SUBSCRIPTION_ID\nLOCATION=$LOCATION\nRESOURCE_GROUP=$RESOURCE_GROUP\nWORKSPACE=$WORKSPACE"
+
+# define how to wait  
+wait_for_completion () {
+    operation_id=$1
+    status="unknown"
+
+    if [[ $operation_id == "" || -z $operation_id  || $operation_id == "null" ]]; then
+        echo "operation id cannot be empty"
+        exit 1
+    fi
+
+    while [[ $status != "Succeeded" && $status != "Failed" ]]
+    do
+        echo "Getting operation status from: $operation_id"
+        operation_result=$(curl --location --request GET $operation_id --header "Authorization: Bearer $TOKEN")
+        # TODO error handling here
+        status=$(echo $operation_result | jq -r '.status')
+        echo "Current operation status: $status"
+        sleep 5
+    done
+
+    if [[ $status == "Failed" ]]
+    then
+        error=$(echo $operation_result | jq -r '.error')
+        echo "Error: $error"
+    fi
+}
+
+# # <get_storage_details>
+# # Get values for storage account
+# response=$(curl --location --request GET "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE/datastores?api-version=$API_VERSION&isDefault=true" \
+# --header "Authorization: Bearer $TOKEN")
+# AZUREML_DEFAULT_DATASTORE=$(echo $response | jq -r '.value[0].name')
+# AZUREML_DEFAULT_CONTAINER=$(echo $response | jq -r '.value[0].properties.containerName')
+# export AZURE_STORAGE_ACCOUNT=$(echo $response | jq -r '.value[0].properties.accountName')
+# # </get_storage_details>
+
+# # <upload_code>
+# az storage blob upload-batch -d $AZUREML_DEFAULT_CONTAINER/score -s model-1/onlinescoring --account-name $AZURE_STORAGE_ACCOUNT
+# # </upload_code>
+
+# # <create_code>
+# az deployment group create -g $RESOURCE_GROUP \
+# --template-file code-version.json \
+# --parameters \
+# workspaceName=$WORKSPACE \
+# codeAssetName="score-sklearn" \
+# codeUri="https://$AZURE_STORAGE_ACCOUNT.blob.core.windows.net/$AZUREML_DEFAULT_CONTAINER/score"
+# # </create_code>
+
+# # <upload_model>
+# az storage blob upload-batch -d $AZUREML_DEFAULT_CONTAINER/model -s model-1/model --account-name $AZURE_STORAGE_ACCOUNT
+# # </upload_model>
+
+# # <create_model>
+# az deployment group create -g $RESOURCE_GROUP \
+# --template-file model-version.json \
+# --parameters \
+# workspaceName=$WORKSPACE \
+# modelAssetVersion=6 \
+# modelAssetName="Llama-2-13b-chat" \
+# modelUri="azureml://registries/azureml-meta/models/Llama-2-13b-chat/versions/6"
+# # </create_model>
+
+# # <read_condafile>
+# CONDA_FILE=$(cat model-1/environment/conda.yaml)
+# # </read_condafile>
+
+# # <create_environment>
+# ENV_VERSION=$RANDOM
+# az deployment group create -g $RESOURCE_GROUP \
+# --template-file environment-version.json \
+# --parameters \
+# workspaceName=$WORKSPACE \
+# environmentAssetName=sklearn-env \
+# environmentAssetVersion=$ENV_VERSION \
+# dockerImage=mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210727.v1 \
+# condaFile="$CONDA_FILE"
+# # </create_environment>
+
+# <create_endpoint>
+az deployment group create -g $RESOURCE_GROUP \
+    --template-file online-endpoint.json \
+    --parameters \
+    workspaceName=$WORKSPACE \
+    onlineEndpointName=$ENDPOINT_NAME \
+    identityType=SystemAssigned \
+    authMode=AMLToken \
+    location=$LOCATION
+# </create_endpoint>
+
+# <get_endpoint>
+response=$(curl --location --request GET "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE/onlineEndpoints/$ENDPOINT_NAME?api-version=$API_VERSION" \
+--header "Content-Type: application/json" \
+--header "Authorization: Bearer $TOKEN")
+
+operation_id=$(echo $response | jq -r '.properties.properties.AzureAsyncOperationUri')
+wait_for_completion $operation_id
+# </get_endpoint>
+
+LLAMA2_7B_CHAT="azureml://registries/azureml-meta/models/Llama-2-7b-chat/versions/8"
+SATVIKAG_CHATBOT="azureml://registries/HuggingFace/models/satvikag-chatbot/versions/3"
+SELECTED_MODEL=$LLAMA2_7B_CHAT
+MODEL_NAME="llama2-7b-chat-8"
+
+# <create_deployment>
+resourceScope="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices"
+az deployment group create -g $RESOURCE_GROUP \
+ --template-file online-endpoint-deployment.json \
+ --parameters \
+ workspaceName=$WORKSPACE \
+ location=$LOCATION \
+ onlineEndpointName=$ENDPOINT_NAME \
+ onlineDeploymentName=$MODEL_NAME \
+ model=$SELECTED_MODEL \
+ endpointComputeType="Managed" \
+ skuName="Standard_NC12s_v3" \
+ skuCapacity=1
+ # </create_deployment>
+
+# <get_deployment>
+response=$(curl --location --request GET "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE/onlineEndpoints/$ENDPOINT_NAME/deployments/blue?api-version=$API_VERSION" \
+--header "Content-Type: application/json" \
+--header "Authorization: Bearer $TOKEN")
+
+operation_id=$(echo $response | jq -r '.properties.properties.AzureAsyncOperationUri')
+wait_for_completion $operation_id
+
+scoringUri=$(echo $response | jq -r '.properties.scoringUri')
+# </get_endpoint>
+
+# <get_endpoint_access_token>
+response=$(curl -H "Content-Length: 0" --location --request POST "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE/onlineEndpoints/$ENDPOINT_NAME/token?api-version=$API_VERSION" \
+--header "Authorization: Bearer $TOKEN")
+accessToken=$(echo $response | jq -r '.accessToken')
+# </get_endpoint_access_token>
+
+# <score_endpoint>
+curl --location --request POST $scoringUri \
+    --header "Authorization: Bearer $accessToken" \
+    --header "Content-Type: application/json" \
+    --data-raw @sample-request.json
+# </score_endpoint>
+
+# <get_deployment_logs>
+curl --location --request POST "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE/onlineEndpoints/$ENDPOINT_NAME/deployments/blue/getLogs?api-version=$API_VERSION" \
+    --header "Authorization: Bearer $TOKEN" \
+    --header "Content-Type: application/json" \
+    --data-raw "{ \"tail\": 100 }"
+# </get_deployment_logs>
+
+# # <delete_endpoint>
+# curl --location --request DELETE "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.MachineLearningServices/workspaces/$WORKSPACE/onlineEndpoints/$ENDPOINT_NAME?api-version=$API_VERSION" \
+# --header "Content-Type: application/json" \
+# --header "Authorization: Bearer $TOKEN" || true
+# # </delete_endpoint>

infra/inference/azure/arm/destroy.sh

@@ -0,0 +1,32 @@
+NAME="${1:-"docq"}"
+
+LOCATION="${2:-"westeurope"}"
+
+RESOURCE_GROUP="${3:-${NAME}-ml-rg-${LOCATION}}"
+
+read -p "This will delete all resources in resource group '${RESOURCE_GROUP}'. Are you sure? [y/n]" confirm
+
+if [ $confirm = "y" ] || [ $confirm = "Y" ] 
+then
+
+  workspaces=($(az ml workspace list --resource-group $RESOURCE_GROUP --query [].name --output tsv))
+  echo "Deleting ${#workspaces[@]} Azure ML Workspaces"
+
+  for workspace in $workspaces 
+  do
+      az ml workspace delete --name $workspace --resource-group $RESOURCE_GROUP --permanently-delete --all-resources --yes
+      echo "'${workspace}' deleted."
+  done
+  wait
+
+  res=$(az group delete --name ${RESOURCE_GROUP} -y)
+  echo $res | jq '.'
+  wait
+
+  echo "Success! All resources in resource group '${RESOURCE_GROUP}' were deleted."
+  exit 0
+elif [ $confirm = "n" ] || [ $confirm = "N" ] 
+then
+    echo "Aborted! nothing was destroyed."
+    exit -1
+fi

infra/inference/azure/arm/environment-version.json

@@ -0,0 +1,51 @@
+{
+  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
+  "contentVersion": "1.0.0.0",
+  "parameters": {
+    "workspaceName": {
+      "type": "string"
+    },
+    "environmentAssetName": {
+      "type": "string"
+    },
+    "environmentAssetVersion": {
+      "defaultValue": "1",
+      "type": "string"
+    },
+    "environmentDescription": {
+      "defaultValue": "This is a test description for an environment created from an ARM template",
+      "type": "string"
+    },
+    "condaFile": {
+      "defaultValue": "",
+      "type": "string",
+      "metadata": {
+        "description": "Standard configuration file used by Conda that lets you install any kind of package, including Python, R, and C/C++ packages."
+      }
+    },
+    "isAnonymous": {
+      "defaultValue": false,
+      "type": "bool"
+    },
+    "dockerImage": {
+      "defaultValue": "",
+      "type": "string",
+      "metadata": {
+        "description": "Docker image path, for example: 'docker.io/tensorflow/serving:latest'."
+      }
+    }
+  },
+  "resources": [
+    {
+      "type": "Microsoft.MachineLearningServices/workspaces/environments/versions",
+      "apiVersion": "2022-05-01",
+      "name": "[concat(parameters('workspaceName'), '/', parameters('environmentAssetName'), '/', parameters('environmentAssetVersion'))]",
+      "properties": {
+        "isAnonymous": "[parameters('isAnonymous')]",
+        "description": "[parameters('environmentDescription')]",
+        "image": "[parameters('dockerImage')]",
+        "condaFile": "[parameters('condaFile')]"
+      }
+    }
+  ]
+}