KontextWork
diff --git a/‎.devcontainer/README.md
+4-4 b/‎.devcontainer/README.md
+4-4
diff --git a/‎.github/ISSUE_TEMPLATE/01_bug.yml
+1-1 b/‎.github/ISSUE_TEMPLATE/01_bug.yml
+1-1
diff --git a/‎.github/ISSUE_TEMPLATE/02_feature.yml
-3 b/‎.github/ISSUE_TEMPLATE/02_feature.yml
-3
diff --git a/‎.github/workflows/build-and-push-image-semver.yaml
+7 b/‎.github/workflows/build-and-push-image-semver.yaml
+7
diff --git a/‎.github/workflows/build-and-push-image.yaml
+14-5 b/‎.github/workflows/build-and-push-image.yaml
+14-5
diff --git a/‎.github/workflows/check-translations.yaml
+1-1 b/‎.github/workflows/check-translations.yaml
+1-1
diff --git a/‎.github/workflows/dev-build.yaml
+11-1 b/‎.github/workflows/dev-build.yaml
+11-1
diff --git a/‎.gitignore
+1 b/‎.gitignore
+1
diff --git a/‎.gitmodules
+2-2 b/‎.gitmodules
+2-2
diff --git a/‎.prettierignore
+1 b/‎.prettierignore
+1
diff --git a/‎.vscode/settings.json
+1 b/‎.vscode/settings.json
+1
diff --git a/‎BARE_METAL.md
+4-7 b/‎BARE_METAL.md
+4-7
diff --git a/‎README.md
+6-3 b/‎README.md
+6-3
diff --git a/‎collector/extensions/resync/index.js
+1-1 b/‎collector/extensions/resync/index.js
+1-1
diff --git a/‎collector/index.js
+2-2 b/‎collector/index.js
+2-2
diff --git a/‎collector/package.json
+5-3 b/‎collector/package.json
+5-3
diff --git a/‎collector/processLink/convert/generic.js
+28-7 b/‎collector/processLink/convert/generic.js
+28-7
@@ -20,7 +20,7 @@ Welcome to the AnythingLLM development container configuration, designed to crea
 
 ## Getting Started
 
-1. Using Github Codepaces. Just select to create a new workspace, and the devcontainer will be created for you.
+1. Using GitHub Codespaces. Just select to create a new workspace, and the devcontainer will be created for you.
 
 2. Using your Local VSCode (Release or Insiders). We suggest you first make a fork of the repo and then clone it to your local machine using VSCode tools. Then open the project folder in VSCode, which will prompt you to open the project in a devcontainer. Select yes, and the devcontainer will be created for you. If this does not happen, you can open the command palette and select "Remote-Containers: Reopen in Container".
 
@@ -36,16 +36,16 @@ Checklist:
 
 - [ ] The usual message asking you to start the Server and Frontend in different windows are now "hidden" in the building process of the devcontainer. Don't forget to do as suggested.
 - [ ] Open a JavaScript file, for example "server/index.js" and check if `eslint` is working. It will complain that `'err' is defined but never used.`. This means it is working.
-- [ ] Open a React File, for example, "frontend/src/main.jsx," and check if `eslint` complains about `Fast refresh only works when a file has exports. Move your component(s) to a separate file.`. Again, it means `eslint` is working. Now check at the status bar if the `Prettier` has a double checkmark :heavy_check_mark: (double). It means Prettier is working. You will see a nice extension `Formatting:`:heavy_check_mark: that can be used to disable the `Format on Save` feature temporarily. 
+- [ ] Open a React File, for example, "frontend/src/main.jsx," and check if `eslint` complains about `Fast refresh only works when a file has exports. Move your component(s) to a separate file.`. Again, it means `eslint` is working. Now check at the status bar if the `Prettier` has a double checkmark :heavy_check_mark: (double). It means Prettier is working. You will see a nice extension `Formatting:`:heavy_check_mark: that can be used to disable the `Format on Save` feature temporarily.
 - [ ] Check if, on the left pane, you have the NPM Scripts (this may be disabled; look at the "Explorer" tree-dots up-right). There will be scripts inside the `package.json` files. You will basically need to run the `dev:collector`, `dev:server` and the `dev:frontend` in this order. When the frontend finishes starting, a window browser will open **inside** the VSCode. Still, you can open it outside.
 
 :warning: **Important for all developers** :warning:
 
 - [ ] When you are using the `NODE_ENV=development` the server will not store the configurations you set for security reasons. Please set the proper config on file `.env.development`. The side-effect if you don't, everytime you restart the server, you will be sent to the "Onboarding" page again.
 
-**Note when using Github Codespaces**
+**Note when using GitHub Codespaces**
 
-- [ ] When running the "Server" for the first time, it will automatically configure its port to be publicly accessible by default, as this is required for the front end to reach the server backend. To know more, read the content of the `.env` file on the frontend folder about this, and if any issues occur, make sure to manually set the port "Visibility" of the "Server" is set to "Public" if needed. Again, this is only needed for developing on Github Codespaces.
+- [ ] When running the "Server" for the first time, it will automatically configure its port to be publicly accessible by default, as this is required for the front end to reach the server backend. To know more, read the content of the `.env` file on the frontend folder about this, and if any issues occur, make sure to manually set the port "Visibility" of the "Server" is set to "Public" if needed. Again, this is only needed for developing on GitHub Codespaces.
 
 
 **For the Collector:**
 
@@ -6,7 +6,7 @@ body:
   - type: markdown
     attributes:
       value: | 
-        Use this template to file a bug report for AnythingLLM. Please be as descriptive as possible to allow everyone to replicate and solve your issue. Want help contributing a PR? Use our repo chatbot by OnboardAI! https://learnthisrepo.com/anythingllm"
+        Use this template to file a bug report for AnythingLLM. Please be as descriptive as possible to allow everyone to replicate and solve your issue.
   - type: dropdown
     id: runtime
     attributes:
 
@@ -9,9 +9,6 @@ body:
         Share a new idea for a feature or improvement. Be sure to search existing
         issues first to avoid duplicates.
 
-        Want help contributing a PR? Use our repo chatbot by OnboardAI! https://learnthisrepo.com/anythingllm
-
-
   - type: textarea
     id: description
     attributes:
 
@@ -35,8 +35,15 @@ jobs:
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
 
+      # Since Docker Buildx v20.1 - segfaults occur on arm64 with ubuntu-latest.
+      # So we pin to v0.19.0 for now until the issue is resolved for runners via QEMU.
+      # Alternatively, we can move to ARM runners and remove the QEMU setup, but this previously
+      # has issues (see comment at the top)
+      # https://github.com/NVIDIA/cuda-quantum/pull/2651#issue-2870353124
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
+        with:
+          version: v0.19.0
 
       - name: Log in to Docker Hub
         uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
 
@@ -1,9 +1,9 @@
-# This Github action is for publishing of the primary image for AnythingLLM
+# This GitHub action is for publishing of the primary image for AnythingLLM
 # It will publish a linux/amd64 and linux/arm64 image at the same time
 # This file should ONLY BY USED FOR `master` BRANCH. 
-# TODO: Update `runs-on` for arm64 when GitHub finally supports
-# native arm environments as QEMU takes around 1 hour to build
-# ref: https://github.com/actions/runner-images/issues/5631 :(
+# TODO: GitHub now has an ubuntu-24.04-arm64 runner, but we still need
+# to use QEMU to build the arm64 image because Chromium is not available for Linux arm64
+# so builds will still fail, or fail much more often. Its inconsistent and frustrating.
 name: Publish AnythingLLM Primary Docker image (amd64/arm64)
 
 concurrency:
@@ -15,6 +15,7 @@ on:
     branches: ['master'] # master branch only. Do not modify.
     paths-ignore:
       - '**.md'
+      - '.gitmodules'
       - 'cloud-deployments/**/*'
       - 'images/**/*'
       - '.vscode/**/*'
@@ -24,6 +25,7 @@ on:
       - 'embed/**/*' # Embed is submodule
       - 'browser-extension/**/*' # Chrome extension is submodule
       - 'server/utils/agents/aibitat/example/**/*' # Do not push new image for local dev testing of new aibitat images.
+      - 'extras/**/*' # Extra is just for news and other local content.
 
 jobs:
   push_multi_platform_to_registries:
@@ -52,8 +54,15 @@ jobs:
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
 
+      # Since Docker Buildx v20.1 - segfaults occur on arm64 with ubuntu-latest.
+      # So we pin to v0.19.0 for now until the issue is resolved for runners via QEMU.
+      # Alternatively, we can move to ARM runners and remove the QEMU setup, but this previously
+      # has issues (see comment at the top)
+      # https://github.com/NVIDIA/cuda-quantum/pull/2651#issue-2870353124
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
+        with:
+          version: v0.19.0
 
       - name: Log in to Docker Hub
         uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
@@ -131,4 +140,4 @@ jobs:
               $tag
             done
           done
-        shell: bash
+        shell: bash
@@ -1,4 +1,4 @@
-# This Github action is for validation of all languages which translations are offered for
+# This GitHub action is for validation of all languages which translations are offered for
 # in the locales folder in `frontend/src`. All languages are compared to the EN translation
 # schema since that is the fallback language setting. This workflow will run on all PRs that
 # modify any files in the translation directory
 
@@ -6,16 +6,19 @@ concurrency:
 
 on:
   push:
-    branches: ['docker-scout-patch'] # put your current branch to create a build. Core team only.
+    branches: ['3536-feat-new-chat-home-page'] # put your current branch to create a build. Core team only.
     paths-ignore:
       - '**.md'
       - 'cloud-deployments/*'
       - 'images/**/*'
       - '.vscode/**/*'
       - '**/.env.example'
       - '.github/ISSUE_TEMPLATE/**/*'
+      - '.devcontainer/**/*'
       - 'embed/**/*' # Embed should be published to frontend (yarn build:publish) if any changes are introduced
+      - 'browser-extension/**/*' # Chrome extension is submodule
       - 'server/utils/agents/aibitat/example/**/*' # Do not push new image for local dev testing of new aibitat images.
+      - 'extras/**/*' # Extra is just for news and other local content.
 
 jobs:
   push_multi_platform_to_registries:
@@ -41,8 +44,14 @@ jobs:
           fi
         id: dockerhub
 
+      # Uncomment this + add linux/arm64 to platforms if you want to build for arm64 as well
+      # - name: Set up QEMU
+      #   uses: docker/setup-qemu-action@v3
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
+        with:
+          version: v0.19.0
 
       - name: Log in to Docker Hub
         uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
@@ -70,6 +79,7 @@ jobs:
           sbom: true
           provenance: mode=max
           platforms: linux/amd64
+          # platforms: linux/amd64,linux/arm64
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
           cache-from: type=gha
 
@@ -9,3 +9,4 @@ v-env
 aws_cf_deploy_anything_llm.json
 yarn.lock
 *.bak
+.idea
@@ -1,7 +1,7 @@
 [submodule "browser-extension"]
 	path = browser-extension
-	url = git@github.com:Mintplex-Labs/anythingllm-extension.git
+	url = https://github.com/Mintplex-Labs/anythingllm-extension.git
 [submodule "embed"]
 	path = embed
-	url = git@github.com:Mintplex-Labs/anythingllm-embed.git
+	url = https://github.com/Mintplex-Labs/anythingllm-embed.git
 	branch = main
@@ -10,6 +10,7 @@ frontend/bundleinspector.html
 
 #server
 server/swagger/openapi.json
+server/**/*.mjs
 
 #embed
 **/static/**
 
@@ -41,6 +41,7 @@
     "opendocument",
     "openrouter",
     "pagerender",
+    "ppio",
     "Qdrant",
     "royalblue",
     "SearchApi",
 
@@ -5,7 +5,7 @@
 > You are fully responsible for securing your deployment and data in this mode.
 > **Any issues** experienced from bare-metal or non-containerized deployments will be **not** answered or supported.
 
-Here you can find the scripts and known working process to run AnythingLLM outside of a Docker container. This method of deployment is preferable for those using local LLMs and want native performance on their devices.
+Here you can find the scripts and known working process to run AnythingLLM outside of a Docker container.
 
 ### Minimum Requirements
 > [!TIP]
@@ -33,7 +33,7 @@ STORAGE_DIR="/your/absolute/path/to/server/storage"
 5. Edit the `frontend/.env` file for the `VITE_BASE_API` to now be set to `/api`. This is documented in the .env for which one you should use.
 ```
 # VITE_API_BASE='http://localhost:3001/api' # Use this URL when developing locally
-# VITE_API_BASE="https://$CODESPACE_NAME-3001.$GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN/api" # for Github Codespaces
+# VITE_API_BASE="https://$CODESPACE_NAME-3001.$GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN/api" # for GitHub Codespaces
 VITE_API_BASE='/api' # Use this URL deploying on non-localhost address OR in docker.
 ```
 
@@ -47,20 +47,17 @@ AnythingLLM is comprised of three main sections. The `frontend`, `server`, and `
 2. Copy `frontend/dist` to `server/public` - `cp -R frontend/dist server/public`.
 This should create a folder in `server` named `public` which contains a top level `index.html` file and various other files/folders.
 
-_(optional)_ Build native LLM support if using `native` as your LLM.
-`cd server && npx --no node-llama-cpp download`
-
 3. Migrate and prepare your database file.
 ```
 cd server && npx prisma generate --schema=./prisma/schema.prisma
 cd server && npx prisma migrate deploy --schema=./prisma/schema.prisma
 ```
 
 4. Boot the server in production
-`cd server && NODE_ENV=production node index.js &` 
+`cd server && NODE_ENV=production node index.js &`
 
 5. Boot the collection in another process
-`cd collector && NODE_ENV=production node index.js &` 
+`cd collector && NODE_ENV=production node index.js &`
 
 AnythingLLM should now be running on `http://localhost:3001`!
 
 
@@ -55,11 +55,13 @@ AnythingLLM divides your documents into objects called `workspaces`. A Workspace
 
 ## Cool features of AnythingLLM
 
-- 🆕 [**Custom AI Agents**](https://docs.anythingllm.com/agent/custom/introduction)
+- 🆕 [**Full MCP-compatibility**](https://docs.anythingllm.com/mcp-compatibility/overview)
+- 🆕 [**No-code AI Agent builder**](https://docs.anythingllm.com/agent-flows/overview)
 - 🖼️ **Multi-modal support (both closed and open-source LLMs!)**
+- [**Custom AI Agents**](https://docs.anythingllm.com/agent/custom/introduction)
 - 👤 Multi-user instance support and permissioning _Docker version only_
-- 🦾 Agents inside your workspace (browse the web, run code, etc)
-- 💬 [Custom Embeddable Chat widget for your website](./embed/README.md) _Docker version only_
+- 🦾 Agents inside your workspace (browse the web, etc)
+- 💬 [Custom Embeddable Chat widget for your website](https://github.com/Mintplex-Labs/anythingllm-embed/blob/main/README.md) _Docker version only_
 - 📖 Multiple document type support (PDF, TXT, DOCX, etc)
 - Simple chat UI with Drag-n-Drop funcitonality and clear citations.
 - 100% Cloud deployment ready.
@@ -98,6 +100,7 @@ AnythingLLM divides your documents into objects called `workspaces`. A Workspace
 - [Apipie](https://apipie.ai/)
 - [xAI](https://x.ai/)
 - [Novita AI (chat models)](https://novita.ai/model-api/product/llm-api?utm_source=github_anything-llm&utm_medium=github_readme&utm_campaign=link)
+- [PPIO](https://ppinfra.com?utm_source=github_anything-llm)
 
 **Embedder models:**
 
 
@@ -95,7 +95,7 @@ async function resyncGithub({ chunkSource }, response) {
       sourceFilePath: source.searchParams.get('path'),
     });
 
-    if (!success) throw new Error(`Failed to sync Github file content. ${reason}`);
+    if (!success) throw new Error(`Failed to sync GitHub file content. ${reason}`);
     response.status(200).json({ success, content });
   } catch (e) {
     console.error(e);
 
@@ -83,9 +83,9 @@ app.post(
   "/util/get-link",
   [verifyPayloadIntegrity],
   async function (request, response) {
-    const { link } = reqBody(request);
+    const { link, captureAs = "text" } = reqBody(request);
     try {
-      const { success, content = null } = await getLinkText(link);
+      const { success, content = null } = await getLinkText(link, captureAs);
       response.status(200).json({ url: link, success, content });
     } catch (e) {
       console.error(e);
 
@@ -10,8 +10,8 @@
     "node": ">=18.12.1"
   },
   "scripts": {
-    "dev": "NODE_ENV=development nodemon --ignore hotdir --ignore storage --trace-warnings index.js",
-    "start": "NODE_ENV=production node index.js",
+    "dev": "cross-env NODE_ENV=development nodemon --ignore hotdir --ignore storage --trace-warnings index.js",
+    "start": "cross-env NODE_ENV=production node index.js",
     "lint": "yarn prettier --ignore-path ../.prettierignore --write ./processSingleFile ./processLink ./utils index.js"
   },
   "dependencies": {
@@ -40,6 +40,7 @@
     "puppeteer": "~21.5.2",
     "sharp": "^0.33.5",
     "slugify": "^1.6.6",
+    "tesseract.js": "^6.0.0",
     "url-pattern": "^1.0.3",
     "uuid": "^9.0.0",
     "wavefile": "^11.0.0",
@@ -48,6 +49,7 @@
   },
   "devDependencies": {
     "nodemon": "^2.0.22",
-    "prettier": "^2.4.1"
+    "prettier": "^2.4.1",
+    "cross-env": "^7.0.3"
   }
 }
@@ -6,9 +6,20 @@ const { writeToServerDocuments } = require("../../utils/files");
 const { tokenizeString } = require("../../utils/tokenizer");
 const { default: slugify } = require("slugify");
 
-async function scrapeGenericUrl(link, textOnly = false) {
-  console.log(`-- Working URL ${link} --`);
-  const content = await getPageContent(link);
+/**
+ * Scrape a generic URL and return the content in the specified format
+ * @param {string} link - The URL to scrape
+ * @param {('html' | 'text')} captureAs - The format to capture the page content as
+ * @param {boolean} processAsDocument - Whether to process the content as a document or return the content directly
+ * @returns {Promise<Object>} - The content of the page
+ */
+async function scrapeGenericUrl(
+  link,
+  captureAs = "text",
+  processAsDocument = true
+) {
+  console.log(`-- Working URL ${link} => (${captureAs}) --`);
+  const content = await getPageContent(link, captureAs);
 
   if (!content.length) {
     console.error(`Resulting URL content was empty at ${link}.`);
@@ -19,7 +30,7 @@ async function scrapeGenericUrl(link, textOnly = false) {
     };
   }
 
-  if (textOnly) {
+  if (!processAsDocument) {
     return {
       success: true,
       content,
@@ -41,7 +52,7 @@ async function scrapeGenericUrl(link, textOnly = false) {
     published: new Date().toLocaleString(),
     wordCount: content.split(" ").length,
     pageContent: content,
-    token_count_estimate: tokenizeString(content).length,
+    token_count_estimate: tokenizeString(content),
   };
 
   const document = writeToServerDocuments(
@@ -52,7 +63,13 @@ async function scrapeGenericUrl(link, textOnly = false) {
   return { success: true, reason: null, documents: [document] };
 }
 
-async function getPageContent(link) {
+/**
+ * Get the content of a page
+ * @param {string} link - The URL to get the content of
+ * @param {('html' | 'text')} captureAs - The format to capture the page content as
+ * @returns {Promise<string>} - The content of the page
+ */
+async function getPageContent(link, captureAs = "text") {
   try {
     let pageContents = [];
     const loader = new PuppeteerWebBaseLoader(link, {
@@ -64,7 +81,11 @@ async function getPageContent(link) {
         waitUntil: "networkidle2",
       },
       async evaluate(page, browser) {
-        const result = await page.evaluate(() => document.body.innerText);
+        const result = await page.evaluate((captureAs) => {
+          if (captureAs === "text") return document.body.innerText;
+          if (captureAs === "html") return document.documentElement.innerHTML;
+          return document.body.innerText;
+        }, captureAs);
         await browser.close();
         return result;
       },
Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-# This Github action is for validation of all languages which translations are offered for`
	`1`	`+# This GitHub action is for validation of all languages which translations are offered for`
`2`	`2`	# in the locales folder in `frontend/src`. All languages are compared to the EN translation
`3`	`3`	`# schema since that is the fallback language setting. This workflow will run on all PRs that`
`4`	`4`	`# modify any files in the translation directory`