Skip to content

Commit 90b60fc

Browse files
ddematheuddematheu
authored
0.0.40 (#59)
* 0.0.40 * unifying documents onto main repo
1 parent 92c1298 commit 90b60fc

File tree

96 files changed

+3723
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

96 files changed

+3723
-1
lines changed

‎docs/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Neum AI Docs powered by Mintlify
2+
3+
### Development
4+
5+
Install the [Mintlify CLI](https://www.npmjs.com/package/mintlify) to preview the documentation changes locally. To install, use the following command
6+
7+
```
8+
npm i -g mintlify
9+
```
10+
11+
Run the following command at the root of your documentation (where mint.json is)
12+
13+
```
14+
mintlify dev
15+
```
16+
17+
### Publishing Changes
18+
19+
Install our Github App to autopropagate changes from youre repo to your deployment. Changes will be deployed to production automatically after pushing to the default branch. Find the link to install on your dashboard.
20+
21+
#### Troubleshooting
22+
23+
- Mintlify dev isn't running - Run `mintlify install` it'll re-install dependencies.
24+
- Page loads as a 404 - Make sure you are running in a folder with `mint.json`

‎docs/_snippets/snippet-example.mdx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
## My Snippet
2+
3+
<Info>This is an example of a reusable snippet</Info>

‎docs/components/chunkers/CharacterChunker.mdx‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
title: 'CharacterChunker'
3+
description: 'This class is responsible for chunking text data into smaller pieces based on character count, with optional overlapping between chunks.'
4+
---
5+
6+
The `CharacterChunker` class is designed to break down large text documents into smaller, more manageable chunks of text. This process is based on the number of characters, which can be defined by the user.
7+
8+
## Properties
9+
10+
Required properties:
11+
- None
12+
13+
Optional properties:
14+
- `chunk_size`: The number of characters each chunk should contain.
15+
- `chunk_overlap`: The number of characters that can overlap between consecutive chunks.
16+
- `batch_size`: The number of chunks to process in one batch.
17+
- `separator`: The character used to separate chunks.
18+
19+
<CodeGroup>
20+
```python Local Development
21+
from neumai.Chunkers import CharacterChunker
22+
23+
character_chunker = CharacterChunker(
24+
chunk_size = 500,
25+
chunk_overlap = 50,
26+
batch_size = 1000,
27+
separator = "\n"
28+
)
29+
```
30+
31+
```json Cloud
32+
{
33+
"sources":[
34+
{
35+
# Add Data Connector and Loader
36+
"chunker": {
37+
"chunker_name":"CharacterChunker",
38+
"chunker_information":{
39+
"chunk_size": 500,
40+
"chunk_overlap": 50,
41+
"batch_size": 1000,
42+
"separators": "\n"
43+
},
44+
}
45+
}
46+
]
47+
}
48+
```
49+
</CodeGroup>

‎docs/components/chunkers/CustomChunker.mdx‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
title: 'CustomChunker'
3+
description: 'CustomChunker is a flexible class designed to chunk text according to user-provided code, allowing for personalized text segmentation strategies.'
4+
---
5+
6+
<Warning>These capabilities are currently in beta. Please contact [founders@tryneum.com](mailto:founders@tryneum.com) with any questions or asks.</Warning>
7+
8+
The `CustomChunker` class in the Neum AI framework chunks documents dynamically based on a user-defined code snippet. This class offers the flexibility to implement custom text chunking logic that can be tailored to specific use cases.
9+
10+
## Properties
11+
12+
Required properties:
13+
- `code`: A string of Python code that defines how the text should be chunked.
14+
15+
Optional properties:
16+
- `batch_size`: The number of chunks to process in one go, with a default of 1000 if not specified.
17+
18+
<CodeGroup>
19+
```python Local Development
20+
!pip install neumai-tools
21+
22+
from neumai.Chunkers import CustomChunker
23+
from neumai.Shared import NeumDocument
24+
from neumai_tools import semantic_chunking
25+
26+
# Define a custom code snippet for chunking
27+
custom_chunking_code = """
28+
def split_text_into_chunks(text) -> List[NeumDocument]:
29+
# Custom logic to split text
30+
return ["Chunk1", "Chunk2", ...]
31+
"""
32+
33+
# Create a CustomChunker instance with the required code
34+
custom_chunker = CustomChunker(
35+
code = custom_chunking_code,
36+
batch_size = 1000
37+
)
38+
39+
```
40+
41+
```json Cloud
42+
{
43+
"sources":[
44+
{
45+
"chunker": {
46+
"chunker_name":"CustomChunker",
47+
"chunker_information":{
48+
"code": "def split_text_into_chunks(text) -> List[NeumDocument]: ...",
49+
"batch_size": 1000
50+
}
51+
}
52+
}
53+
]
54+
}
55+
```
56+
</CodeGroup>

‎docs/components/chunkers/RecursiveChunker.mdx‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: 'RecursiveChunker'
3+
description: 'The RecursiveChunker class is tailored for segmenting text into smaller chunks with a recursive approach, allowing for different levels of text granularity.'
4+
---
5+
6+
The `RecursiveChunker` class specializes in dividing text into smaller, more manageable sections using a recursive method. This allows for various levels of granularity depending on the specified separators.
7+
8+
## Properties
9+
10+
Required properties:
11+
- None
12+
13+
Optional properties:
14+
- `chunk_size`: The target size for each text chunk.
15+
- `chunk_overlap`: The amount of overlap desired between adjacent text chunks.
16+
- `batch_size`: The number of text chunks to process together.
17+
- `separators`: A list of strings used to split the text at different granularity levels.
18+
19+
<CodeGroup>
20+
```python Local Development
21+
from neumai.Chunkers import RecursiveChunker
22+
from neumai.Shared import NeumDocument
23+
24+
recursive_chunker = RecursiveChunker(
25+
chunk_size = 500,
26+
chunk_overlap = 50,
27+
batch_size = 1000,
28+
separators = ["\n\n", "\n", " ", ""]
29+
)
30+
31+
```
32+
33+
```json Cloud
34+
{
35+
"sources":[
36+
{
37+
# Add Data Connector and Loader
38+
"chunker": {
39+
"chunker_name":"RecursiveChunker",
40+
"chunker_information":{
41+
"chunk_size": 500,
42+
"chunk_overlap": 50,
43+
"batch_size": 1000,
44+
"separators": ["\n\n", "\n", " ", ""]
45+
},
46+
}
47+
}
48+
]
49+
}
50+
```
51+
</CodeGroup>

‎docs/components/chunkers/sample-chunker.mdx‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
title: Class Name
3+
description: 'Class Description'
4+
---
5+
6+
Short description of connector
7+
8+
## Properties
9+
10+
Required properties:
11+
-a
12+
-b
13+
-c
14+
15+
Optional properties:
16+
-a
17+
-b
18+
-c
19+
20+
Available metadata:
21+
-a
22+
-b
23+
-c
24+
25+
Available content:
26+
-a
27+
-b
28+
-c
29+
30+
Compatible loaders:
31+
-a
32+
-b
33+
-c
34+
35+
<CodeGroup>
36+
```python Local Development
37+
from neumai.DataConnectors import NeumWebsiteConnector
38+
from neumai.Shared import Selector
39+
40+
neumWebsiteConnector = NeumWebsiteConnector(
41+
connector_information = {
42+
"url":"https://www.neum.ai/post/retrieval-augmented-generation-at-scale"
43+
},
44+
selector = Selector(
45+
to_embed=['website'],
46+
to_metadata=['url']
47+
)
48+
)
49+
```
50+
51+
```json Cloud
52+
{
53+
"sources":[
54+
{
55+
"connector": {
56+
"connector_name":"NeumWebsiteConnector",
57+
"connector_information":{
58+
"url":"https://www.neum.ai/post/retrieval-augmented-generation-at-scale"
59+
},
60+
"selector": {
61+
"to_embed": [
62+
"website"
63+
],
64+
"to_metadata": [
65+
"url"
66+
]
67+
}
68+
}
69+
}
70+
]
71+
}
72+
```
73+
74+
</CodeGroup>

‎docs/components/data-connectors/AzureBlobConnector.mdx‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
title: 'Azure Blob Connector'
3+
description: 'Retrieve data from Azure Blob storage'
4+
---
5+
6+
Connect to an existing Azure Blob Storage and pull any data stored. To connect Azure Blob Storage you will need your credentials to access the resource including the connection string and container name you want to access.
7+
8+
## Properties
9+
10+
Required properties:
11+
- `connection_string`: Connection string to the Azure Blob Storage
12+
- `container_name`: Name of the container in Azure Blob Storage
13+
14+
Available metadata
15+
- `name`: Name of the blob
16+
- `last_modified`: Time of last modification
17+
- `creation_time`: Time of blob creation
18+
- `last_access_on`: Time of last blob access
19+
20+
Compatible loaders:
21+
- AutoLoader
22+
- HTMLLoader
23+
- MarkdownLoader
24+
- NeumCSVLoader
25+
- NeumJSONLoader
26+
- PDFLoader
27+
28+
## Usage
29+
30+
<CodeGroup>
31+
```python Local Development
32+
from neumai.DataConnectors import AzureBlobConnector
33+
from neumai.Shared import Selector
34+
35+
azure_blob_connector = AzureBlobConnector(
36+
connection_string = "<INSERT AZURE BLOB CONNECTION STRING>",
37+
container_name = "<INSER CONTAINER NAME IN BLOB STORAGE>",
38+
selector = Selector(
39+
to_metadata=['name']
40+
)
41+
)
42+
```
43+
44+
```json Cloud
45+
{
46+
"sources":[
47+
{
48+
"data_connector": {
49+
"connector_name":"AzureBlobConnector",
50+
"connector_information":{
51+
"connection_string":"<INSERT AZURE BLOB CONNECTION STRING",
52+
"container_name":"<INSER CONTAINER NAME IN BLOB STORAGE",
53+
"selector": {
54+
"to_metadata": [
55+
"name"
56+
]
57+
}
58+
},
59+
}
60+
}
61+
]
62+
}
63+
```
64+
65+
</CodeGroup>

‎docs/components/data-connectors/FileConnector.mdx‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
title: 'File Connector'
3+
description: 'Process any public file'
4+
---
5+
6+
The connector extracts and processes any publicly available file. Make sure the URL to the file provided is publicly available.
7+
8+
## Properties
9+
10+
Required properties:
11+
- `url`: URL to public file
12+
13+
Available metadata
14+
- `url`: URL of file
15+
16+
Compatible loaders:
17+
- AutoLoader
18+
- HTMLLoader
19+
- MarkdownLoader
20+
- NeumCSVLoader
21+
- NeumJSONLoader
22+
- PDFLoader
23+
24+
## Usage
25+
26+
<CodeGroup>
27+
```python Local Development
28+
from neumai.DataConnectors import FileConnector
29+
from neumai.Shared import Selector
30+
31+
file_connector = FileConnector(
32+
url = "https://www.neum.ai/secrets.pdf",
33+
selector = Selector(
34+
to_metadata=['url']
35+
)
36+
)
37+
```
38+
39+
```json Cloud
40+
{
41+
"sources":[
42+
{
43+
"data_connector": {
44+
"connector_name":"FileConnector",
45+
"connector_information":{
46+
"url":"https://www.neum.ai/secrets.pdf",
47+
"selector": {
48+
"to_metadata": [
49+
"url"
50+
]
51+
}
52+
}
53+
}
54+
}
55+
]
56+
}
57+
```
58+
59+
</CodeGroup>

0 commit comments

Comments
 (0)