Doc updates: LanceDB, HuggingFace and Marqo (#60)

 docs and a couple small fixes

Author: ddematheu

docs/components/embed-connectors/HuggigFaceEmbed.mdx

@@ -0,0 +1,26 @@
+---
+title: 'HuggingFaceEmbed'
+description: 'The HuggingFaceEmbed connector is designed to generate embeddings for text data using Hugging Face hosted emedding models.
+---
+
+Access the plethora of open-source models hosted through Hugging face using the `HuggingFaceEmbed` connector. 
+
+<Note>Looking for embedding models in HuggingFace? Here is a great place to start: [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard)</Note>
+
+## Properties
+
+Required properties:
+- model: HuggingFace model ID or a URL to a deployed Inference Endpoint.
+- token: HuggingFace token.
+
+<CodeGroup>
+```python Local Development
+from neumai.EmbedConnectors import HuggingFaceEmbed
+
+azure_openai_embed = HuggingFaceEmbed(
+    model = "model_id_or_url",
+    token = "huggingface_token"
+)
+
+```
+</CodeGroup>

docs/components/sink-connectors/LanceDBSink.mdx

@@ -0,0 +1,45 @@
+---
+
+title: 'LanceDBSink'
+description: 'LanceDBSink enables seamless integration with LanceDB vector database, supporting vector storage and similarity search for advanced data retrieval.'
+
+---
+
+The `LanceDBSink` class is a connector for LanceDB, an open-source, serverless vector database built for seamless integration and scale.
+
+## Properties
+
+Required properties:
+
+- uri: URI for the LanceDB database.
+- table_name: Name of the LanceDB table to be used.
+
+Optional properties:
+
+- api_key: If provided, connect to LanceDB cloud; otherwise, connect to a database on file system or cloud storage.
+region: Region for the use of LanceDB cloud.
+- create_index: Boolean to decide whether to create an index for ANN search or use flat search.
+- metric: The distance metric to use (default is 'cosine').
+- num_partitions: The number of partitions of the index.
+- num_sub_vectors: The number of sub-vectors created during Product Quantization (PQ).
+- accelerator: Specifies the accelerator to use for the index creation process (e.g., GPU or MPS).
+
+<Note>Index creation is only required when dealing with 100k+ vectors. Below that threshold, set create_index to `false`. For more information on index creation and configuring partitions and sub vectors see: [LanceDB documentation](https://lancedb.github.io/lancedb/ann_indexes/#creating-an-ivf_pq-index)</Note>
+
+<CodeGroup>
+```python Local Development
+from neumai.SinkConnectors import LanceDBSink
+
+# Setup the LanceDBSink with required credentials and index information
+lancedb_sink = LanceDBSink(
+    uri = "lancedb_uri",
+    table_name = "test_table",
+    # if using LanceDB Cloud add 
+    # api_key = "lancedb_cloud_api_key",
+    # if using for more thatn 100k vectors then add:
+    # create_index = True
+    # ensure that your vector dimensions (ex. 1536 for OpenAI text-ada-002) is divisible by num_sub_vectors (default 96)
+    # ensure that num_partitions is less than the number of vectors you are adding (default to 256)
+)
+```
+</CodeGroup>

docs/components/sink-connectors/MarqoSink.mdx

@@ -0,0 +1,33 @@
+---
+
+title: 'MarqoSink'
+description: 'MarqoSink enables seamless integration with Marqo's vector database, supporting vector storage and similarity search for advanced data retrieval.'
+
+---
+
+The `MarqoSink` class is a connector for Marqo, a vector database provider that enables multimodal vector search.
+
+## Properties
+
+Required properties:
+
+- url: URL for accessing the Marqo service.
+- index_name: Name of the index in Marqo where the data will be stored.
+
+Optional properties:
+
+- api_key: API key required for authenticating with the Marqo cloud service.
+
+<CodeGroup>
+```python Local Development
+from neumai.SinkConnectors import MarqoSink
+
+# Setup the PineconeSink with required credentials and index information
+marqo_sink = MarqoSink(
+    url = "marqo_url",
+    index_name = "test_index",
+    # If using Marqo cloud then add:
+    # api_key = "marqo_cloud_api_key"
+)
+```
+</CodeGroup>

…i/EmbedConnectors/HuggingfaceEmbedder.py …umai/EmbedConnectors/HuggingFaceEmbed.pyneumai/neumai/EmbedConnectors/HuggingfaceEmbedder.py renamed to neumai/neumai/EmbedConnectors/HuggingFaceEmbed.py neumai/neumai/EmbedConnectors/HuggingfaceEmbedder.py renamed to neumai/neumai/EmbedConnectors/HuggingFaceEmbed.py

@@ -17,7 +17,7 @@ def embed_name(self) -> str:
 
     @property
     def required_properties(self) -> List[str]:
-        return ["token"]
+        return ["model","token"]
 
     @property
     def optional_properties(self) -> List[str]:

neumai/neumai/EmbedConnectors/__init__.py

@@ -2,4 +2,5 @@
 from .AzureOpenAIEmbed import AzureOpenAIEmbed
 from .EmbedConnector import EmbedConnector
 from .OpenAIEmbed import OpenAIEmbed
-from .EmbedConnector import EmbedConnector
+from .EmbedConnector import EmbedConnector
+from .HuggingFaceEmbed import HuggingFaceEmbed