Long text gets discarded during splitter step #103
Description
Initial Checks
- I have searched GitHub for a duplicate issue and I'm sure this is something new
- I have searched Google & StackOverflow for a solution and couldn't find anything
- I have read and followed the documentation and still think this is a bug
Description
In the following example, a long text is discarded during the splitter step. This behavior is unexpected. Even a full paragraph is removed between chunk 3 and 4.
Discarded text:
and outputs a searchable knowledge graph. With PyKnowFlow, each of these steps is a well-defined module, and connecting them is just a matter of configuration.
## Comparative Table of PyKnowFlow Modules
Below is a comparative table listing 10 core modules in PyKnowFlow. Each row gives an idea about the module's name, purpose, average execution time in seconds, and a long description of its behavior and utility in knowledge pipelines.
Input:
# PyKnowFlow: An Awesome Python Framework for Knowledge Pipelines
## Introduction
In the rapidly evolving world of data and artificial intelligence, efficient knowledge management is essential. From raw data ingestion to the final knowledge products, every step in the pipeline needs to be optimized, scalable, and easy to maintain. Enter **PyKnowFlow**, a fictional but revolutionary Python framework tailored for building and managing knowledge pipelines. This document provides an overview of PyKnowFlow, discussing its capabilities, architecture, and the features that make it a standout choice for developers and data engineers alike.
## Why PyKnowFlow?
The need for a robust framework stems from the challenges data professionals face daily: integrating multiple data sources, transforming data effectively, ensuring data lineage, managing versioning, and deploying models or insights to various endpoints. Traditional ETL pipelines or data workflows often fall short when it comes to semantic understanding and knowledge enrichment. PyKnowFlow bridges that gap.
Here are some compelling reasons to consider PyKnowFlow:
* **Semantic Awareness**: Unlike conventional pipelines, PyKnowFlow understands the meaning and context behind the data, enabling more intelligent transformations.
* **Modular Architecture**: Each stage of the pipeline is a reusable component, making it easy to plug and play different modules depending on the use case.
* **Open Source and Extensible**: Developers can extend the core functionalities or contribute new modules.
* **Integration-Ready**: Works seamlessly with popular tools like Pandas, Spark, SQLAlchemy, FastAPI, and more.
## Core Features
PyKnowFlow is built with scalability and developer productivity in mind. Here are the core features that define its power:
### 1. Declarative Pipeline Configuration
PyKnowFlow uses YAML or JSON-based configuration files to define pipelines. This promotes reproducibility and allows non-developers to understand and modify workflows easily.
### 2. Intelligent Data Parsers
Data ingestion is made smarter with built-in parsers that can infer data types, schema, and even potential relationships between entities.
### 3. Graph-Based Workflow Execution
The framework models pipelines as directed acyclic graphs (DAGs), similar to Airflow, but with richer semantics and better support for knowledge-specific operations like entity resolution, ontological tagging, and inference.
### 4. Built-In Ontology Support
PyKnowFlow supports integration with popular ontologies and knowledge bases. You can enrich your data with semantic metadata effortlessly.
### 5. Monitoring and Lineage Tracking
Track every transformation, every version, and every movement of data across the pipeline using intuitive visualizations and comprehensive logs.
## Example Use Case
Let’s say you’re working on a pipeline that extracts academic papers, identifies authors, links them to institutions, enriches the content with keywords and tags from an ontology, and outputs a searchable knowledge graph. With PyKnowFlow, each of these steps is a well-defined module, and connecting them is just a matter of configuration.
## Comparative Table of PyKnowFlow Modules
Below is a comparative table listing 10 core modules in PyKnowFlow. Each row gives an idea about the module's name, purpose, average execution time in seconds, and a long description of its behavior and utility in knowledge pipelines.
| Module Name | Purpose | Avg Execution Time (s) | Description |
| ------------------ | -------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------- |
| DataIngestor | Load raw data from sources | 5.2 | Connects to various data sources (APIs, databases, flat files) and loads raw, unprocessed data streams. |
| SchemaInferencer | Infer structure and schema | 3.1 | Analyzes raw data to automatically determine data types, structures, and inter-field relationships. |
| EntityResolver | Resolve entity duplicates | 7.6 | Uses fuzzy logic and ML techniques to merge similar records referring to the same real-world entity. |
| OntologyTagger | Semantic tagging | 4.4 | Enriches records by tagging entities and concepts based on a predefined or custom ontology. |
| RelationshipMapper | Link related data entities | 6.3 | Identifies and maps relationships between different data entities to build a coherent knowledge graph. |
| DataCleaner | Clean and normalize data | 2.5 | Removes duplicates, handles nulls, standardizes formats, and ensures consistency in datasets. |
| PipelineVisualizer | Visualize the workflow | 1.8 | Renders interactive DAG-based visual representations of the pipeline’s structure and flow. |
| LineageTracker | Track data provenance | 2.9 | Captures and visualizes the flow and transformation history of each data element in the pipeline. |
| ModelDeployer | Deploy models or insights | 4.0 | Supports seamless deployment of models or analytical results to APIs, dashboards, or external systems. |
| FeedbackLoop | Integrate human feedback | 3.6 | Allows integration of user feedback to refine, retrain or adjust knowledge representations dynamically. |
## Extensibility and Plugins
PyKnowFlow supports a robust plugin system. Whether you're building a connector to a new database or designing a transformation module for a proprietary format, plugins allow you to easily extend functionality. Community contributions are encouraged, and the plugin registry ensures quality and security.
## Community and Ecosystem
Although fictional, PyKnowFlow boasts an enthusiastic open-source community (in our imagination). The framework is well-documented, with extensive tutorials, templates, and a supportive forum. Regular community calls, webinars, and collaborative hackathons foster innovation and inclusivity.
## Final Thoughts
In summary, PyKnowFlow offers a refreshing take on building knowledge pipelines with Python. Its semantic-first, modular approach not only simplifies development but also unlocks new capabilities in how we structure and utilize data. Whether you are managing academic research, enterprise knowledge bases, or building a semantic web crawler, PyKnowFlow is designed to handle it all.
Give it a try, contribute a plugin, or just explore the possibilities — PyKnowFlow might just be the missing piece in your data puzzle.
Chunk 3/6:
# PyKnowFlow: An Awesome Python Framework for Knowledge Pipelines
## Core Features
PyKnowFlow is built with scalability and developer productivity in mind. Here are the core features that define its power:
### 1. Declarative Pipeline Configuration
PyKnowFlow uses YAML or JSON-based configuration files to define pipelines. This promotes reproducibility and allows non-developers to understand and modify workflows easily.
### 2. Intelligent Data Parsers
Data ingestion is made smarter with built-in parsers that can infer data types, schema, and even potential relationships between entities.
### 3. Graph-Based Workflow Execution
The framework models pipelines as directed acyclic graphs (DAGs), similar to Airflow, but with richer semantics and better support for knowledge-specific operations like entity resolution, ontological tagging, and inference.
### 4. Built-In Ontology Support
PyKnowFlow supports integration with popular ontologies and knowledge bases. You can enrich your data with semantic metadata effortlessly.
### 5. Monitoring and Lineage Tracking
Track every transformation, every version, and every movement of data across the pipeline using intuitive visualizations and comprehensive logs.
## Example Use Case
Let’s say you’re working on a pipeline that extracts academic papers, identifies authors, links them to institutions, enriches the content with keywords and tags from an ontology,
Chunk 4/6:
# PyKnowFlow: An Awesome Python Framework for Knowledge Pipelines
- Core Features
| Module Name | Purpose | Avg Execution Time (s) | Description |
| ------------------ | -------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------- |
| DataIngestor | Load raw data from sources | 5.2 | Connects to various data sources (APIs, databases, flat files) and loads raw, unprocessed data streams. |
| SchemaInferencer | Infer structure and schema | 3.1 | Analyzes raw data to automatically determine data types, structures, and inter-field relationships. |
| EntityResolver | Resolve entity duplicates | 7.6 | Uses fuzzy logic and ML techniques to merge similar records referring to the same real-world entity. |
| OntologyTagger | Semantic tagging | 4.4 | Enriches records by tagging entities and concepts based on a predefined or custom ontology. |
| RelationshipMapper | Link related data entities | 6.3 | Identifies and maps relationships between different data entities to build a coherent knowledge graph. |
| DataCleaner | Clean and normalize data | 2.5 | Removes duplicates, handles nulls, standardizes formats, and ensures consistency in datasets. |
Example Code
Version & Environment
table splitter PR