update to the paper for JOSS

Author: mantzaris

.github/workflows/compile-paper.yml

@@ -13,3 +13,9 @@ generate_corpus_CFG(complexity=20, num_sentences=100, enable_polysemy=false, bas
   No Changes to `~/Documents/repos/BenchmarkDataNLP.jl/Project.toml`
   No Changes to `~/Documents/repos/BenchmarkDataNLP.jl/Manifest.toml`
 .../repos/BenchmarkDataNLP.jl$ julia --project=. docs/make.jl
+
+
+# for the paper
+pandoc paper.md --pdf-engine=xelatex -o paper.pdf
+sudo apt-get install pandoc-citeproc
+pandoc paper.md   --filter pandoc-citeproc   --pdf-engine=xelatex   -o paper.pdf

notes.txt

@@ -0,0 +1,48 @@
+
+@inproceedings{copestake1996applying,
+  title={Applying natural language processing techniques to speech prostheses},
+  author={Copestake, Ann},
+  booktitle={Working Notes of the 1996 AAAI Fall Symposium on Developing Assistive Technology for People with Disabilities},
+  year={1996}
+}
+
+
+
+@article{maletti2017survey,
+  title={Survey: Finite-state technology in natural language processing},
+  author={Maletti, Andreas},
+  journal={Theoretical Computer Science},
+  volume={679},
+  pages={2--17},
+  year={2017},
+  publisher={Elsevier}
+}
+
+
+
+@article{faye2012survey,
+  title={A survey of RDF storage approaches},
+  author={Faye, David C and Cure, Olivier and Blin, Guillaume},
+  journal={Revue Africaine de Recherche en Informatique et Math{\'e}matiques Appliqu{\'e}es},
+  volume={15},
+  year={2012},
+  publisher={Episciences. org}
+}
+
+
+@article{van1996generalized,
+  title={Generalized context-free grammars},
+  author={van Vugt, Nik{\`e}},
+  journal={Internal Report},
+  year={1996},
+  publisher={Citeseer}
+}
+
+@inproceedings{samsi2023words,
+  title={From words to watts: Benchmarking the energy costs of large language model inference},
+  author={Samsi, Siddharth and Zhao, Dan and McDonald, Joseph and Li, Baolin and Michaleas, Adam and Jones, Michael and Bergeron, William and Kepner, Jeremy and Tiwari, Devesh and Gadepally, Vijay},
+  booktitle={2023 IEEE High Performance Extreme Computing Conference (HPEC)},
+  pages={1--9},
+  year={2023},
+  organization={IEEE}
+}

paper/paper.bib

@@ -1,28 +1,51 @@
 ---
-title: 'Your Software Title'
+title: "BenchmarkDataNLP.jl: Synthetic Data Generation for NLP Benchmarking"
 tags:
-  - tag1
-  - tag2
-  - tag3
+  - Julia
+  - NLP
+  - benchmarking
+  - data generation
+  - language models
 authors:
-  - name: Your Name
-    orcid: Your ORCID
+  - name: "Alexander V. Mantzaris"
+    orcid: 0000-0002-0026-5725
     affiliation: 1
 affiliations:
-  - name: Your Institution
+  - name: "Department of Statistics and Data Science, University of Central Florida (UCF), USA"
     index: 1
-date: YYYY-MM-DD
+date: 25 January 2025
 bibliography: paper.bib
 ---
 
 # Summary
 
-A brief summary of what your software does and its high-level functionality.
+**BenchmarkDataNLP.jl** is a package written in Julia Lang for generating synthetic text corpora that can be used to systematically benchmark and evaluate Natural Language Processing (NLP) models such as RNNs, LSTMs, and Large Language Models (LLMs). By enabling users to control core linguistic parameters such as the alphabet size, vocabulary size, grammatical expansion complexity, and semantic structures this library can help users test (as well as evaluate or debug) NLP models. Instead of exposing many of the key internal parameter choices to the user a minimal number are so that users do not have to focus on how to correctly configure the generation process. The key value for the user is the *complexity* integer value which controls the size of the alphabet, vocabulary and grammar expansions. This allows a range starting from 1 so that very simple text corpora can be generated. For example at complexity=1 there are 5 letters in the alphabet and 10 words used in the vocabulary with 2 grammar roles when a Context Free Grammar generator is select. Users can choose how many independent productions are desired which are each supplied as entries in a .jsonl file. The defaults provided in the documentation should suffice for most use cases. 
 
-# Statement of Need
+The generator methodology approaches are:
 
-Explain the research purpose of the software and its context in related work.
+  - Context Free Grammar [@van1996generalized]
+  - Resource Description Framework (RDF, triple store) [@faye2012survey]
+  - Finite State Machine [@maletti2017survey]
+  - Template Strings [@copestake1996applying]
+
+Each method offers different options to the user. With the Finite State Machine approach, the function *generate_fsm_corpus* allows for users to produce a deterministic set of productions so that upon successful training accuracies of 100% can be achieved. An application of this is that different models can be tested on the amount of computational requirements they have for a particular value of complexity. The 4 approaches cover a wide range of text expansion production methods that does not have parameter estimations as part of their usage requirements. The letters of the alphabet begin in the *HANGUL* start (44032, 0xAC00) and does not apply repetition of labels with integer identifiers at the end which are only convenient for implementations but may not be ideal for some tokenization approaches.
+
+This package also hopes to bring down the cost required when exploring different models and architectures that would require large amounts of energy, time and finances to train on large natural language datasets [@samsi2023words]. Risks of this nature can often be expected when very novel approaches are explored.
+
+# Statement of need
+
+Performance evaluation of NLP systems often hinges on realistic datasets that capture the target domain’s linguistic nuances. However, training on large-scale, text corpora can be expensive or impractical, especially when testing very particular aspects of language complexity or model robustness. Synthetic data generation helps bridge these gaps by:
+
+1. **Reproducibility**: Controlled parameters (e.g., sentence length, grammar depth, or concept re-use) allow reproducible experiments.
+2. **Customization**: Researchers can stress-test models by systematically varying language properties, such as the number of roles in a grammar or the frequency of filler tokens.
+3. **Scalability**: Large-scale data can be generated for benchmarking advanced architectures without the need for extensive, real-world data collection.
+4. **Targeted Evaluation**: By manipulating semantic structures (for example, adding context continuity with RDF triples or specialized placeholders), researchers can investigate whether models capture specific linguistic or contextual features.
+
+Although several libraries and benchmarks (e.g., GLUE, SuperGLUE, and others) provide curated datasets, **BenchmarkDataNLP.jl** offers a unique approach by allowing *fine-grained control* of the underlying complexity of a  synthetic corpus generation process. This capability is especially valuable when exploring model failure modes or for rapid prototyping of new model architectures that require specialized text patterns. It is hoped that this will bring down the cost for initial prototyping of new model architectures and allow a greater exploration. This can also help compare different modeling approaches.
+
+# Acknowledgements
+
+We thank the Julia community for their continued support of open-source scientific computing.
 
 # References
 
-Add your references here.