Creating SQLite features.

scrollmapper · Nov 9, 2024 · 0b23ed4 · 0b23ed4
1 parent e402dbc
commit 0b23ed4
Show file tree

Hide file tree

Showing 55 changed files with 333 additions and 27 deletions.
diff --git a/docs/2_scripting.md b/docs/2_scripting.md
@@ -2,61 +2,76 @@
 
 Python is the primary scripting language used in this project. To ensure consistency and compatibility, we recommend using Python 3.9. This version has been tested thoroughly with our scripts and provides the necessary features and stability for text management and data processing.
 
-
 ### Scripts Breakdown
 
-#### `build_mysql_database.py`
+#### `assemble_readme.py`
+- **Description**: Dynamically assembles the README.md file from various parts located in the `docs/main_readme` directory.
+- **Usage**: Run the script to generate the complete README.md.
 
-- **Description**: Builds a MySQL database from Bible translations and optionally includes cross-references. Prompts the user for database credentials and uses SQL dumps to populate the database.
-- **Usage**: Run the script and follow the prompts to select translations and include cross-references.
+- **export_sqlite_database.py**
+  - **Description**: Creates an SQLite database for a selected Bible translation and includes cross references. Prompts the user for the path where the new database should be built.
+  - **Usage**: Run the script and follow the prompts to create the SQLite database with cross references.
 
-#### `generate_all_versions.py`
+#### `build_sqlite_database.py`
+- **Description**: Creates an SQLite database for a selected Bible translation and includes cross references. Prompts the user for the path where the new database should be built.
+- **Usage**: Run the script and follow the prompts to create the SQLite database with cross references.
 
-- **Description**: Automates the generation of all Bible translations in multiple formats (SQL, CSV, JSON, TXT, YAML, MD). Iterates through all available translations and creates the corresponding files.
+#### `generate_all_versions.py`
+- **Description**: Automates the generation of all Bible translations in multiple formats (SQL, SQLite, CSV, JSON, TXT, YAML, MD). Iterates through all available translations and creates the corresponding files.
 - **Usage**: Run the script to generate all formats for each translation.
 
-#### `generate_cross_references_mysql.py`
+#### `generate_cross_references.py`
+- **Description**: Generates cross-reference data for Bible translations. Processes raw cross-reference data and formats it for use with Bible translations.
+- **Usage**: Run the script to create cross-reference files.
 
+#### `generate_cross_references_mysql.py`
 - **Description**: Specifically generates cross-reference data for MySQL databases. Formats the cross-reference data into SQL insert statements.
 - **Usage**: Run the script to create cross-reference SQL files.
 
 #### `generate_csv.py`
-
 - **Description**: Generates CSV files for Bible translations. Each translation is processed and output as a CSV file.
 - **Usage**: Run the script to create CSV files for each translation.
 
 #### `generate_json.py`
-
 - **Description**: Generates JSON files for Bible translations. Each translation is processed and output as a JSON file.
 - **Usage**: Run the script to create JSON files for each translation.
 
 #### `generate_md.py`
-
 - **Description**: Generates Markdown (MD) files for Bible translations. Each translation is processed and output as a Markdown file.
 - **Usage**: Run the script to create Markdown files for each translation.
 
 #### `generate_mysql.py`
-
 - **Description**: Generates MySQL SQL dump files for Bible translations. Each translation is processed and output as an SQL dump file.
 - **Usage**: Run the script to create SQL dump files for each translation.
 
-#### `generate_txt.py`
+#### `generate_sqlite.py`
+- **Description**: Generates SQLite database files for Bible translations. Each translation is processed and output as an SQLite database file.
+- **Usage**: Run the script to create SQLite database files for each translation.
 
+#### `generate_txt.py`
 - **Description**: Generates plain text (TXT) files for Bible translations. Each translation is processed and output as a text file.
 - **Usage**: Run the script to create text files for each translation.
 
 #### `generate_yaml.py`
-
 - **Description**: Generates YAML files for Bible translations. Each translation is processed and output as a YAML file.
 - **Usage**: Run the script to create YAML files for each translation.
 
-#### `generate_cross_references.py`
+#### `export_sqlite_database.py`
+- **Description**: Exports data from the SQLite database to a specified format.
+- **Usage**: Run the script to export data from the SQLite database.
 
-- **Description**: Generates cross-reference data for Bible translations. Processes raw cross-reference data and formats it for use with Bible translations.
-- **Usage**: Run the script to create cross-reference files.
+### How to Run the Scripts
 
-#### `assemble_readme.py`
+1. **Setup**:
+   - Ensure you have Python **3.9** or higher installed on your machine.
+   - Clone the repository to your local machine.
+   - Navigate to the directory containing the scripts.
+
+2. **Run a Script**:
+   - Open a terminal or command prompt.
+   - Navigate to the script directory.
+   - Run the script by typing `python <script_name>.py` and follow any prompts if applicable.
 
-- **Description**: This script dynamically assembles the README.md file from various parts located in the `docs/main_readme` directory.
-- **Purpose**: Ensures the main README.md is up-to-date with the latest information and structure.
-- **Usage**: Run the script to generate the complete README.md.
+Example:
+```bash
+python generate_all_versions.py
diff --git a/docs/main_readme/introduction.md b/docs/main_readme/introduction.md
@@ -1,4 +1,4 @@
-# Bible Versions and Cross-Reference Databases: MySQL, CSV, JSON, YAML, TXT, MD.
+# Bible Versions and Cross-Reference Databases: MySQL, SQLite, CSV, JSON, YAML, TXT, MD.
 
 This is a collection of bible versions in different formats. Here are some quick introductions:
 

diff --git a/formats/sqlite/ACV.db b/formats/sqlite/ACV.db
diff --git a/formats/sqlite/ASV.db b/formats/sqlite/ASV.db
diff --git a/formats/sqlite/BBE.db b/formats/sqlite/BBE.db
diff --git a/formats/sqlite/BSB.db b/formats/sqlite/BSB.db
diff --git a/formats/sqlite/CPDV.db b/formats/sqlite/CPDV.db
diff --git a/formats/sqlite/DRC.db b/formats/sqlite/DRC.db
diff --git a/formats/sqlite/Darby.db b/formats/sqlite/Darby.db
diff --git a/formats/sqlite/Geneva1599.db b/formats/sqlite/Geneva1599.db
diff --git a/formats/sqlite/JPS.db b/formats/sqlite/JPS.db
diff --git a/formats/sqlite/Jubilee2000.db b/formats/sqlite/Jubilee2000.db
diff --git a/formats/sqlite/KJV.db b/formats/sqlite/KJV.db
diff --git a/formats/sqlite/KJVA.db b/formats/sqlite/KJVA.db
diff --git a/formats/sqlite/KJVPCE.db b/formats/sqlite/KJVPCE.db
diff --git a/formats/sqlite/LEB.db b/formats/sqlite/LEB.db
diff --git a/formats/sqlite/LITV.db b/formats/sqlite/LITV.db
diff --git a/formats/sqlite/MKJV.db b/formats/sqlite/MKJV.db
diff --git a/formats/sqlite/NHEB.db b/formats/sqlite/NHEB.db
diff --git a/formats/sqlite/NHEBJE.db b/formats/sqlite/NHEBJE.db
diff --git a/formats/sqlite/NHEBME.db b/formats/sqlite/NHEBME.db
diff --git a/formats/sqlite/Noyes.db b/formats/sqlite/Noyes.db
diff --git a/formats/sqlite/OEB.db b/formats/sqlite/OEB.db
diff --git a/formats/sqlite/RLT.db b/formats/sqlite/RLT.db
diff --git a/formats/sqlite/RWebster.db b/formats/sqlite/RWebster.db
diff --git a/formats/sqlite/Rotherham.db b/formats/sqlite/Rotherham.db
diff --git a/formats/sqlite/Twenty.db b/formats/sqlite/Twenty.db
diff --git a/formats/sqlite/Tyndale.db b/formats/sqlite/Tyndale.db
diff --git a/formats/sqlite/UKJV.db b/formats/sqlite/UKJV.db
diff --git a/formats/sqlite/Webster.db b/formats/sqlite/Webster.db
diff --git a/formats/sqlite/YLT.db b/formats/sqlite/YLT.db
diff --git a/formats/sqlite/extras/cross_references_0.db b/formats/sqlite/extras/cross_references_0.db
diff --git a/formats/sqlite/extras/cross_references_1.db b/formats/sqlite/extras/cross_references_1.db
diff --git a/formats/sqlite/extras/cross_references_2.db b/formats/sqlite/extras/cross_references_2.db
diff --git a/formats/sqlite/extras/cross_references_3.db b/formats/sqlite/extras/cross_references_3.db
diff --git a/formats/sqlite/extras/cross_references_4.db b/formats/sqlite/extras/cross_references_4.db
diff --git a/formats/sqlite/extras/cross_references_5.db b/formats/sqlite/extras/cross_references_5.db
diff --git a/formats/sqlite/extras/cross_references_6.db b/formats/sqlite/extras/cross_references_6.db
diff --git a/generators/__pycache__/base_generator.cpython-312.pyc b/generators/__pycache__/base_generator.cpython-312.pyc
diff --git a/generators/json/__pycache__/json_generator.cpython-312.pyc b/generators/json/__pycache__/json_generator.cpython-312.pyc
diff --git a/generators/sql/__pycache__/cross_references_generator_mysql.cpython-312.pyc b/generators/sql/__pycache__/cross_references_generator_mysql.cpython-312.pyc
diff --git a/generators/sql/__pycache__/mysql_generator.cpython-312.pyc b/generators/sql/__pycache__/mysql_generator.cpython-312.pyc
diff --git a/generators/sql/postgresql_generator.py b/generators/sql/postgresql_generator.py
diff --git a/generators/sql/sqlite_generator.py b/generators/sql/sqlite_generator.py
diff --git a/generators/sqlite/__pycache__/cross_references_generator_sqlite.cpython-312.pyc b/generators/sqlite/__pycache__/cross_references_generator_sqlite.cpython-312.pyc
diff --git a/generators/sqlite/__pycache__/sqlite_generator.cpython-312.pyc b/generators/sqlite/__pycache__/sqlite_generator.cpython-312.pyc
diff --git a/generators/sqlite/sqlite_generator.py b/generators/sqlite/sqlite_generator.py
@@ -0,0 +1,92 @@
+import os
+import json
+import sqlite3
+
+class SQLiteGenerator:
+    def __init__(self, source_dir, format_dir):
+        self.source_dir = source_dir
+        self.format_dir = format_dir
+
+    def generate(self, language, translation):
+        data = self.load_json(language, translation)
+        translation_name = self.get_readme_title(language, translation)
+        license_info = self.get_license_info(language, translation)
+        prepared_data = self.prepare_data(data)
+        sqlite_path = os.path.join(self.format_dir, 'sqlite', f'{translation}.db')
+
+        # Connect to SQLite database
+        conn = sqlite3.connect(sqlite_path)
+        cursor = conn.cursor()
+
+        # Create translations table if it doesn't exist
+        cursor.execute("""
+        CREATE TABLE IF NOT EXISTS translations (
+            translation TEXT PRIMARY KEY,
+            title TEXT,
+            license TEXT
+        );
+        """)
+        cursor.execute("""
+        INSERT OR IGNORE INTO translations (translation, title, license)
+        VALUES (?, ?, ?);
+        """, (translation, translation_name, license_info))
+
+        # Create books table
+        cursor.execute(f"""
+        CREATE TABLE IF NOT EXISTS {translation}_books (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            name TEXT
+        );
+        """)
+
+        # Insert books
+        for book in prepared_data['books']:
+            cursor.execute(f"INSERT INTO {translation}_books (name) VALUES (?);", (book['name'],))
+
+        # Create verses table
+        cursor.execute(f"""
+        CREATE TABLE IF NOT EXISTS {translation}_verses (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            book_id INTEGER,
+            chapter INTEGER,
+            verse INTEGER,
+            text TEXT,
+            FOREIGN KEY (book_id) REFERENCES {translation}_books(id)
+        );
+        """)
+
+        # Insert verses
+        for book_index, book in enumerate(prepared_data['books'], start=1):
+            for chapter in book['chapters']:
+                for verse in chapter['verses']:
+                    cursor.execute(f"""
+                    INSERT INTO {translation}_verses (book_id, chapter, verse, text)
+                    VALUES (?, ?, ?, ?);
+                    """, (book_index, chapter['chapter'], verse['verse'], verse['text']))
+
+        conn.commit()
+        conn.close()
+
+        print(f"SQLite database for {translation_name} ({translation}) generated at {sqlite_path}")
+
+    def get_license_info(self, language, translation):
+        readme_path = os.path.join(self.source_dir, language, translation, "README.md")
+        with open(readme_path, 'r', encoding='utf-8') as file:
+            for line in file:
+                if line.startswith("**License:**"):
+                    return line.split("**License:** ")[1].strip()
+        return "Unknown"
+
+    def load_json(self, language, translation):
+        json_path = os.path.join(self.source_dir, language, translation, f"{translation}.json")
+        with open(json_path, 'r', encoding='utf-8') as file:
+            return json.load(file)
+
+    def get_readme_title(self, language, translation):
+        readme_path = os.path.join(self.source_dir, language, translation, "README.md")
+        with open(readme_path, 'r', encoding='utf-8') as file:
+            return file.readline().strip()
+
+    def prepare_data(self, data):
+        # This method should prepare and return the data in the required format
+        return data
diff --git a/generators/text/__pycache__/csv_generator.cpython-312.pyc b/generators/text/__pycache__/csv_generator.cpython-312.pyc
diff --git a/generators/text/__pycache__/markdown_generator.cpython-312.pyc b/generators/text/__pycache__/markdown_generator.cpython-312.pyc
diff --git a/generators/text/__pycache__/plaintext_generator.cpython-312.pyc b/generators/text/__pycache__/plaintext_generator.cpython-312.pyc
diff --git a/generators/text/__pycache__/yaml_generator.cpython-312.pyc b/generators/text/__pycache__/yaml_generator.cpython-312.pyc
diff --git a/scripts/README.md b/scripts/README.md
@@ -5,19 +5,23 @@ This repository contains a collection of scripts designed to generate various fo
 ### Scripts
 
 - **build_mysql_database.py**
-  - **Description**: This script builds a MySQL database from Bible translations and optionally includes cross-references. It prompts the user for database credentials and uses SQL dumps to populate the database.
+  - **Description**: Builds a MySQL database from Bible translations and optionally includes cross-references. Prompts the user for database credentials and uses SQL dumps to populate the database.
   - **Usage**: Run the script and follow the prompts to select translations and include cross-references.
 
+- **export_sqlite_database.py**
+  - **Description**: Creates an SQLite database for a selected Bible translation and includes cross references. Prompts the user for the path where the new database should be built.
+  - **Usage**: Run the script and follow the prompts to create the SQLite database with cross references.
+
 - **generate_all_versions.py**
-  - **Description**: This script automates the generation of all Bible translations in multiple formats (SQL, CSV, JSON, TXT, YAML, MD). It iterates through all available translations and creates the corresponding files.
+  - **Description**: Automates the generation of all Bible translations in multiple formats (SQL, SQLite, CSV, JSON, TXT, YAML, MD). Iterates through all available translations and creates the corresponding files.
   - **Usage**: Run the script to generate all formats for each translation.
 
 - **generate_cross_references.py**
-  - **Description**: Generates cross-reference data for Bible translations. This script processes raw cross-reference data and formats it for use with Bible translations.
+  - **Description**: Generates cross-reference data for Bible translations. Processes raw cross-reference data and formats it for use with Bible translations.
   - **Usage**: Run the script to create cross-reference files.
 
 - **generate_cross_references_mysql.py**
-  - **Description**: Specifically generates cross-reference data for MySQL databases. It formats the cross-reference data into SQL insert statements.
+  - **Description**: Specifically generates cross-reference data for MySQL databases. Formats the cross-reference data into SQL insert statements.
   - **Usage**: Run the script to create cross-reference SQL files.
 
 - **generate_csv.py**
@@ -36,6 +40,10 @@ This repository contains a collection of scripts designed to generate various fo
   - **Description**: Generates MySQL SQL dump files for Bible translations. Each translation is processed and output as an SQL dump file.
   - **Usage**: Run the script to create SQL dump files for each translation.
 
+- **generate_sqlite.py**
+  - **Description**: Generates SQLite database files for Bible translations. Each translation is processed and output as an SQLite database file.
+  - **Usage**: Run the script to create SQLite database files for each translation.
+
 - **generate_txt.py**
   - **Description**: Generates plain text (TXT) files for Bible translations. Each translation is processed and output as a text file.
   - **Usage**: Run the script to create text files for each translation.

diff --git a/scripts/export_sqlite_database.py b/scripts/export_sqlite_database.py
@@ -0,0 +1,142 @@
+import sys
+import os
+import json
+import sqlite3
+
+def list_options(options, prompt):
+    for i, option in enumerate(options, 1):
+        print(f"{i}. {option}")
+    choice = int(input(prompt)) - 1
+    return options[choice]
+
+def create_sqlite_db(db_path):
+    # Create the database file if it doesn't exist
+    os.makedirs(os.path.dirname(db_path), exist_ok=True)
+    conn = sqlite3.connect(db_path)
+    cursor = conn.cursor()
+    return conn, cursor
+
+def generate_translation_tables(language, translation, source_directory, cursor):
+    json_path = os.path.join(source_directory, language, translation, f"{translation}.json")
+    with open(json_path, 'r', encoding='utf-8') as file:
+        data = json.load(file)
+
+    readme_path = os.path.join(source_directory, language, translation, "README.md")
+    with open(readme_path, 'r', encoding='utf-8') as file:
+        translation_name = file.readline().strip()
+        license_info = "Unknown"
+        for line in file:
+            if line.startswith("**License:**"):
+                license_info = line.split("**License:** ")[1].strip()
+
+    # Create translations table if it doesn't exist
+    cursor.execute("""
+    CREATE TABLE IF NOT EXISTS translations (
+        translation TEXT PRIMARY KEY,
+        title TEXT,
+        license TEXT
+    );
+    """)
+    cursor.execute("""
+    INSERT OR IGNORE INTO translations (translation, title, license)
+    VALUES (?, ?, ?);
+    """, (translation, translation_name, license_info))
+
+    # Create books table
+    cursor.execute(f"""
+    CREATE TABLE IF NOT EXISTS {translation}_books (
+        id INTEGER PRIMARY KEY AUTOINCREMENT,
+        name TEXT
+    );
+    """)
+
+    # Insert books
+    for book in data['books']:
+        cursor.execute(f"INSERT INTO {translation}_books (name) VALUES (?);", (book['name'],))
+
+    # Create verses table
+    cursor.execute(f"""
+    CREATE TABLE IF NOT EXISTS {translation}_verses (
+        id INTEGER PRIMARY KEY AUTOINCREMENT,
+        book_id INTEGER,
+        chapter INTEGER,
+        verse INTEGER,
+        text TEXT,
+        FOREIGN KEY (book_id) REFERENCES {translation}_books(id)
+    );
+    """)
+
+    # Insert verses
+    for book_index, book in enumerate(data['books'], start=1):
+        for chapter in book['chapters']:
+            for verse in chapter['verses']:
+                cursor.execute(f"""
+                INSERT INTO {translation}_verses (book_id, chapter, verse, text)
+                VALUES (?, ?, ?, ?);
+                """, (book_index, chapter['chapter'], verse['verse'], verse['text']))
+
+def generate_cross_references(source_directory, cursor):
+    json_dir = os.path.join(source_directory, 'extras')
+
+    cursor.execute("""
+    CREATE TABLE IF NOT EXISTS cross_references (
+        id INTEGER PRIMARY KEY AUTOINCREMENT,
+        from_book TEXT,
+        from_chapter INTEGER,
+        from_verse INTEGER,
+        to_book TEXT,
+        to_chapter INTEGER,
+        to_verse_start INTEGER,
+        to_verse_end INTEGER,
+        votes INTEGER
+    );
+    """)
+
+    cross_reference_files = [f for f in os.listdir(json_dir) if f.startswith('cross_references') and f.endswith('.json')]
+    for file in cross_reference_files:
+        with open(os.path.join(json_dir, file), 'r', encoding='utf-8') as jsonfile:
+            data = json.load(jsonfile)
+
+        for ref in data['cross_references']:
+            from_verse = ref['from_verse']
+            for to_verse in ref['to_verse']:
+                cursor.execute("""
+                INSERT INTO cross_references (from_book, from_chapter, from_verse, to_book, to_chapter, to_verse_start, to_verse_end, votes)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?);
+                """, (from_verse['book'], from_verse['chapter'], from_verse['verse'], to_verse['book'], to_verse['chapter'], to_verse['verse_start'], to_verse['verse_end'], ref['votes']))
+
+def main():
+    # Set base directories relative to the script location
+    base_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+    source_directory = os.path.join(base_dir, 'sources')
+
+    # Ask user for the path where the new database should be built
+    target_db_path = input("Enter the path where the new SQLite database should be built: ").strip()
+
+    # Step 1: Select Language
+    languages = [d for d in os.listdir(source_directory) if os.path.isdir(os.path.join(source_directory, d)) and d != "extras"]
+    print("Choose your language:")
+    language = list_options(languages, "Enter the number corresponding to your language: ")
+
+    # Step 2: Select Translation
+    translations = [d for d in os.listdir(os.path.join(source_directory, language)) if os.path.isdir(os.path.join(source_directory, language, d))]
+    print(f"Choose your translation for {language}:")
+    translation = list_options(translations, "Enter the number corresponding to your translation: ")
+
+    # Create SQLite database
+    conn, cursor = create_sqlite_db(target_db_path)
+
+    # Generate translation tables
+    generate_translation_tables(language, translation, source_directory, cursor)
+
+    # Generate cross references
+    generate_cross_references(source_directory, cursor)
+
+    # Commit changes and close connection
+    conn.commit()
+    conn.close()
+
+    print("SQLite database with cross references built successfully!")
+
+if __name__ == "__main__":
+    main()