DRAFT: Autogenerating Seed Data Script from Data with Django

Overview

Once you create your models in Django, you have various options on how to generate seed data. This article describes how to autogenerate seed data scripts from data that has been entered into a database. This presumes you have a database with the models created that you will use as the source.

Setup

These are general steps. Your application may have more specific steps. The steps presume you have set up the development environment for the app.

• Setup (activate) python env
• Identify directory for saving JSON files. The People Depot app uses core/initial_data.
• If applicable, run script from terminal to set up dev environment settings that at a minimum sets DJANGO_SETTINGS_MODULE. This should point to a script that sets up your database. It may already be incorporated into your scripts. Here is an example of a script:

#!/bin/bash
(return 0 2>/dev/null) && sourced="true" || sourced="false"
if [ "$sourced" != "true" ]; then
    echo "Error, script not sourced.  Please run 'source ./set-dev-settings.sh'"
    exit 1
fi
export DJANGO_SETTINGS_MODULE=DjangoRestApisPostgreSQL.dev_settings

Create JSON file with dumpdata

The Django dumpdata commands dumps data from your selected database into a JSON formatted data. You can then use those files in a script that uses the files.

Autogenerate JSON

You can use any of these commands to autogenerate JSON:

python3 manage.py dumpdata <app_name>.<model_name> > <filename>.json
python3 manage.py dumpdata <app_name> > <filename>.json
python3 manage.py dumpdata > <filename>.json

Examples:

python3 manage.py dumpdata knowledgebase.gdocs > fixtures/knowledgebase-gdocs.json
python3 manage.py dumpdata auth > auth.json
python3 manage.py dumpdata > alldata.json 

Here's an example for what data for people model might look like in JSON:

[
  {
    "model": "myapp.person",
    "pk": 1,
    "fields": {
      "first_name": "John",
      "last_name": "Snow"
    }
  },
  {
    "model": "myapp.person",
    "pk": 2,
    "fields": {
      "first_name": "Paul",
      "last_name": "McCartney"
    }
  }
]

Autogenerate Security Groups and Permissions (Authentication and Authorization) JSON

python3 manage.py dumpdata auth > <json data directory>/auth.json

Edit JSON file

• Format document**: Use your editor's autoformat for better readability.
• Remove extraneous non-json**: The JSON file may have output that is generated before the JSON data.
• Remove data for models if you don't want the data to be included.

Methods for Loading Data Using Django loaddata Command

The Django loaddata command populates a data model based on a JSON file generated with dumpddata or a JSON file with similar format.

Loading Data from Shell Script

The following is an example script for loading seeddata.

source scripts/set_dev.sh
python3 manage.py loaddata <json data directory>/bookTypes.json
python3 manage.py loaddata <json data directory>/emplyeeTypes.json

To run this script multiple times without giving an error message, add code to check if data already exists before running loadddata.

Create Sequenced Migration Script

Create a new sequenced load script in the models directory. The script name should start with a sequence that comes after the last script. For example,if the last script is 0007_person.py and you want a script to load books data, you could call the script 0008_loadBookTypes.py. Django will keep track if you have executed the script. If you are in initial stages of development and regenerating the model scripts. you will need to make sure not to delete these scripts.

Example code:

Script name: 0008_loadBookTypes.py

from django.db import migrations

def loadJsonData(app, schema_editor):
    print("Loading JSON data for book rypwa")
    python3 manage.py loaddata <path to json data file>/book.json

    
class Migration(migrations.Migration):


    dependencies = [
        ("knowledgebase", "0006_book"),
    ]

    operations = [
        migrations.RunPython(loadJsonData),
    ]

Resources

• django 3.1 docs on initializing data
• Automating Data Seed for Tests