Change to UUID indexes and use django-postgres-copy

[mauch]

Mostly based on #707 but with changes to refect more recent changes to the pipeline:

• Updated delete_run_raw_sql to deal with UUIDs
• Some minor bugfixes from original implementation in v2.0 Experiment: Add Django PostgreSQL copy support (3/4) #707 (eg. initialise mem_csv in copy_upload_model, make object_id a UUID in CommentModel)
• UUIDs broke the prev and next buttons in the web interface so I fixed these, and made minor changes to the text in the web interface (ID number -> ID)
• Fixed a bug with delete_run_raw_sql where multiple sources using the same tag cause the delete to crash (The fix was to only delete a tag when there are no more database sources referenceing it)
• Updated migrations to install q3c into the SQL database.

1. Need to see if object_id in Comment model needs to be UUID so we can delet using raw SQL on that rather than id
2. The associations_df upload and prep for it might be reverted to behaviour from pre #787 - will have to check and update if necessary
3. Need to set up q3c migrations for the database automatically.
4. Need to merge #800 into this so that vaex doesn't break everything.

[mauch]

Maybe just truncation to a certain number of characters, along with removing hyphens?

Alright. Ill take a closer look on Monday.

[ddobie]

Let's leave the UUID character set stuff as-is for now and I can tweak the details later. I've opened an issue for that: #807.

As discussed yesterday, let's go with a "short" option (5 characters) for runs, images etc. medium (12 characters) for sources and long (15 characters) for measurements.

[ddobie]

Looks good to me at this stage, although I suspect you're still working on things

[ddobie]

Just to double check, the migrations file is automatically generated? So even though the character sets and lengths are hardcoded in here, they're functionally not hardcoded anywhere except where they're defined?

[mauch]

Just to double check, the migrations file is automatically generated? So even though the character sets and lengths are hardcoded in here, they're functionally not hardcoded anywhere except where they're defined?

They are defined here: https://github.com/askap-vast/vast-pipeline/blob/UUID_indexes/vast_pipeline/utils/utils.py#L28

then they are put into the models here (e.g.): https://github.com/askap-vast/vast-pipeline/blob/UUID_indexes/vast_pipeline/models.py#L110

The migrations are created from the models using migrate.py makemigrations so if you change the character set in utils.py you need to update the migrations as well with that command (Ideally you aren't going to be updating the migrations very often!).

And likely only when you start a new database from scratch!

[mauch]

Also -there are some hardcoded source strings in the test .csv files so those would also have to change if you changed the string length.

[ddobie]

Good to go - just a minor typo in the comments