Skip to content

Commit be64e88

Browse files
BrainIsDeadBrainIsDead
committed
The changes made in this commit include:
- Modified `data_import.rst` to include information about importing data from the admin interface using the Django Import-Export library. - Added `deployment.rst` file to the `src/docs/source` directory. - Modified `deployment.rst` to include information about the deployment process for the application. - Added `index.rst` file to the `src/docs/source` directory. - Modified `index.rst` to include references to the newly added documentation files. - Added `modules.rst` file to the `src/docs/source` directory. - Modified `modules.rst` to include references to the different modules in the Signal - Added `overview.rst` file to the `src/docs/source` directory. - Modified `overview.rst` to include an overview of the Signal Documentation system, installation instructions, and basic usage information. Here is a summary of the changes made in this commit: - Added `data_import.rst` file to provide information about importing data from the admin interface using the Django Import-Export library. - Added `deployment.rst` file to provide information about the deployment process for the application. - Added `index.rst` file to serve as the main index for the documentation and include references to the newly added documentation files. - Added `modules.rst` file to provide references to the different modules in the Signal Documentation system. - Added `overview.rst` file to provide an overview of the Signal Documentation system, installation instructions, and basic usage information. Please let me know if you need any further assistance.
1 parent c9a5b05 commit be64e88

File tree

6 files changed

+291
-2
lines changed

6 files changed

+291
-2
lines changed

‎src/docs/source/_static/import_confirm_signals_base.png

111 KB
Loading

‎src/docs/source/data_import.rst

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
Import data from admin interface
2+
================================
3+
4+
For data import used
5+
`django-import-export <https://django-import-export.readthedocs.io/en/latest/index.html>`__
6+
library
7+
8+
The code consists of Django resource classes that are used for importing
9+
and exporting CSV files using the Django Import-Export library:
10+
11+
Resources
12+
---------
13+
14+
1. ``SignalResource`` - this resource class is used for importing ``Signal`` models:
15+
16+
- Defines various fields such as ``name``, ``display_name``, ``pathogen``, ``signal_type``, and ``source``.
17+
- Defines the ``before_import_row`` method, which is called before importing each row and allows for pre-processing of the data.
18+
- Includes methods like ``is_url_in_domain`` to check if a URL belongs to a specific domain, ``fix_boolean_fields`` to handle boolean fields, and ``process_links`` to process the links field.
19+
20+
2. ``SignalBaseResource`` - This resource class is used for updating
21+
already created ``Signal`` models ``base`` fields with base Signals:
22+
23+
- Defines various fields such as ``name``, ``display_name``,
24+
``base``, and ``source``.
25+
- Defines the ``before_import_row`` method, which is called before
26+
importing each row and allows for pre-processing of the data.
27+
- The ``process_base`` method is responsible for processing the
28+
``base`` field by retrieving the corresponding ``Signal`` object
29+
based on the provided ``name`` and ``source``.
30+
31+
3. ``SourceSubdivisionResource`` - this resource class is used for
32+
importing ``SourceSubdivision`` models:
33+
34+
- It defines fields such as ``name``, ``display_name``,
35+
``description``, ``data_source``, and ``links``.
36+
- It includes the ``before_import_row`` method for pre-processing
37+
each row before importing.
38+
- The ``process_links`` method is responsible for processing the
39+
``links`` field by creating ``Link`` objects based on the provided
40+
URLs.
41+
- The ``process_datasource`` method processes the ``data_source``
42+
field by creating or retrieving a ``DataSource`` object based on
43+
the provided name.
44+
45+
These resource classes provide a structured way to import CSV files.
46+
They define the fields, handle pre-processing of data, and interact with
47+
the corresponding models and related objects.
48+
49+
Import data flow
50+
----------------
51+
52+
CSV preparation
53+
~~~~~~~~~~~~~~~
54+
55+
To import data from a CSV file must meet the requirements:
56+
- CSV file should be properly formatted and contains all the required fields for
57+
importing, as specified by the resource classes (``SignalResource``, ``SignalBaseResource``, ``SourceSubdivisionResource``).
58+
- The header rowof the CSV file should match the field names defined in the resource classes.
59+
- It should not contain empty rows (empty rows may cause validation errors during the import)
60+
- Colums should be saparateb by ``","``
61+
62+
Othervice you will receive Errors during import process:
63+
64+
|Import errors|
65+
66+
Data import
67+
~~~~~~~~~~~
68+
69+
1. Import ``SourceSubdivision`` instances with
70+
``SourceSubdivisionResource`` -
71+
http://localhost:8000/admin/datasources/sourcesubdivision/import/
72+
|Import ``SourceSubdivision`` instances| |Confirm importing
73+
``SourceSubdivision`` instances|
74+
75+
76+
2. Import ``Signal`` instances with ``SignalResource`` -
77+
http://localhost:8000/admin/signals/signal/import/ |Import ``Signal``
78+
instances| |Confirm importing ``Signal`` instances|
79+
80+
81+
3. Import ``Signal.base`` fields with ``SignalBaseResource`` -
82+
http://localhost:8000/admin/signals/signal/import/ |Import
83+
``Signal.base`` field| |Confirm importing ``Signal.base`` fields|
84+
85+
.. |Import errors| image:: ./_static/import_errors.png
86+
.. |Import ``SourceSubdivision`` instances| image:: ./_static/import_source_subdivision.png
87+
.. |Confirm importing ``SourceSubdivision`` instances| image:: ./_static/import_confirm_source_subdivision.png
88+
.. |Import ``Signal`` instances| image:: ./_static/import_signals.png
89+
.. |Confirm importing ``Signal`` instances| image:: ./_static/import_confirm_signals.png
90+
.. |Import ``Signal.base`` field| image:: ./_static/import_signals_base.png
91+
.. |Confirm importing ``Signal.base`` fields| image:: ./_static/import_confirm_signals_base.png

‎src/docs/source/deployment.rst

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
Deployment
2+
==========
3+
4+
This application gets deployed (at a minimum) to two environmetns:
5+
6+
Production -
7+
`https://delphi.cmu.edu/{app_name} <https://delphi.cmu.edu/{app_name}>`__
8+
9+
Staging -
10+
`https://staging.delphi.cmu.edu/{app_name} <https://staging.delphi.cmu.edu/{app_name}>`__
11+
12+
Each environment is essentially a bunch of different services all
13+
governed by ``docker-compose``, running across multiple hosts, with
14+
various layering of proxies and load balancers.
15+
16+
Basic workflow
17+
--------------
18+
19+
- A PR merged to either ``development`` or ``master`` will trigger CI
20+
to build container images that are then tagged (based on the branch
21+
name and ":latest" respectively) and stored in our GitHub Packages
22+
container image repository.
23+
- CI triggers a webhook that tells the host systems to pull and run new
24+
container images and restart any services that have been updated.
25+
26+
Control of the deployed environment
27+
-----------------------------------
28+
29+
The environment and secrets used for deployment live in
30+
https://github.com/cmu-delphi/delphi-ansible-web. Any changes to the
31+
environment should be made there and then tested and validated by devops
32+
folks.

‎src/docs/source/index.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ Welcome to Signal Documentation's documentation!
1010
:maxdepth: 2
1111
:caption: Contents:
1212

13+
overview
14+
data_import
15+
deployment
1316
modules
1417

1518
Indices and tables

‎src/docs/source/modules.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
src
2-
====
1+
Signal Documentation Modules
2+
============================
33

44
.. toctree::
55
:maxdepth: 4

‎src/docs/source/overview.rst

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
Overview and installation
2+
=========================
3+
4+
Single Source of Documentation System
5+
6+
.. image:: ./_static/signal_documentation_schema.jpg
7+
:alt: Signal Documentation Schema
8+
9+
10+
Entity Relationship Diagram
11+
---------------------------
12+
13+
.. image:: ./_static/models.png
14+
:alt: Signal Documentation ERD
15+
16+
Core libs and DB
17+
----------------
18+
19+
1. `Django <https://www.djangoproject.com/>`__
20+
2. `Django-filter <https://django-filter.readthedocs.io/en/stable/index.html>`__
21+
3. `PostgreSQL <https://www.postgresql.org/>`__
22+
23+
All requirements you can find in ``Pipfile``
24+
25+
Getting started
26+
---------------
27+
28+
Setup Env Vars
29+
~~~~~~~~~~~~~~
30+
31+
create ``.env`` file and add variables like in ``.env.example``
32+
33+
To run locally
34+
~~~~~~~~~~~~~~
35+
36+
Install ``python:3.10``, ``pip3``, ``pipenv``
37+
38+
Using `pipenv <https://github.com/pypa/pipenv>`__ run ``pipenv shell``
39+
and ``pipenv install`` to create virtual environment and install
40+
dependencies
41+
42+
.. code:: sh
43+
44+
$ pipenv shell
45+
$ pipenv install
46+
47+
Go to ``src`` directory and run
48+
49+
.. code:: sh
50+
51+
$ python manage.py migrate
52+
$ python manage.py test
53+
$ python manage.py runserver
54+
55+
load fixtures
56+
57+
.. code:: sh
58+
59+
$ python manage.py loaddata .\fixtures\available_geography.json
60+
$ python manage.py loaddata .\fixtures\pathogens.json
61+
$ python manage.py loaddata .\fixtures\signal_categories.json
62+
$ python manage.py loaddata .\fixtures\signal_types.json
63+
64+
if you need test coverage
65+
66+
.. code:: sh
67+
68+
$ coverage erase
69+
$ coverage python manage.py test
70+
$ coverage report
71+
72+
you can also get test coverage with one command
73+
74+
.. code:: sh
75+
76+
$ coverage erase && coverage run manage.py test && coverage report
77+
78+
sort imports
79+
80+
.. code:: sh
81+
82+
$ isort .
83+
84+
check flake
85+
86+
.. code:: sh
87+
88+
$ flake8 --show-source
89+
90+
To run via docker
91+
~~~~~~~~~~~~~~~~~
92+
93+
Install ``Docker`` and ``docker-compose``
94+
95+
Run
96+
97+
.. code:: sh
98+
99+
$ docker-compose build
100+
$ docker-compose up
101+
102+
Open ``http://localhost:8000`` to view it in the browser
103+
104+
To run via docker and emulate production
105+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
106+
107+
Though probably not necessary in most cases, if you want to
108+
test/modify/emulate how this will run in production you can:
109+
110+
- In ``.env`` set:
111+
112+
::
113+
114+
DEBUG = 'False'
115+
116+
- Modify the app container's command in ``docker-compose.yaml`` to run:
117+
118+
::
119+
120+
"gunicorn signal_documentation.wsgi:application --bind 0.0.0.0:8000"
121+
122+
*(Essentially you'll replace just the last line of the command, switching out the "runserver" line)
123+
124+
Open ``http://localhost`` to view it in the browser. In this usage your
125+
request will be serviced by Nginx instead of the application directly.
126+
127+
The primary use case for this will be when making changes to the Nginx
128+
container image that runs in production and hosts the static file
129+
content, or also if making changes to the Gunicorn config.
130+
131+
Changes of this sort should be carefully evaluated as they may require
132+
interaction with systems managed by devops folks.
133+
134+
135+
`Django admin <https://docs.djangoproject.com/en/4.1/ref/contrib/admin/>`__ web interface
136+
-----------------------------------------------------------------------------------------
137+
Django admin is a web interface for managing the project web application. It is available at the following URL.
138+
139+
``http://localhost:8000/admin``
140+
141+
The user should have ``is_staff`` or ``is_superuser`` permissions to access the admin interface.
142+
143+
144+
Read the docs (Sphynx)
145+
----------------------
146+
Auto generated documentation for the project web appplication is available at the following URL.
147+
148+
``http://localhost:8000/<MAIN_PAGE>/docs/index.html``
149+
150+
To clean the documentation, run the following commands:
151+
152+
.. code:: sh
153+
154+
$ cd ./docs
155+
$ make clean
156+
157+
158+
To generate the documentation, run the following commands:
159+
160+
.. code:: sh
161+
162+
$ cd ./docs
163+
$ make html

0 commit comments

Comments
 (0)