Full extraction notebook (course visualization currently broken)

notebooks/full_extraction/README.md

@@ -0,0 +1,107 @@
+# RTK Full Extraction Pipeline
+
+## Installation
+0. (Optional) Copy this folder to somewhere you'll remember it
+1. Ensure python is installed (highly recommend running from within VSCode)
+2. Create a virtual environment: 
+```bash
+python3 -m venv rtk # Creates a virtual environment called "rtk"
+source rtk/bin/activate # Enters the virtual environment
+pip install -r requirements.txt # Installs the required packages
+```
+3. Install RTKLIB (https://github.com/tomojitakasu/RTKLIB)
+- MacOS users will have to modify some Makefiles and build from source
+- Linux users will have to build from source
+- Windows users can install a prebuilt version (google RTKLIB)
+
+
+## How to use
+1. Download the Hammerschlag Hall Basestaion Data
+2. Open this folder with VSCode
+3. Open `rtk_extraction.ipynb` and edit the fields in the first code section:
+    - BUGGIES: List of rovers that you want to do data analysis on
+    - DATE: Date you want to do data analysis for (format: YYYY-MM-DD)
+    - CLEANUP: If true, deletes all the temp files created during analysis
+    - DEBUG: Adds extra print statements if getting unexpected outputs
+4. Run the first 3 code sections. They will create a bunch of directories to
+   ensure data can be batch processed easily.
+5. Move rover ubx files (firmware uses extension .BIN) to the `raw_ubx` folder
+   corresponding to the date in the corresponding rover directories.
+6. If zipped, extract the basestation data and move it into the basestation_data
+   directory corresponding to the date.
+7. Attempt to run all cells in the notebook. If unsuccessful, ask for help.
+
+## Known issues
+- API for map of buggy course is broken so displaying shows on a white background
+
+### (MacOS users) Installing from source
+1. Edit the below files as per the diff
+2. Run `make && make install` from `(RTKLIB)/app/consapp`
+
+```diff
+diff --git a/app/consapp/convbin/gcc/makefile b/app/consapp/convbin/gcc/makefile
+index 1df0e93..6532465 100644
+--- a/app/consapp/convbin/gcc/makefile
++++ b/app/consapp/convbin/gcc/makefile
+@@ -11,7 +11,7 @@ OPTIONS= -DTRACE -DENAGLO -DENAQZS -DENAGAL -DENACMP -DENAIRN -DNFREQ=3 -DNEXOBS
+
+ CFLAGS = -O3 -ansi -pedantic -Wall -Wno-unused-but-set-variable $(INCLUDE) $(OPTIONS) -g
+
+-LDLIBS = -lm -lrt
++LDLIBS = -lm
+
+ all  : convbin
+
+diff --git a/app/consapp/pos2kml/gcc/makefile b/app/consapp/pos2kml/gcc/makefile
+index 032f309..3874383 100644
+--- a/app/consapp/pos2kml/gcc/makefile
++++ b/app/consapp/pos2kml/gcc/makefile
+@@ -3,7 +3,7 @@
+ BINDIR = /usr/local/bin
+ SRC    = ../../../../src
+ CFLAGS = -Wall -O3 -ansi -pedantic -I$(SRC) -DTRACE
+-LDLIBS  = -lm -lrt
++LDLIBS  = -lm
+
+ pos2kml    : pos2kml.o convkml.o convgpx.o solution.o geoid.o rtkcmn.o preceph.o
+
+diff --git a/app/consapp/rnx2rtkp/gcc/makefile b/app/consapp/rnx2rtkp/gcc/makefile
+index 0e33e2d..e76cf05 100644
+--- a/app/consapp/rnx2rtkp/gcc/makefile
++++ b/app/consapp/rnx2rtkp/gcc/makefile
+@@ -8,7 +8,7 @@ OPTS    = -DTRACE -DENAGLO -DENAQZS -DENAGAL -DENACMP -DENAIRN -DNFREQ=3 -DNEXOB
+
+ # for no lapack
+ CFLAGS  = -Wall -O3 -ansi -pedantic -Wno-unused-but-set-variable -I$(SRC) $(OPTS)
+-LDLIBS  = -lgfortran -lm -lrt
++LDLIBS  = -lm 
+
+ #CFLAGS  = -Wall -O3 -ansi -pedantic -Wno-unused-but-set-variable -I$(SRC) -DLAPACK $(OPTS)
+ #LDLIBS  = -lm -lrt -llapack -lblas
+diff --git a/app/consapp/rtkrcv/gcc/makefile b/app/consapp/rtkrcv/gcc/makefile
+index 89ae61c..b6d7830 100644
+--- a/app/consapp/rtkrcv/gcc/makefile
++++ b/app/consapp/rtkrcv/gcc/makefile
+@@ -7,7 +7,7 @@ CTARGET= -DTRACE -DENAGLO -DENAQZS -DENACMP -DENAGAL -DENAIRN -DNFREQ=3 -DNEXOBS
+ #CTARGET= -DENAGLO -DENAQZS -DENACMP -DENAGAL -DENAIRN -DNFREQ=3 -DIERS_MODEL -DSVR_REUSEADDR
+
+ CFLAGS = -Wall -O3 -ansi -pedantic -Wno-unused-but-set-variable -I$(SRC) -I.. -DTRACE $(CTARGET) -g
+-LDLIBS  = -lm -lrt -lpthread
++LDLIBS  = -lm -lpthread
+ #LDLIBS  = ../../../lib/iers/gcc/iers.a -lm -lrt -lpthread
+
+ all        : rtkrcv
+diff --git a/app/consapp/str2str/gcc/makefile b/app/consapp/str2str/gcc/makefile
+index 3d88200..a02c3a6 100644
+--- a/app/consapp/str2str/gcc/makefile
++++ b/app/consapp/str2str/gcc/makefile
+@@ -9,7 +9,7 @@ CTARGET=
+
+ OPTION = -DENAGLO -DENAGAL -DENAQZS -DENACMP -DENAIRN -DTRACE -DNFREQ=3 -DNEXOBS=3 -DSVR_REUSEADDR
+ CFLAGS = -Wall -O3 -ansi -pedantic -Wno-unused-but-set-variable -I$(SRC) $(OPTION) $(CTARGET) -g
+-LDLIBS  = -lm -lrt -lpthread
++LDLIBS  = -lm -lpthread
+
+ all        : str2str
+ str2str    : str2str.o stream.o rtkcmn.o solution.o sbas.o geoid.o
+```

notebooks/full_extraction/constants.py

@@ -0,0 +1,33 @@
+PARSE_CSV_HEADER = ["timestamp", # TODO change this to "timestamp_str"
+                    "latitude",
+                    "longitude",
+                    "altitude",
+                    "fix-type",
+                    "satellites",
+                    "sdn",
+                    "sde",
+                    "sdu",
+                    "sdne",
+                    "sdeu",
+                    "sdun",
+                    "age",
+                    "ratio"
+]
+
+CALC_CSV_HEADER = [
+    "timestamp", # UTC time (float)
+    "timestamp_str",
+    "fix_type",
+    "latitude",
+    "latitude_processed",
+    "longitude",
+    "longitude_processed",
+    "altitude", # meters
+    "velocity", # 2d vector
+    "speed", # scalar (abs val of vector)
+    "acceleration_vector",
+    "acceleration_magnitude",
+    "horizontal_accuracy",
+    "vertical_accuracy",
+    "speed_accuracy"
+ ]

notebooks/full_extraction/requirements.txt

@@ -0,0 +1,56 @@
+appnope==0.1.4
+asttokens==2.4.1
+comm==0.2.2
+contourpy==1.3.0
+cycler==0.12.1
+debugpy==1.8.6
+decorator==5.1.1
+executing==2.1.0
+fonttools==4.54.1
+geographiclib==2.0
+geopy==2.4.1
+ipykernel==6.29.5
+ipympl==0.9.4
+ipython==8.28.0
+ipython-genutils==0.2.0
+ipywidgets==8.1.5
+jedi==0.19.1
+jupyter_client==8.6.3
+jupyter_core==5.7.2
+jupyterlab_widgets==3.0.13
+kiwisolver==1.4.7
+matplotlib==3.9.2
+matplotlib-inline==0.1.7
+nest-asyncio==1.6.0
+numpy==2.1.1
+packaging==24.1
+pandas==2.2.3
+parso==0.8.4
+pexpect==4.9.0
+pillow==10.4.0
+platformdirs==4.3.6
+plotly==5.24.1
+prompt_toolkit==3.0.48
+psutil==6.0.0
+ptyprocess==0.7.0
+pure_eval==0.2.3
+Pygments==2.18.0
+pynmeagps==1.0.42
+pyparsing==3.1.4
+pyrtcm==1.1.1
+python-dateutil==2.9.0.post0
+pytz==2024.2
+pyubx2==1.2.46
+pyzmq==26.2.0
+scipy==1.14.1
+seaborn==0.13.2
+six==1.16.0
+stack-data==0.6.3
+tenacity==9.0.0
+tornado==6.4.1
+tqdm==4.66.5
+traitlets==5.14.3
+tzdata==2024.2
+utm==0.7.0
+wcwidth==0.2.13
+widgetsnbextension==4.0.13