Userland connections

.gitignore

@@ -22,3 +22,5 @@ docs/_build/
 .env
 tst
 tst2
+user_dbs/*
+tmp2

LICENSE

@@ -1,6 +1,11 @@
-The MIT License (MIT)
+* All content that resides under the "explorer/ee/" directory of this repository is licensed under the license defined
+in "explorer/ee/LICENSE".
 
-Copyright (c) 2013 Chris Clark, ePantry LLC
+* Content outside of the above mentioned directory is provided under the "MIT" license as defined below.
+
+** The MIT License (MIT) **
+
+Copyright (c) 2024, SQL Explorer, Inc
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +23,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

docs/features.rst

@@ -3,19 +3,16 @@ Features
 
 Security
 --------
-- Let's not kid ourselves - this tool is all about giving people
-  access to running SQL in production. So if that makes you
-  nervous (**and it should**) - you've been warned. Explorer makes an
-  effort to not allow terrible things to happen, but be careful!
-  It's recommended you setup read-only roles for each of your database
+- It's recommended you setup read-only roles for each of your database
   connections and only use these particular connections for your queries through the
-  ``EXPLORER_CONNECTIONS`` setting.
-- Explorer supports two different permission checks for users of
+  ``EXPLORER_CONNECTIONS`` setting -- or set up userland connections via DatabaseConnections in
+  the Django admin, or the SQL Explorer front-end.
+- SQL Explorer supports two different permission checks for users of
   the tool. Users passing the ``EXPLORER_PERMISSION_CHANGE`` test can
   create, edit, delete, and execute queries. Users who do not pass
   this test but pass the ``EXPLORER_PERMISSION_VIEW`` test can only
   execute queries. Other users cannot access any part of
-  Explorer. Both permission groups are set to is_staff by default
+  SQL Explorer. Both permission groups are set to is_staff by default
   and can be overridden in your settings file.
 - Enforces a SQL blacklist so destructive queries don't get
   executed (delete, drop, alter, update etc). This is not
@@ -37,19 +34,25 @@ SQL Assistant
   to quickly get help with your query, with relevant schema
   automatically injected into the prompt.
 
+Configurable Connections
+------------------------
+- Configure connections via the settings.py file, or via the SQL Explorer UI.
+- Supports drag-and-drop uploading of CSV files or SQLite databases for instant SQL access to your data.
+
 Snapshots
 ---------
 - Tick the 'snapshot' box on a query, and Explorer will upload a
   .csv snapshot of the query results to S3. Configure the snapshot
-  frequency via a celery cron task, e.g. for daily at 1am:
+  frequency via a celery cron task, e.g. for daily at 1am
+  (see test_project/celery_config.py for an example of this, along with test_project/__init__.py):
 
 .. code-block:: python
 
     app.conf.beat_schedule = {
-       'explorer.tasks.snapshot_queries': {
-           'task': 'explorer.tasks.snapshot_queries',
-           'schedule': crontab(hour=1, minute=0)
-       }
+       "explorer.tasks.snapshot_queries": {
+            "task": "explorer.tasks.snapshot_queries",
+            "schedule": crontab(hour="1", minute="0")
+        },
     }
 
 - Requires celery, obviously. Also uses boto3. All
@@ -64,6 +67,7 @@ Email query results
 - Click the email icon in the query listing view, enter an email
   address, and the query results (zipped .csv) will be sent to you
   asynchronously. Very handy for long-running queries.
+- You must also have the setting ``EXPLORER_TASKS_ENABLED`` enabled.
 
 Parameterized Queries
 ---------------------
@@ -168,10 +172,10 @@ Query Logs
 .. code-block:: python
 
    app.conf.beat_schedule = {
-       'explorer.tasks.truncate_querylogs': {
-           'task': 'explorer.tasks.truncate_querylogs',
-           'schedule': crontab(hour=1, minute=0),
-           'kwargs': {'days': 30}
+       "explorer.tasks.truncate_querylogs": {
+           "task": "explorer.tasks.truncate_querylogs",
+           "schedule": crontab(hour="1", minute="10"),
+           "kwargs": {"days": 30}
        }
    }
 

docs/settings.rst

@@ -357,4 +357,13 @@ The following three settings control the SQL Assistant. More information is avai
 
     EXPLORER_AI_API_KEY = getattr(settings, "EXPLORER_AI_API_KEY", None)
     EXPLORER_ASSISTANT_BASE_URL = getattr(settings, "EXPLORER_ASSISTANT_BASE_URL", "https://api.openai.com/v1")
-    EXPLORER_ASSISTANT_MODEL
+    EXPLORER_ASSISTANT_MODEL = getattr(settings, "EXPLORER_ASSISTANT_MODEL",
+                                   # Return the model name and max_tokens it supports
+                                   {"name": "gpt-4o",
+                                    "max_tokens": 128000})
+
+
+User-Configured DB Connections
+******************************
+Set `EXPLORER_DB_CONNECTIONS_ENABLED` to `True` to enable DB connections to get configured in the browser (e.g. not
+just in settings.py). This also allows uploading of CSV or SQLite files for instant querying.

explorer/admin.py

@@ -2,6 +2,7 @@
 
 from explorer.actions import generate_report_action
 from explorer.models import Query, ExplorerValue
+from explorer.ee.db_connections.admin import DatabaseConnectionAdmin  # noqa
 
 
 @admin.register(Query)
@@ -16,12 +17,10 @@ class QueryAdmin(admin.ModelAdmin):
 class ExplorerValueAdmin(admin.ModelAdmin):
     list_display = ("key", "value", "display_key")
     list_filter = ("key",)
-    readonly_fields = ("key",)
     search_fields = ("key", "value")
 
     def display_key(self, obj):
         # Human-readable name for the key
         return dict(ExplorerValue.EXPLORER_SETTINGS_CHOICES).get(obj.key, "")
 
     display_key.short_description = "Setting Name"
-

explorer/app_settings.py

@@ -69,8 +69,11 @@
 EXPLORER_PERMISSION_CHANGE = getattr(
     settings, "EXPLORER_PERMISSION_CHANGE", lambda r: r.user.is_staff
 )
+EXPLORER_PERMISSION_CONNECTIONS = getattr(
+    settings, "EXPLORER_PERMISSION_CONNECTIONS", lambda r: r.user.is_staff
+)
 EXPLORER_RECENT_QUERY_COUNT = getattr(
-    settings, "EXPLORER_RECENT_QUERY_COUNT", 10
+    settings, "EXPLORER_RECENT_QUERY_COUNT", 5
 )
 EXPLORER_ASYNC_SCHEMA = getattr(settings, "EXPLORER_ASYNC_SCHEMA", False)
 
@@ -125,7 +128,7 @@
 S3_REGION = getattr(settings, "EXPLORER_S3_REGION", "us-east-1")
 S3_ENDPOINT_URL = getattr(settings, "EXPLORER_S3_ENDPOINT_URL", None)
 S3_DESTINATION = getattr(settings, "EXPLORER_S3_DESTINATION", "")
-S3_SIGNATURE_VERSION = getattr(settings, "EXPLORER_S3_SIGNATURE_VERSION", "v2")
+S3_SIGNATURE_VERSION = getattr(settings, "EXPLORER_S3_SIGNATURE_VERSION", "v4")
 
 UNSAFE_RENDERING = getattr(settings, "EXPLORER_UNSAFE_RENDERING", False)
 
@@ -146,8 +149,17 @@
 
 # AI Assistant settings. Setting the first to an OpenAI key is the simplest way to enable the assistant
 EXPLORER_AI_API_KEY = getattr(settings, "EXPLORER_AI_API_KEY", None)
+
 EXPLORER_ASSISTANT_BASE_URL = getattr(settings, "EXPLORER_ASSISTANT_BASE_URL", "https://api.openai.com/v1")
 EXPLORER_ASSISTANT_MODEL = getattr(settings, "EXPLORER_ASSISTANT_MODEL",
                                    # Return the model name and max_tokens it supports
-                                   {"name": "gpt-4-0125-preview",
+                                   {"name": "gpt-4o",
                                     "max_tokens": 128000})
+
+EXPLORER_DB_CONNECTIONS_ENABLED = getattr(settings, "EXPLORER_DB_CONNECTIONS_ENABLED", False)
+EXPLORER_PRUNE_LOCAL_UPLOAD_COPY_DAYS_INACTIVITY = getattr(settings,
+                                                           "EXPLORER_PRUNE_LOCAL_UPLOAD_COPY_DAYS_INACTIVITY", 7)
+
+
+def has_assistant(): return EXPLORER_AI_API_KEY is not None
+def db_connections_enabled(): return EXPLORER_DB_CONNECTIONS_ENABLED

explorer/apps.py

@@ -28,8 +28,9 @@ def _get_explorer_connections():
 
 def _validate_connections():
 
-    # Validate connections
-    if _get_default() not in _get_explorer_connections().values():
+    # Validate connections, when using settings.EXPLORER_CONNECTIONS
+    # Skip if none are configured, as the app will use user-configured connections (DatabaseConnection models)
+    if _get_explorer_connections().values() and _get_default() not in _get_explorer_connections().values():
         raise ImproperlyConfigured(
             f"EXPLORER_DEFAULT_CONNECTION is {_get_default()}, "
             f"but that alias is not present in the values of "