Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2.13.1 to dev #1447

Merged
merged 20 commits into from
Apr 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a74f51a
v2.11.0 (#1181)
pl-buiquang Dec 6, 2022
16a71c1
v2.11.1 (#1198)
laurent-laporte-pro Jan 11, 2023
7b8218d
v2.12.0 (#1224)
pl-buiquang Jan 19, 2023
f0ca022
build: update version to 2.12.1
pl-buiquang Jan 23, 2023
7df40cc
Merge branch 'release_v2.12.1'
laurent-laporte-pro Jan 30, 2023
9c7b597
style: fix mypy error: Unused "type: ignore" comment.
laurent-laporte-pro Jan 30, 2023
5e5165a
build: remove redundant call to `mypy` in GitHub actions.
laurent-laporte-pro Jan 30, 2023
962a33e
style: Reformat the source code and unit tests in accordance with the…
laurent-laporte-pro Feb 2, 2023
d6eae53
build: upgrade Black version in `requirements-dev.txt` and `.github/w…
laurent-laporte-pro Feb 2, 2023
dcf5d14
v2.12.2 (#1347)
laurent-laporte-pro Feb 28, 2023
e7c9ceb
v2.13.0
skamril Mar 9, 2023
c5aefd8
build: prepare next bugfix release v2.13.1 (unreleased)
laurent-laporte-pro Mar 21, 2023
6979e87
fix(desktop): use Antares Solver v8.5 for Antares Web Desktop version…
laurent-laporte-pro Mar 22, 2023
101dd8c
fix(launcher): improved reliability of task state retrieval sent to S…
laurent-laporte-pro Mar 22, 2023
12bfa84
fix(api): show Antares Launcher version in the `/version` end point (…
laurent-laporte-pro Mar 22, 2023
8f55667
fix(desktop): use Antares Solver v8.5.0 for Antares Web Desktop versi…
laurent-laporte-pro Mar 22, 2023
a2d0de0
fix(api): better handling of exception to catch the stacktrace (#1422)
MartinBelthle Mar 23, 2023
35c4fa7
feat: standardize log messsage formats (#1431)
laurent-laporte-pro Mar 29, 2023
425c5f0
docs: update change log to prepare the release
laurent-laporte-pro Apr 5, 2023
e19cae8
build: new hotfix release v2.13.1 (2023-04-11)
laurent-laporte-pro Apr 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion antares-launcher
Submodule antares-launcher updated 36 files
+18 −0 CHANGES.md
+0 −0 __init__.py
+2 −2 antareslauncher/__init__.py
+15 −32 antareslauncher/data_repo/data_repo_tinydb.py
+2 −0 antareslauncher/main.py
+1 −1 antareslauncher/main_option_parser.py
+6 −6 antareslauncher/parameters_reader.py
+0 −78 antareslauncher/remote_environnement/iremote_environment.py
+290 −126 antareslauncher/remote_environnement/remote_environment_with_slurm.py
+77 −22 antareslauncher/remote_environnement/ssh_connection.py
+1 −3 antareslauncher/use_cases/check_remote_queue/check_queue_controller.py
+4 −4 antareslauncher/use_cases/check_remote_queue/slurm_queue_show.py
+1 −1 antareslauncher/use_cases/create_list/study_list_composer.py
+3 −3 antareslauncher/use_cases/kill_job/job_kill_controller.py
+4 −6 antareslauncher/use_cases/launch/launch_controller.py
+3 −3 antareslauncher/use_cases/launch/study_submitter.py
+3 −3 antareslauncher/use_cases/launch/study_zip_uploader.py
+4 −2 antareslauncher/use_cases/retrieve/clean_remote_server.py
+4 −2 antareslauncher/use_cases/retrieve/download_final_zip.py
+4 −2 antareslauncher/use_cases/retrieve/log_downloader.py
+6 −10 antareslauncher/use_cases/retrieve/retrieve_controller.py
+4 −3 antareslauncher/use_cases/retrieve/state_updater.py
+3 −9 antareslauncher/use_cases/retrieve/study_retriever.py
+1 −1 tests/integration/test_integration_check_queue_controller.py
+2 −1 tests/integration/test_integration_job_kill_controller.py
+1 −1 tests/integration/test_integration_launch_controller.py
+7 −9 tests/unit/launcher/test_launch_controller.py
+3 −3 tests/unit/launcher/test_submitter.py
+4 −4 tests/unit/launcher/test_zip_uploader.py
+6 −4 tests/unit/retriever/test_download_final_zip.py
+5 −2 tests/unit/retriever/test_log_downloader.py
+6 −5 tests/unit/retriever/test_retrieve_controller.py
+5 −2 tests/unit/retriever/test_server_cleaner.py
+6 −12 tests/unit/retriever/test_study_retriever.py
+168 −92 tests/unit/test_remote_environment_with_slurm.py
+18 −10 tests/unit/test_ssh_connection.py
4 changes: 2 additions & 2 deletions antarest/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@

# Standard project metadata

__version__ = "2.13.0"
__version__ = "2.13.1"
__author__ = "RTE, Antares Web Team"
__date__ = "2023-03-09"
__date__ = "2023-04-11"
# noinspection SpellCheckingInspection
__credits__ = "(c) Réseau de Transport de l’Électricité (RTE)"

Expand Down
42 changes: 21 additions & 21 deletions antarest/core/core_blueprint.py
Original file line number Diff line number Diff line change
@@ -1,39 +1,26 @@
import logging
import subprocess
from pathlib import Path
from typing import Any, Optional
from typing import Any

from fastapi import APIRouter, Depends
from pydantic import BaseModel

from antarest import __version__
from antarest.core.config import Config
from antarest.core.jwt import JWTUser
from antarest.core.requests import UserHasNotPermissionError
from antarest.core.utils.utils import get_commit_id
from antarest.core.utils.web import APITag
from antarest.core.version_info import VersionInfoDTO, get_commit_id
from antarest.login.auth import Auth
from fastapi import APIRouter, Depends
from pydantic import BaseModel


class StatusDTO(BaseModel):
status: str


class VersionDTO(BaseModel):
version: str
gitcommit: Optional[str]


def create_utils_routes(config: Config) -> APIRouter:
"""
Utility endpoints

Args:
storage_service: study service facade to handle request
config: main server configuration

Returns:

"""
bp = APIRouter()
auth = Auth(config)
Expand All @@ -46,11 +33,24 @@ def health() -> Any:
"/version",
tags=[APITag.misc],
summary="Get application version",
response_model=VersionDTO,
response_model=VersionInfoDTO,
)
def version() -> Any:
return VersionDTO(
version=__version__, gitcommit=get_commit_id(config.resources_path)
def version_info() -> Any:
"""
Returns the current version of the application, along with relevant dependency information.

- `version`: The current version of the application.
- `gitcommit`: The commit ID of the current version's Git repository.
- `dependencies`: A dictionary of dependencies, where the key is
the dependency name and the value is its version number.
"""
from antareslauncher import __version__ as antares_launcher_version
from antarest import __version__ as antarest_version

return VersionInfoDTO(
version=antarest_version,
gitcommit=get_commit_id(config.resources_path),
dependencies={"Antares_Launcher": antares_launcher_version},
)

@bp.get("/kill", include_in_schema=False)
Expand Down
34 changes: 24 additions & 10 deletions antarest/core/logging/utils.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,18 @@
import logging
import logging.config
import os
import re
import uuid
from pythonjsonlogger.jsonlogger import JsonFormatter
from contextvars import ContextVar, Token
from typing import Optional, Type, Any, Dict
from typing import Any, Dict, Optional, Type

from antarest.core.config import Config
from starlette.middleware.base import (
BaseHTTPMiddleware,
RequestResponseEndpoint,
)
from starlette.requests import Request
from starlette.responses import Response

from antarest.core.config import Config

_request: ContextVar[Optional[Request]] = ContextVar("_request", default=None)
_request_id: ContextVar[Optional[str]] = ContextVar(
"_request_id", default=None
Expand All @@ -41,11 +38,29 @@ def configure_logger(config: Config) -> None:
"formatters": {
"console": {
"class": "antarest.core.logging.utils.CustomDefaultFormatter",
"format": "%(asctime)s - %(trace_id)s - %(threadName)s - %(name)s - %(ip)s - %(user)s - %(pid)s - %(levelname)s - %(message)s",
"format": (
"[%(asctime)s] [%(process)s] [%(name)s]"
" - %(trace_id)s"
" - %(threadName)s"
" - %(ip)s"
" - %(user)s"
" - %(levelname)s"
" - %(message)s"
),
},
"json": {
"class": f"{JsonFormatter.__module__}.{JsonFormatter.__name__}",
"format": "%(asctime)s - %(trace_id)s - %(threadName)s - %(name)s - %(ip)s - %(user)s - %(pid)s - %(levelname)s - %(message)s",
"class": "pythonjsonlogger.jsonlogger.JsonFormatter",
"format": (
"%(asctime)s"
" - %(process)s"
" - %(name)s"
" - %(trace_id)s"
" - %(threadName)s"
" - %(ip)s"
" - %(user)s"
" - %(levelname)s"
" - %(message)s"
),
},
},
"handlers": {
Expand Down Expand Up @@ -110,9 +125,8 @@ def filter(self, record: logging.LogRecord) -> bool:
request: Optional[Request] = _request.get()
request_id: Optional[str] = _request_id.get()
if request is not None:
record.ip = request.scope.get("client", ("undefined"))[0]
record.ip = request.scope.get("client", "undefined")[0]
record.trace_id = request_id
record.pid = os.getpid()
return True


Expand Down
46 changes: 2 additions & 44 deletions antarest/core/utils/utils.py
Original file line number Diff line number Diff line change
@@ -1,37 +1,17 @@
import logging
import os
import shutil
import subprocess
import tempfile
import time
from glob import escape
from pathlib import Path
from typing import (
IO,
Any,
Optional,
Callable,
TypeVar,
List,
Union,
Awaitable,
Tuple,
)
from zipfile import (
ZipFile,
BadZipFile,
ZIP_STORED,
ZIP_DEFLATED,
ZIP_BZIP2,
ZIP_LZMA,
)
from typing import IO, Any, Callable, List, Optional, Tuple, TypeVar
from zipfile import ZIP_DEFLATED, BadZipFile, ZipFile

import redis

from antarest.core.config import RedisConfig
from antarest.core.exceptions import BadZipBinary, ShouldNotHappenException


logger = logging.getLogger(__name__)


Expand Down Expand Up @@ -105,28 +85,6 @@ def get_local_path() -> Path:
return filepath


def get_commit_id(path_resources: Path) -> Optional[str]:
commit_id = None

path_commit_id = path_resources / "commit_id"
if path_commit_id.exists():
commit_id = path_commit_id.read_text()[:-1]
else:
command = "git log -1 HEAD --format=%H"
process = subprocess.run(command, stdout=subprocess.PIPE, shell=True)
if process.returncode == 0:
commit_id = process.stdout.decode("utf-8")

if commit_id is not None:

def remove_carriage_return(value: str) -> str:
return value[:-1]

commit_id = remove_carriage_return(commit_id)

return commit_id


def new_redis_instance(config: RedisConfig) -> redis.Redis: # type: ignore
redis_client = redis.Redis(
host=config.host,
Expand Down
45 changes: 45 additions & 0 deletions antarest/core/version_info.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
"""
Python module that is dedicated to printing application version and dependencies information
"""
import subprocess
from pathlib import Path
from typing import Dict, Optional

from pydantic import BaseModel


class VersionInfoDTO(BaseModel):
name: str = "AntaREST"
version: str
gitcommit: str
dependencies: Dict[str, str]


def get_commit_id(resources_dir: Path) -> str:
"""
Returns the contents of the file :file:`resources/commit_id`
if it exists and is not empty, or the commit ID of the current Git HEAD, if available.
If neither the commit ID nor the file is available, returns "".

Note:
The :file:`commit_id` is generated during the "deploy" stage
in the :file:`.github/workflows/deploy.yml` GitHub workflow.

Args:
resources_dir: The path to the ``resources`` directory.

Returns:
The contents of the file :file:`resources/commit_id`,
the commit ID of the current Git HEAD, or "".
"""
path_commit_id = resources_dir.joinpath("commit_id")
try:
return path_commit_id.read_text(encoding="utf-8").strip()
except FileNotFoundError:
command = "git log -1 HEAD --format=%H"
try:
return subprocess.check_output(
command, encoding="utf-8", shell=True
).strip()
except (subprocess.CalledProcessError, FileNotFoundError):
return ""
4 changes: 2 additions & 2 deletions antarest/eventbus/business/redis_eventbus.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ def get_events(self) -> List[Event]:
event = self.pubsub.get_message(ignore_subscribe_messages=True)
if event is not None:
return [Event.parse_raw(event["data"])]
except Exception as e:
logger.error("Failed to retrieve or parse event !", exc_info=e)
except Exception:
logger.error("Failed to retrieve or parse event !", exc_info=True)

return []

Expand Down
Loading