Skip to content

Name files of serialized arguments with uuid. #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ releases are available on [PyPI](https://pypi.org/project/pytask-julia) and

## 0.x.x - 2024-xx-xx

- {pull}`32` uses trusted publishing for PyPI.
- {pull}`31` uses pixi to provision Julia if possible.
- {pull}`32` uses trusted publishing for PyPI.
- {pull}`33` uses uuid4 to generate more robust file names for serialized arguments.

## 0.4.0 - 2023-10-08

Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,20 +31,20 @@ You also need to have Julia installed and `julia` on your command line. Test it
typing the following on the command line

```console
$ julia -h
julia -h
```

If an error is shown instead of a help page, you can install Julia on Unix systems with

```console
$ conda install -c conda-forge julia
conda install -c conda-forge julia
```

or choose one of the installers on this [page](https://julialang.org/downloads/).

## Usage

To create a task which runs a Julia script, define a task function with the
To create a task that runs a Julia script, define a task function with the
`@pytask.mark.julia` decorator. The `script` keyword provides a path relative to the
task module to the Julia script.

Expand Down Expand Up @@ -111,11 +111,11 @@ config["number"] # Is 1.
### Debugging

In case a task throws an error, you might want to execute the script independently from
pytask. After a failed execution, you see the command which executed the Julia script in
pytask. After a failed execution, you see the command that executed the Julia script in
the report of the task. It looks roughly like this

```console
$ julia <options> -- script.jl <path-to>/.pytask/task_py_task_example.json
julia <options> -- script.jl <path-to>/.pytask/pytask-julia/<uuid>.json
```

### Managing Julia environments
Expand Down Expand Up @@ -150,7 +150,7 @@ def task_run_jl_script():

### Command Line Options

Command line options can be pass via the `options` keyword argument.
Command line options can be passed via the `options` keyword argument.

```python
@task(kwargs={"path": Path("out.csv")})
Expand All @@ -159,7 +159,7 @@ def task_run_jl_script():
pass
```

This example will execute the script using to threads.
This example will execute the script using threads.

### Repeating tasks with different scripts or inputs

Expand Down Expand Up @@ -225,8 +225,8 @@ config = YAML.load_file(ARGS[1])

Note that the `YAML` package needs to be installed.

If you need a custom serializer, you can also provide any callable to `serializer` which
transforms data to a string. Use `suffix` to set the correct file ending.
If you need a custom serializer, you can also provide any callable for `serializer`
which transforms data into a string. Use `suffix` to set the correct file ending.

Here is a replication of the JSON example.

Expand Down Expand Up @@ -255,7 +255,7 @@ julia_serializer = "json"
**`julia_suffix`**

Use this option to set the default suffix of the file which contains serialized paths to
dependencies and products and more.
dependencies, products and more.

```toml
[tool.pytask.ini_options]
Expand Down
24 changes: 6 additions & 18 deletions src/pytask_julia/serialization.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@
from __future__ import annotations

import json
import uuid
from pathlib import Path
from typing import Any
from typing import Callable

from pytask import PTask
from pytask import PTaskWithPath

__all__ = ["create_path_to_serialized", "serialize_keyword_arguments", "SERIALIZERS"]

_HIDDEN_FOLDER = ".pytask/pytask-julia"


Expand All @@ -28,25 +31,10 @@

def create_path_to_serialized(task: PTask, suffix: str) -> Path:
"""Create path to serialized."""
parent = task.path.parent if isinstance(task, PTaskWithPath) else Path.cwd()
file_name = create_file_name(task, suffix)
return parent.joinpath(_HIDDEN_FOLDER, file_name).with_suffix(suffix)


def create_file_name(task: PTask, suffix: str) -> str:
"""Create the file name of the file containing the serialized kwargs.

Some characters need to be escaped since they are not valid characters on file
systems.

"""
return (
task.name.replace("[", "_")
.replace("]", "_")
.replace("::", "_")
.replace(".", "_")
.replace("/", "_")
+ suffix
(task.path.parent if isinstance(task, PTaskWithPath) else Path.cwd())
.joinpath(_HIDDEN_FOLDER, str(uuid.uuid4()))
.with_suffix(suffix)
)


Expand Down