feat: ROOT-11: Support reading JSONL from source cloud storages #7555

matt-bernstein · 2025-05-16T17:23:13Z

[x] Factor out JSON parsing and validation logic from import storages into a function that can be hot-swapped in LSO/LSE
[x] add it to settings as an app-specific variable
[x] replace stdlib parsing with pyarrow parsing to handle JSONL as well as JSON
[x] test coverage for JSONL
[x] feature flag
[x] unskip multitask import tests in LSO, to prepare for testing different behavior in LSE and LSO

Implementation notes:

had to move computation of row_index and row_group into get_data so that load_tasks_json, which will be swapped in LSO/LSE, will have the correct type in all cases - makes for a pretty large diff
simplified error messages significantly since the relevant logic is shared/deduped
had to stick with stdlib json parsing instead of removing it entirely:
- in the case of single dict, pyarrow can't distinguish between this and a one-line JSONL file, so need stdlib json to set row_index=None
- in the case of top-level list, pyarrow can't parse this at all since it assumes each line is an entry in a table. There is an open issue for this.
- consider peeking at file extensions to avoid double-parsing (for clarity, not speed) - see TODO comment

…k to task within file of tasks on cloud storage

Co-authored-by: Jo Booth <[email protected]>

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15027678115

… cloud storage

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15030860401

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15045532193

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15045586529

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15052428789

netlify · 2025-05-16T17:23:18Z

✅ Deploy Preview for label-studio-docs-new-theme canceled.

Name	Link
🔨 Latest commit	`1e82dd6`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-docs-new-theme/deploys/682e426c52d79f0008aa6095

netlify · 2025-05-16T17:23:18Z

✅ Deploy Preview for label-studio-storybook canceled.

Name	Link
🔨 Latest commit	`1e82dd6`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-storybook/deploys/682e426c693fe4000848ba03

netlify · 2025-05-16T17:23:28Z

✅ Deploy Preview for heartex-docs canceled.

Name	Link
🔨 Latest commit	`1e82dd6`
🔍 Latest deploy log	https://app.netlify.com/projects/heartex-docs/deploys/682e426c5f22180008b6e49d

jombooth · 2025-05-20T20:41:58Z

/fm sync

Workflow run

matt-bernstein · 2025-05-20T22:15:39Z

/fm sync

Workflow run

jombooth · 2025-05-21T21:00:21Z

label_studio/core/settings/base.py

@@ -598,6 +598,7 @@
 MEMBER_PERM = 'core.api_permissions.MemberHasOwnerPermission'
 RECALCULATE_ALL_STATS = None
 GET_STORAGE_LIST = 'io_storages.functions.get_storage_list'
+STORAGE_LOAD_TASKS_JSON = 'io_storages.utils._load_tasks_json'


Usually the meaning of a function or method that starts with a _ is "this should not be used outside of this class/module/whatever" - seems to me that we are treating this function as more of a public one, since it's being referred to in this other file. Would it be a pain to rename?

jombooth · 2025-05-21T21:01:25Z

label_studio/io_storages/utils.py

+
+
+@dataclass
+class StorageObjectParams:


I think StorageObject would be a better name for this, since it actually contains the task now

jombooth · 2025-05-21T21:02:43Z

label_studio/io_storages/utils.py

+
+    @classmethod
+    def bulk_create(
+        cls, task_datas: list[dict], key, row_idxs: list[int] | None = None, row_groups: list[int] | None = None


I would prefer row_indexes - generally in favor of avoiding abbreviations in variable names unless they're truly ubiquitous.

matt-bernstein · 2025-05-21T21:06:20Z

Note for posterity: pa.Table.from_pylist has a footgun in that it only constructs the schema from the first row; use from_json with BytesIO next time since it constructs the schema from all rows

label_studio/io_storages/utils.py

Co-authored-by: Jo Booth <[email protected]>

jombooth · 2025-05-21T21:10:34Z

pyproject.toml

@@ -49,6 +49,7 @@ dependencies = [
    "ordered-set (==4.0.2)",
    "pandas (>=2.2.3)",
    "psycopg2-binary (==2.9.10)",
+    "pyarrow (>=18.0.0,<19.0.0)",


Suggested change

"pyarrow (>=18.0.0,<19.0.0)",

"pyarrow (>=18.1.0,<19.0.0)",

Would recommend we don't allow people to go older than the version in poetry.lock, note this will require a relock

jombooth · 2025-05-21T21:11:25Z

label_studio/io_storages/utils.py

+    _error_wrapper()
+
+
+def load_tasks_json(blob_str: str, key: str) -> tuple[list[dict], list[StorageObjectParams]]:


Suggested change

def load_tasks_json(blob_str: str, key: str) -> tuple[list[dict], list[StorageObjectParams]]:

def load_tasks_json(blob_str: str, key: str) -> list[StorageObjectParams]:

jombooth · 2025-05-21T21:13:09Z

label_studio/io_storages/base_models.py

@@ -515,6 +523,7 @@ def sync(self):
                self.info_set_queued()
                import_sync_background(self.__class__, self.id)
            except Exception:
+                logger.debug(f'Storage {self} failed', exc_info=True)


Suggested change

logger.debug(f'Storage {self} failed', exc_info=True)

# needed to facilitate debugging storage-related testcases, since otherwise no exception is logged

logger.debug(f'Storage {self} failed', exc_info=True)

This is a rare case where I'm in favor of a comment - the reason it's here is fairly subtle and not at all obvious from the context

jombooth · 2025-05-21T21:14:56Z

label_studio/io_storages/base_models.py

@@ -431,7 +438,8 @@ def _scan_and_create_links(self, link_class):

            logger.debug(f'{self}: found new key {key}')
            try:
-                tasks_data = self.get_data(key)
+                # list of (task data + ImportStorageLink details)
+                links_params = self.get_data(key)


with the change to StorageObject, this can be storage_objects, which I think will be a lot more readable than links_params vs link_params (although, valiant effort at making the plurals situation readable enough!)

jombooth · 2025-05-21T21:15:14Z

label_studio/io_storages/base_models.py

@@ -431,7 +438,8 @@ def _scan_and_create_links(self, link_class):

            logger.debug(f'{self}: found new key {key}')
            try:
-                tasks_data = self.get_data(key)
+                # list of (task data + ImportStorageLink details)


Suggested change

# list of (task data + ImportStorageLink details)

matt-bernstein and others added 25 commits May 14, 2025 12:15

feat: ROOT-1: Add offset to storage link model to support storage lin…

3d23bf8

…k to task within file of tasks on cloud storage

review comments

9e2a7bb

remove whitespace

2cbcb41

Co-authored-by: Jo Booth <[email protected]>

update comment

3de9ffb

Co-authored-by: Jo Booth <[email protected]>

Sync Follow Merge dependencies

dc423d0

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15027678115

Merge branch 'develop' into 'fb-ROOT-1'

ad6a221

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15027678115

feat: ROOT-9: Allow reading multiple tasks from a JSON file in source…

411940c

… cloud storage

update comment

f37b8af

update storagelink creation

cfbb076

fix DM action

d3c02d9

blue

1033d26

Merge remote-tracking branch 'origin/develop' into fb-ROOT-9

c18d792

udpate test

4e75268

futureproof create method

b629817

Sync Follow Merge dependencies

8f9d6e8

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15030860401

Merge branch 'develop' into 'fb-ROOT-9'

b0c25a7

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15045532193

Sync Follow Merge dependencies

5714e98

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15045586529

Merge remote-tracking branch 'origin/develop' into fb-ROOT-9

027ad73

Sync Follow Merge dependencies

85e7b0a

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15052428789

make row_index=none default

f6841e3

get_data return type

810ba62

blue

8b41a3d

test new model fields

3ae1230

feat: ROOT-11: Support reading JSONL from source cloud storages

7cb8e9c

split out repeated json parsing logic from import storages

36e181e

matt-bernstein requested a review from a team as a code owner May 16, 2025 17:23

github-actions bot added the feat label May 16, 2025

matt-bernstein added 4 commits May 20, 2025 15:44

change structured type, add tests

f4c7d38

wip fix ff mocking

92e14fc

Merge remote-tracking branch 'origin/develop' into fb-ROOT-11

384bc3f

wip convert to pytest

065b6a3

jombooth and others added 4 commits May 20, 2025 16:47

regen lockfile

561c446

fix tests

2000629

place django_db mark at file level

78fbdd4

handle None

882c08a

matt-bernstein added 7 commits May 21, 2025 10:03

wip mock ffs

cf0c0d3

ensure blob_str enters as bytes

bae6ae1

add debug log, fix azure test

9001423

fix imputed field in task from other tasks in file

10e9725

Merge remote-tracking branch 'origin/develop' into fb-ROOT-11

39e4178

fix redis

2109679

fix mixed task formats and tests

3f065f5

jombooth reviewed May 21, 2025

View reviewed changes

Apply pre-commit linters

2806f0c

jombooth reviewed May 21, 2025

View reviewed changes

label_studio/io_storages/utils.py Outdated Show resolved Hide resolved

fix docstring

2aef19e

Co-authored-by: Jo Booth <[email protected]>

jombooth reviewed May 21, 2025

View reviewed changes

rename symbols

1e82dd6

jombooth reviewed May 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: ROOT-11: Support reading JSONL from source cloud storages #7555

feat: ROOT-11: Support reading JSONL from source cloud storages #7555

matt-bernstein commented May 16, 2025 •

edited

Loading

netlify bot commented May 16, 2025 •

edited

Loading

netlify bot commented May 16, 2025 •

edited

Loading

netlify bot commented May 16, 2025 •

edited

Loading

jombooth commented May 20, 2025 •

edited by robot-ci-heartex

Loading

matt-bernstein commented May 20, 2025 •

edited by robot-ci-heartex

Loading

jombooth May 21, 2025

jombooth May 21, 2025

jombooth May 21, 2025

matt-bernstein commented May 21, 2025

jombooth May 21, 2025 •

edited

Loading

jombooth May 21, 2025

jombooth May 21, 2025

jombooth May 21, 2025

jombooth May 21, 2025

jombooth May 21, 2025

		_error_wrapper()


		def load_tasks_json(blob_str: str, key: str) -> tuple[list[dict], list[StorageObjectParams]]:

	logger.debug(f'Storage {self} failed', exc_info=True)
	# needed to facilitate debugging storage-related testcases, since otherwise no exception is logged
	logger.debug(f'Storage {self} failed', exc_info=True)



		@dataclass
		class StorageObjectParams:

feat: ROOT-11: Support reading JSONL from source cloud storages #7555

Are you sure you want to change the base?

feat: ROOT-11: Support reading JSONL from source cloud storages #7555

Conversation

matt-bernstein commented May 16, 2025 • edited Loading

netlify bot commented May 16, 2025 • edited Loading

✅ Deploy Preview for label-studio-docs-new-theme canceled.

netlify bot commented May 16, 2025 • edited Loading

✅ Deploy Preview for label-studio-storybook canceled.

netlify bot commented May 16, 2025 • edited Loading

✅ Deploy Preview for heartex-docs canceled.

jombooth commented May 20, 2025 • edited by robot-ci-heartex Loading

matt-bernstein commented May 20, 2025 • edited by robot-ci-heartex Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matt-bernstein commented May 21, 2025

jombooth May 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matt-bernstein commented May 16, 2025 •

edited

Loading

netlify bot commented May 16, 2025 •

edited

Loading

netlify bot commented May 16, 2025 •

edited

Loading

netlify bot commented May 16, 2025 •

edited

Loading

jombooth commented May 20, 2025 •

edited by robot-ci-heartex

Loading

matt-bernstein commented May 20, 2025 •

edited by robot-ci-heartex

Loading

jombooth May 21, 2025 •

edited

Loading