Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(log-viewer-webui): Add support for loading IR/JSON streams from S3. #673

Merged
merged 51 commits into from
Jan 20, 2025
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
28b597f
Initial backup
haiqi96 Jan 9, 2025
75e32ea
Adding
haiqi96 Jan 9, 2025
a08c558
Adding binary side support
haiqi96 Jan 9, 2025
dda24c5
Fix for binary support
haiqi96 Jan 10, 2025
722430c
Support s3 uploading for extraction task
haiqi96 Jan 10, 2025
3624586
C++ linter
haiqi96 Jan 10, 2025
85a832b
python linter
haiqi96 Jan 10, 2025
b970950
Use some subclass trick
haiqi96 Jan 13, 2025
0d34484
Merge branch 'main' into jsonl_upload
haiqi96 Jan 13, 2025
fdf8de3
Linter and other rearrangements
haiqi96 Jan 13, 2025
f933099
Merge branch 'main' into jsonl_upload
haiqi96 Jan 13, 2025
e506044
Remove unnecessary print
haiqi96 Jan 13, 2025
5aac350
Missing changes
haiqi96 Jan 13, 2025
5735449
Initial working version
haiqi96 Jan 14, 2025
bb5f9cc
simple renaming
haiqi96 Jan 15, 2025
2e47b72
Rename arguments for now
haiqi96 Jan 16, 2025
7481d53
Add default value
haiqi96 Jan 16, 2025
285384c
Address code review comments.
haiqi96 Jan 16, 2025
856664f
Apply suggestions from code review
haiqi96 Jan 16, 2025
473a613
Merge branch 'main' into jsonl_upload
haiqi96 Jan 16, 2025
7340c71
Update to work with latest codebase
haiqi96 Jan 16, 2025
9b09987
Merge branch 'jsonl_upload' into webui_s3_chunks
haiqi96 Jan 16, 2025
c6a1ad2
Update config (it's so small so I don't bother to make it a separate PR)
haiqi96 Jan 16, 2025
8a295df
Merge branch 'jsonl_upload' into webui_s3_chunks
haiqi96 Jan 16, 2025
ff14a5b
Linter
haiqi96 Jan 17, 2025
429b976
Merge branch 'jsonl_upload' into webui_s3_chunks
haiqi96 Jan 17, 2025
ef7fd82
Use environmental variable to handle S3 credentials for log viewer webui
haiqi96 Jan 17, 2025
518eec9
Attempt to fix linter
haiqi96 Jan 17, 2025
b2c842b
Merge branch 'main' into jsonl_upload
haiqi96 Jan 17, 2025
05bf3f6
Apply suggestions from code review
haiqi96 Jan 17, 2025
b6c8e69
Resolve simpler comments
haiqi96 Jan 17, 2025
db47554
Combine argument checking
haiqi96 Jan 17, 2025
4098ee5
fix stupid mistake
haiqi96 Jan 17, 2025
04b94ba
Add CLP prefix to env
haiqi96 Jan 17, 2025
b1827d1
Merge branch 'jsonl_upload' into webui_s3_chunks
haiqi96 Jan 17, 2025
abaabd8
Address review comment
haiqi96 Jan 17, 2025
2da0c15
Apply suggestions from code review
haiqi96 Jan 17, 2025
3ba8b89
Merge branch 'webui_s3_chunks' of https://github.com/haiqi96/clp_fork…
haiqi96 Jan 17, 2025
cf6a6c1
Update AWS sdk version
haiqi96 Jan 17, 2025
812de08
Use constant as requested in code review
haiqi96 Jan 17, 2025
59751c0
Linter
haiqi96 Jan 17, 2025
f37f40e
Merge branch 'main' into webui_s3_chunks
haiqi96 Jan 17, 2025
8e47f61
Add error check for S3 URL generation
haiqi96 Jan 17, 2025
a537b73
Use Null for optional values
haiqi96 Jan 18, 2025
78625a3
Linter
haiqi96 Jan 18, 2025
e8560d4
Apply suggestions from code review
haiqi96 Jan 19, 2025
33a7f8b
Address comments
haiqi96 Jan 19, 2025
dca12aa
update api versions
haiqi96 Jan 19, 2025
49fa164
fix docstrings
haiqi96 Jan 19, 2025
3a31483
Update components/log-viewer-webui/server/settings.json
kirkrodrigues Jan 20, 2025
1476f7b
Apply suggestions from code review
kirkrodrigues Jan 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -928,6 +928,28 @@ def start_log_viewer_webui(
"StreamTargetUncompressedSize": container_clp_config.stream_output.target_uncompressed_size,
"LogViewerDir": str(container_log_viewer_webui_dir / "yscope-log-viewer"),
}

container_cmd_extra_opts = []

stream_storage = clp_config.stream_output.storage
if StorageType.S3 == stream_storage.type:
s3_config = stream_storage.s3_config

settings_json_updates["StreamFilesS3Region"] = s3_config.region_code
settings_json_updates["StreamFilesS3PathPrefix"] = (
f"{s3_config.bucket}/{s3_config.key_prefix}"
)
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved
access_key_id, secret_access_key = s3_config.get_credentials()
if access_key_id is not None and secret_access_key is not None:
container_cmd_extra_opts.extend(
[
"-e",
f"AWS_ACCESS_KEY_ID={access_key_id}",
"-e",
f"AWS_SECRET_ACCESS_KEY={secret_access_key}",
]
)

settings_json = read_and_update_settings_json(settings_json_path, settings_json_updates)
with open(settings_json_path, "w") as settings_json_file:
settings_json_file.write(json.dumps(settings_json))
Expand All @@ -949,6 +971,8 @@ def start_log_viewer_webui(
"-u", f"{os.getuid()}:{os.getgid()}",
]
# fmt: on
container_cmd.extend(container_cmd_extra_opts)
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved

necessary_mounts = [
mounts.clp_home,
mounts.stream_output_dir,
Expand Down
2 changes: 1 addition & 1 deletion components/log-viewer-webui/client/src/ui/QueryStatus.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ const QueryStatus = () => {
setQueryState(QUERY_LOADING_STATES.LOADING);

const innerLogEventNum = logEventIdx - data.begin_msg_ix + 1;
window.location = `/log-viewer/index.html?filePath=/streams/${data.path}` +
window.location = `/log-viewer/index.html?filePath=${data.path}` +
`#logEventNum=${innerLogEventNum}`;
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved
})
.catch((e) => {
Expand Down
Loading
Loading