-
Notifications
You must be signed in to change notification settings - Fork 795
feat: impl Task for private #18311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
feat: impl Task for private #18311
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
bf17de1
feat: impl Task for private
KKould 4ccb7b7
feat: imp when_condition for Task & store TaskRun on system table
KKould 6baff0a
feat: use Meta's Watch mechanism to distribute TaskMessage
KKould c42aac3
refactor: Using SQL to simplify DAG
KKould 5fe7a8b
chore: fix meta_store init
KKould b4eb4bd
chore: log error on spawn
KKould d9b4576
test: add test for private task
KKould 51bcf06
chore: add license for private task
KKould cc80564
chore: fix test_display_license_info
KKould 181a438
chore: codefmt
KKould 3a8a3dc
chore: add `accept` for Delete & After
KKould d7ba36b
chore: add restart test
KKould f0721b1
chore: codefmt
KKould 0b8b72f
chore: add `system.task_history` for Private Task & use `TaskMgr::acc…
KKould c15189b
fix: TableFunctionFactory create fail
KKould 739b8be
ci: rename to private task test
KKould 2e9d55a
chore: add cron test
KKould a5f4931
feat: add system table: `system.tasks`
KKould 863f120
chore: fix `update_or_create_task_run` correct on `ScheduleTask`
KKould 547aed3
chore: add Task when test on `test-private-task.sh`
KKould fc5fe53
chore: add private task config check on `GlobalServices::init_with`
KKould 22ea27f
chore: update Task version on Meta
KKould 578694f
chore: codefmt
KKould 98addfd
chore: remove `TaskMgr::list_task_fallback`
KKould 6848da5
chore: codefmt
KKould File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
name: "Test private task for databend query" | ||
description: "Test private task for databend query" | ||
runs: | ||
using: "composite" | ||
steps: | ||
- uses: ./.github/actions/setup_test | ||
|
||
- name: Install lsof | ||
shell: bash | ||
run: sudo apt-get update -yq && sudo apt-get install -yq lsof | ||
|
||
- name: Minio Setup for (ubuntu-latest only) | ||
shell: bash | ||
run: | | ||
docker run -d --network host --name minio \ | ||
-e "MINIO_ACCESS_KEY=minioadmin" \ | ||
-e "MINIO_SECRET_KEY=minioadmin" \ | ||
-e "MINIO_ADDRESS=:9900" \ | ||
-v /tmp/data:/data \ | ||
-v /tmp/config:/root/.minio \ | ||
minio/minio server /data | ||
|
||
export AWS_ACCESS_KEY_ID=minioadmin | ||
export AWS_SECRET_ACCESS_KEY=minioadmin | ||
export AWS_EC2_METADATA_DISABLED=true | ||
|
||
aws --endpoint-url http://127.0.0.1:9900/ s3 mb s3://testbucket | ||
aws --endpoint-url http://127.0.0.1:9900/ s3 cp tests/data s3://testbucket/data --recursive --no-progress | ||
|
||
- name: Run Private Task Tests | ||
shell: bash | ||
run: | | ||
bash ./tests/task/test-private-task.sh | ||
|
||
- name: Upload failure | ||
if: failure() | ||
uses: ./.github/actions/artifact_failure | ||
with: | ||
name: test-tasks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,165 @@ | ||
// Copyright 2021 Datafuse Labs | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
use std::collections::BTreeMap; | ||
|
||
use chrono::DateTime; | ||
use chrono::Utc; | ||
|
||
pub const EMPTY_TASK_ID: u64 = 0; | ||
|
||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] | ||
pub enum ScheduleType { | ||
IntervalType = 0, | ||
CronType = 1, | ||
} | ||
|
||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] | ||
pub enum Status { | ||
Suspended = 0, | ||
Started = 1, | ||
} | ||
|
||
#[derive(Debug, Clone, Copy, PartialEq)] | ||
pub enum State { | ||
Scheduled = 0, | ||
Executing = 1, | ||
Succeeded = 2, | ||
Failed = 3, | ||
Cancelled = 4, | ||
} | ||
|
||
#[derive(Debug, Clone, PartialEq)] | ||
pub struct ScheduleOptions { | ||
pub interval: Option<i32>, | ||
pub cron: Option<String>, | ||
pub time_zone: Option<String>, | ||
pub schedule_type: ScheduleType, | ||
pub milliseconds_interval: Option<u64>, | ||
} | ||
|
||
#[derive(Debug, Clone, PartialEq)] | ||
pub struct WarehouseOptions { | ||
pub warehouse: Option<String>, | ||
pub using_warehouse_size: Option<String>, | ||
} | ||
|
||
#[derive(Debug, Clone, PartialEq)] | ||
pub struct Task { | ||
pub task_id: u64, | ||
pub task_name: String, | ||
pub query_text: String, | ||
pub when_condition: Option<String>, | ||
pub after: Vec<String>, | ||
pub comment: Option<String>, | ||
// expired useless | ||
pub owner: String, | ||
pub owner_user: String, | ||
pub schedule_options: Option<ScheduleOptions>, | ||
pub warehouse_options: Option<WarehouseOptions>, | ||
pub next_scheduled_at: Option<DateTime<Utc>>, | ||
pub suspend_task_after_num_failures: Option<u64>, | ||
// TODO | ||
pub error_integration: Option<String>, | ||
pub status: Status, | ||
pub created_at: DateTime<Utc>, | ||
pub updated_at: DateTime<Utc>, | ||
pub last_suspended_at: Option<DateTime<Utc>>, | ||
// TODO | ||
pub session_params: BTreeMap<String, String>, | ||
} | ||
|
||
#[derive(Debug, Clone, PartialEq)] | ||
pub struct TaskRun { | ||
pub task: Task, | ||
pub run_id: u64, | ||
pub attempt_number: i32, | ||
pub state: State, | ||
pub scheduled_at: DateTime<Utc>, | ||
pub completed_at: Option<DateTime<Utc>>, | ||
pub error_code: i64, | ||
pub error_message: Option<String>, | ||
// expired useless | ||
pub root_task_id: u64, | ||
} | ||
|
||
#[derive(Debug, Clone, PartialEq)] | ||
pub enum TaskMessageType { | ||
Execute, | ||
Schedule, | ||
Delete, | ||
After, | ||
} | ||
|
||
#[derive(Debug, Clone, PartialEq)] | ||
pub enum TaskMessage { | ||
// Execute Task immediately. If an error occurs, if it is a SQL error in the task, | ||
// it will be recorded in the error message of the task run. | ||
ExecuteTask(Task), | ||
// Schedule Task will try to spawn a thread in Query to continue running according to the time set in schedule | ||
ScheduleTask(Task), | ||
// Delete the task information and try to cancel the scheduled task in the query. | ||
DeleteTask(String), | ||
// After Task will bind Task to the tasks in Task.afters. | ||
// When Execute Task is executed, after all the after tasks of Task are completed, | ||
// the execution will continue. | ||
AfterTask(Task), | ||
} | ||
|
||
impl TaskMessage { | ||
pub fn task_name(&self) -> &str { | ||
match self { | ||
TaskMessage::ExecuteTask(task) | ||
| TaskMessage::ScheduleTask(task) | ||
| TaskMessage::AfterTask(task) => task.task_name.as_str(), | ||
TaskMessage::DeleteTask(task_name) => task_name.as_str(), | ||
} | ||
} | ||
|
||
pub fn ty(&self) -> TaskMessageType { | ||
match self { | ||
TaskMessage::ExecuteTask(_) => TaskMessageType::Execute, | ||
TaskMessage::ScheduleTask(_) => TaskMessageType::Schedule, | ||
TaskMessage::DeleteTask(_) => TaskMessageType::Delete, | ||
TaskMessage::AfterTask(_) => TaskMessageType::After, | ||
} | ||
} | ||
|
||
pub fn key(&self) -> String { | ||
Self::key_with_type(self.ty(), self.task_name()) | ||
} | ||
|
||
pub fn key_with_type(ty: TaskMessageType, task_name: &str) -> String { | ||
let ty_num = match ty { | ||
TaskMessageType::Execute => 0, | ||
TaskMessageType::Schedule => 1, | ||
TaskMessageType::Delete => 2, | ||
TaskMessageType::After => 3, | ||
}; | ||
format!("{}-{}-{}", TaskMessage::prefix_range().0, ty_num, task_name) | ||
} | ||
|
||
/// Returns the inclusive range of key prefixes used by `TaskMessage`. | ||
/// | ||
/// This range can be used to scan all keys generated by `TaskMessage::key()` | ||
/// and related methods (e.g., `schedule_key`). The prefix `0` is prepended | ||
/// to all task-related keys to group them under the same prefix range, | ||
/// enabling efficient key scanning or iteration. | ||
/// | ||
/// The returned range is (0, 1), which includes all keys starting with `0-` | ||
/// (as produced by `TaskMessage::prefix()`), and excludes any other unrelated prefixes. | ||
pub fn prefix_range() -> (i64, i64) { | ||
(0, 1) | ||
} | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.