Netflix · cclauss · Dec 23, 2025
diff --git a/.github/workflows/metaflow.s3_tests.minio.yml b/.github/workflows/metaflow.s3_tests.minio.yml
@@ -21,7 +21,7 @@ jobs:
       matrix:
         os: [ubuntu-22.04]
         ver: ['3.8', '3.9', '3.10', '3.11', '3.12']
-    
+
     steps:
     - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2.7.0
       with:
@@ -52,7 +52,7 @@ jobs:
         export METAFLOW_DATASTORE_SYSROOT_S3=s3://metaflow-test/metaflow/
         export AWS_ENDPOINT_URL_S3=http://localhost:9000
         export MINIO_TEST=1
-        
+
         # Run the same test command as the original workflow
         cd test/data
         PYTHONPATH=\$(pwd)/../../ python3 -m pytest --benchmark-skip -s -v

diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -3,7 +3,7 @@ on:
   release:
     types: [published]
 jobs:
-  test: 
+  test:
     uses: './.github/workflows/test.yml'
   test-stubs:
     uses: './.github/workflows/test-stubs.yml'

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -5,6 +5,8 @@ repos:
     hooks:
       - id: check-json
       - id: check-yaml
+      - id: trailing-whitespace
+        exclude: metaflow/plugins/cards/card_modules/main.js
   - repo: https://github.com/ambv/black
     rev: 25.12.0
     hooks:

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -3,7 +3,7 @@
 First off, thanks for taking the time! We'd love to hear from you! Drop us a line in our [chatroom](http://chat.metaflow.org)!
 
 If you are interested in contributing to Metaflow, we wrote a [guide](https://docs.metaflow.org/introduction/contributing-to-metaflow#contributing-code-and-issues)
-to get you started. 
+to get you started.
 
 We'd appreciate [issue reports](https://github.com/Netflix/metaflow/issues) if you run into trouble using Metaflow.
 
@@ -12,8 +12,8 @@ We'd appreciate [issue reports](https://github.com/Netflix/metaflow/issues) if y
 Everyone is welcome to join us in our [chatroom](http://chat.metaflow.org)!
 
 Please maintain appropriate, professional conduct while participating in our community. This includes all channels of
-communication. We take reports of harassment or unwelcoming behavior very seriously. To report such behavior, please 
-contact us via [email](mailto:[email protected]). 
+communication. We take reports of harassment or unwelcoming behavior very seriously. To report such behavior, please
+contact us via [email](mailto:[email protected]).
 
 ### Code style
 

diff --git a/R/DESCRIPTION b/R/DESCRIPTION
@@ -2,13 +2,13 @@ Package: metaflow
 Type: Package
 Title: Metaflow for R-Lang
 Version: 2.3.0
-Author: Jason Ge [aut] <[email protected]>, 
+Author: Jason Ge [aut] <[email protected]>,
   Savin Goyal [aut, cre] <[email protected]>,
   David Neuzerling [ctb] <[email protected]>
 Maintainer: Jason Ge <[email protected]>
-Description: Metaflow is a human-friendly R package 
-  that helps scientists and engineers build and manage real-life data science projects. 
-  Metaflow was originally developed at Netflix to boost productivity of data scientists 
+Description: Metaflow is a human-friendly R package
+  that helps scientists and engineers build and manage real-life data science projects.
+  Metaflow was originally developed at Netflix to boost productivity of data scientists
   who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.
 Encoding: UTF-8
 License: Apache License (>= 2.0) | file LICENSE
@@ -30,7 +30,7 @@ Suggests:
     rmarkdown
 RoxygenNote: 7.1.1
 Roxygen: list(markdown = TRUE)
-Collate: 
+Collate:
     'decorators-aws.R'
     'decorators-environment.R'
     'decorators-errors.R'

diff --git a/R/R/decorators-aws.R b/R/R/decorators-aws.R
@@ -1,6 +1,6 @@
 #' Decorator that configures resources allocated to a step
 #'
-#' @description 
+#' @description
 #' These decorators control the resources allocated to step running either
 #' locally or on _AWS Batch_. The `resources` decorator allocates resources for
 #' local execution. However, when a flow is executed with the `batch` argument
@@ -42,27 +42,27 @@
 #'   absolutely necessary. A swappiness value of `100` causes pages to be
 #'   swapped very aggressively. Accepted values are whole numbers between `0`
 #'   and `100`.
-#'   
+#'
 #' @inherit decorator return
 #'
 #' @export
-#' 
+#'
 #' @examples \dontrun{
-#' # This example will generate a large random matrix which takes up roughly 
+#' # This example will generate a large random matrix which takes up roughly
 #' # 48GB of memory, and sums the entries. The `batch` decorator forces this
 #' # step to run in an environment with 60000MB of memory.
-#' 
+#'
 #' start <- function(self) {
 #'   big_matrix <- matrix(rexp(80000*80000), 80000)
 #'   self$sum <- sum(big_matrix)
 #' }
-#' 
+#'
 #' end <- function(self) {
 #'   message(
 #'     "sum is: ", self$sum
 #'   )
 #' }
-#' 
+#'
 #' metaflow("BigSumFlowR") %>%
 #'   step(
 #'     batch(memory=60000, cpu=1),
@@ -91,7 +91,7 @@ batch <- function(
   queue = queue %||% pkg.env$mf$metaflow_config$BATCH_JOB_QUEUE
   iam_role = iam_role %||% pkg.env$mf$metaflow_config$ECS_S3_ACCESS_IAM_ROLE
   execution_role = execution_role %||% pkg.env$mf$metaflow_config$ECS_FARGATE_EXECUTION_ROLE
-  
+
   decorator(
     "batch",
     cpu = cpu,

diff --git a/R/R/decorators-environment.R b/R/R/decorators-environment.R
@@ -6,21 +6,21 @@
 #'   execution.
 #'
 #' @inherit decorator return
-#' 
+#'
 #' @export
 #'
 #' @examples \dontrun{
 #' start <- function(self) {
 #'   print(paste("The cutest animal is the", Sys.getenv("CUTEST_ANIMAL")))
 #'   print(paste("The", Sys.getenv("ALSO_CUTE"), "is also cute, though"))
 #' }
-#' 
+#'
 #' metaflow("EnvironmentVariables") %>%
-#'   step(step="start", 
+#'   step(step="start",
 #'        environment_variables(CUTEST_ANIMAL = "corgi", ALSO_CUTE = "penguin"),
-#'        r_function=start, 
+#'        r_function=start,
 #'        next_step="end") %>%
-#'   step(step="end") %>% 
+#'   step(step="end") %>%
 #'   run()
 #' }
 environment_variables <- function(...) {
@@ -32,7 +32,7 @@ environment_variables <- function(...) {
     if (is.null(env_vars_names) || "" %in% env_vars_names) {
       stop("All environment variables must be named")
     }
-    
+
     # Note that in this case, "TRUE" does not become Pythonic "True" ---
     # each environment variable value is immediately coerced to a character.
     env_var_dict <- lapply(
@@ -47,6 +47,6 @@ environment_variables <- function(...) {
     )
     env_var_dict <- paste0("{", paste(env_var_dict, collapse = ", "), "}")
   }
-  
+
   decorator("environment", vars = env_var_dict, .convert_args = FALSE)
 }
diff --git a/R/R/decorators-errors.R b/R/R/decorators-errors.R
@@ -1,10 +1,10 @@
 #' Decorator that configures a step to retry upon failure
-#' 
-#' @description 
+#'
+#' @description
 #' Use this decorator to configure a step to retry if it fails. Alternatively,
 #' retry _any_ failing steps in an entire flow with `run(with = c("retry")`.
-#' 
-#' See \url{https://docs.metaflow.org/v/r/metaflow/failures} for more 
+#'
+#' See \url{https://docs.metaflow.org/v/r/metaflow/failures} for more
 #' information on how to use this decorator.
 #'
 #' @param times Integer number of times to retry this step. Defaults to `3`. Set
@@ -13,9 +13,9 @@
 #'   retried.
 #' @param minutes_between_retries Integer Number of minutes between retries.
 #'   Defaults to `2`.
-#'   
+#'
 #' @inherit decorator return
-#'   
+#'
 #' @export
 #'
 #' @examples \dontrun{
@@ -24,23 +24,23 @@
 #' start <- function(self){
 #'   n <- rbinom(n=1, size=1, prob=0.5)
 #'   if (n==0){
-#'     stop("Bad Luck!") 
+#'     stop("Bad Luck!")
 #'   } else{
 #'     print("Lucky you!")
 #'   }
 #' }
-#' 
+#'
 #' end <- function(self){
 #'   print("Phew!")
 #' }
-#' 
+#'
 #' metaflow("RetryFlow") %>%
-#'   step(step="start", 
+#'   step(step="start",
 #'        retry(times=3),
-#'        r_function=start, 
+#'        r_function=start,
 #'        next_step="end") %>%
-#'   step(step="end", 
-#'        r_function=end) %>% 
+#'   step(step="end",
+#'        r_function=end) %>%
 #'   run()
 #' }
 retry <- function(times = 3L, minutes_between_retries = 2L) {
@@ -53,35 +53,35 @@ retry <- function(times = 3L, minutes_between_retries = 2L) {
 
 #' Decorator that configures a step to catch an error
 #'
-#' @description 
+#' @description
 #' Use this decorator to configure a step to catch any errors that occur during
 #' evaluation. For steps that can't be safely retried, it is a good idea to use
 #' this decorator along with `retry(times = 0)`.
-#' 
+#'
 #' See \url{https://docs.metaflow.org/v/r/metaflow/failures#catching-exceptions-with-the-catch-decorator}
 #' for more information on how to use this decorator.
 #'
 #' @param var Character. Name of the artifact in which to store the caught
 #' exception. If `NULL` (the default), the exception is not stored.
 #' @param print_exception Boolean. Determines whether or not the exception is
 #'   printed to stdout when caught. Defaults to `TRUE`.
-#'   
+#'
 #' @inherit decorator return
 #'
 #' @export
 #'
 #' @examples \donttest{
-#' 
+#'
 #' start <- function(self) {
 #'   stop("Oh no!")
 #' }
-#' 
+#'
 #' end <- function(self) {
 #'   message(
 #'     "Error is : ", self$start_failed
 #'   )
 #' }
-#' 
+#'
 #' metaflow("AlwaysErrors") %>%
 #'   step(
 #'     catch(var = "start_failed"),

diff --git a/R/R/decorators.R b/R/R/decorators.R
@@ -1,11 +1,11 @@
 #' Metaflow Decorator.
 #'
-#' @description 
+#' @description
 #' Decorates the `step` with the parameters present in its arguments. For this
 #' method to work properly, the `...` arguments should be named, and decorator
 #' type should be the first argument. It may be more convenient to use one of
 #' the _decorator wrappers_ listed below:
-#' 
+#'
 #' * \code{\link{resources}}
 #' * \code{\link{batch}}
 #' * \code{\link{retry}}
@@ -18,16 +18,16 @@
 #' @param .convert_args Boolean. If `TRUE` (the default), argument values will
 #'   be converted to analogous Python values, with strings quoted and escaped.
 #'   Disable this if argument values are already formatted for Python.
-#'   
+#'
 #' @return A object of class "decorator"
-#' 
+#'
 #' @export
-#' 
+#'
 #' @examples \dontrun{
 #' decorator("catch", print_exception=FALSE)
 #' decorator("resources", cpu=2, memory=10000)
 #' }
-#' 
+#'
 decorator <- function(x, ..., .convert_args = TRUE) {
   fmt_decorator(x, ..., .convert_args = .convert_args) %>%
     new_decorator()
@@ -55,7 +55,7 @@ new_decorator <- function(x) {
 #'
 #' @return character vector
 #' @keywords internal
-#' 
+#'
 #' @examples \dontrun{
 #' add_decorators(list(decorator("batch", cpu = 4), decorator("retry")))
 #' #> c("@batch(cpu=4)", "\n", "@retry", "\n")
@@ -66,13 +66,13 @@ add_decorators <- function(decorators) {
 }
 
 #' Format an R decorator as a Python decorator
-#' 
+#'
 #' @inheritSection add_decorators Python decorators
 #'
 #' @param x Decorator name.
 #' @inheritParams decorator
 #'
-#' @return character vector of length two, in which the first element is the 
+#' @return character vector of length two, in which the first element is the
 #' translated decorator and the second element is a new line character.
 #' @keywords internal
 #'
@@ -118,13 +118,13 @@ decorator_arguments <- function(args, .convert_args = TRUE) {
     stop("duplicate decorator arguments")
   }
   unlist(lapply(seq_along(args), function(x) {
-    
+
     wrapped_arg <- if (.convert_args) {
       wrap_argument(args[x])
     } else {
       args[x]
     }
-    
+
     if (x != length(args)) {
       paste0(names(args[x]), "=", wrapped_arg, ",")
     } else {

diff --git a/R/R/flags.R b/R/R/flags.R
@@ -8,7 +8,7 @@ flags <- function(...) {
 parse_flags <- function(arguments = commandArgs(TRUE)) {
   config_name <- Sys.getenv("R_CONFIG_ACTIVE", unset = "default")
 
-  configs <- pkg.env$configs 
+  configs <- pkg.env$configs
   loaded_configs <- list()
   for (key in names(configs[[config_name]])) {
     loaded_configs[[key]] <- eval(configs[[config_name]][[key]])
@@ -44,7 +44,7 @@ parse_arguments <- function(arguments = NULL) {
       if (i <= n){
         values$step_functions <- arguments[i]
       } else {
-        values$step_functions <- "" 
+        values$step_functions <- ""
       }
       next
     }
@@ -143,12 +143,12 @@ split_parameters <- function(flags) {
     "help", "resume",
     "max_num_splits", "max_workers",
     "other_args", "show", "user",
-    "my_runs", "run_id", 
+    "my_runs", "run_id",
     "origin_run_id", "with", "tag",
     # step-functions subcommands and options
-    "step_functions", 
+    "step_functions",
     "only_json", "generate_new_token",
-    "running", "succeeded", "failed", 
+    "running", "succeeded", "failed",
     "timed_out", "aborted", "namespace",
     "new_token", "workflow_timeout"
   )