batch.Rd

% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/decorators-aws.R
\name{batch}
\alias{batch}
\alias{resources}
\title{Decorator that configures resources allocated to a step}
\usage{
batch(
  cpu = 1L,
  gpu = 0L,
  memory = 4096L,
  image = NULL,
  queue = NULL,
  iam_role = NULL,
  execution_role = NULL,
  shared_memory = NULL,
  max_swap = NULL,
  swappiness = NULL
)

resources(cpu = 1L, gpu = 0L, memory = 4096L, shared_memory = NULL)
}
\arguments{
\item{cpu}{Integer number of CPUs required for this step. Defaults to \code{1}.}

\item{gpu}{Integer number of GPUs required for this step. Defaults to \code{0}.}

\item{memory}{Integer memory size (in MiB) required for this step. Defaults to
\code{4096}.}

\item{image}{Character. Specifies the image to use when launching on AWS
Batch. If not specified, an appropriate
\href{https://hub.docker.com/r/rocker/ml}{Rocker Docker image} will be
used.}

\item{queue}{Character. Specifies the queue to submit the job to. Defaults to
the queue determined by the environment variable "METAFLOW_BATCH_JOB_QUEUE"}

\item{iam_role}{Character. IAM role that AWS Batch can use to access Amazon
S3. Defaults to the one determined by the environment variable
METAFLOW_ECS_S3_ACCESS_IAM_ROLE}

\item{execution_role}{Character. IAM role that AWS Batch can use to trigger
AWS Fargate tasks. Defaults to the one determined by the environment
variable METAFLOW_ECS_FARGATE_EXECUTION_ROLE. See the
\href{https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html}{AWS
Documentation} for more information.}

\item{shared_memory}{Integer. The value for the size (in MiB) of the
\verb{/dev/shm} volume for this step. This parameter maps to the \code{--shm-size}
option to \verb{docker run}.}

\item{max_swap}{Integer. The total amount of swap memory (in MiB) a container
can use for this step. This parameter is translated to the \code{--memory-swap}
option to docker run where the value is the sum of the container memory
plus the \code{max_swap} value.}

\item{swappiness}{This allows you to tune memory swappiness behavior for this
step. A swappiness value of \code{0} causes swapping not to happen unless
absolutely necessary. A swappiness value of \code{100} causes pages to be
swapped very aggressively. Accepted values are whole numbers between \code{0}
and \code{100}.}
}
\value{
A object of class "decorator"
}
\description{
These decorators control the resources allocated to step running either
locally or on \emph{AWS Batch}. The \code{resources} decorator allocates resources for
local execution. However, when a flow is executed with the \code{batch} argument
(\verb{run(with = c("batch")}.), it will also control which resources requested
from AWS. The \code{batch} decorator instead \emph{forces} the step to be run on \emph{AWS
Batch}. See \url{https://docs.metaflow.org/v/r/metaflow/scaling} for more
information on how to use these decorators.

If both \code{resources} and \code{batch} decorators are provided, the maximum values
from all decorators is used.
}
\examples{
\dontrun{
# This example will generate a large random matrix which takes up roughly 
# 48 GiB of memory, and sums the entries. The `batch` decorator forces this
# step to run in an environment with 60,000 MiB of memory.

start <- function(self) {
  big_matrix <- matrix(rexp(80000*80000), 80000)
  self$sum <- sum(big_matrix)
}

end <- function(self) {
  message(
    "sum is: ", self$sum
  )
}

metaflow("BigSumFlowR") \%>\%
  step(
    batch(memory=60000, cpu=1),
    step = "start",
    r_function = start,
    next_step = "end"
  ) \%>\%
  step(
    step = "end",
    r_function = end
  ) \%>\%
  run()
}
}