diff --git a/01-rstudio_and_basics.Rmd b/01-rstudio_and_basics.Rmd index b6a7144..bb6d8c6 100644 --- a/01-rstudio_and_basics.Rmd +++ b/01-rstudio_and_basics.Rmd @@ -9,7 +9,16 @@ Before we start with our meta-analysis, we have to download and prepare a **comp ## Getting RStudio to run on your computer {#RStudio} - +<<<<<<< HEAD + +======= +```{r, echo=FALSE, fig.width=3,fig.height=2} +library(png) +library(grid) +img <- readPNG("rstudiologo.PNG") +grid.raster(img) +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 As a prerequisite for this guide, you need to have **RStudio** and a few essential **R packages** installed. diff --git a/02-getting_data_in_R.Rmd b/02-getting_data_in_R.Rmd index 6263f3d..6ddbf83 100644 --- a/02-getting_data_in_R.Rmd +++ b/02-getting_data_in_R.Rmd @@ -2,11 +2,25 @@ ![](chappool.jpg) +<<<<<<< HEAD This chapter will tell you about how you can **import** your effect size data in RStudio. We will also show you a few commands which make it easier to **manipulate data** directly in R. Data preparation can be tedious and exhausting at times, but it is the backbone of all later steps when doing meta-analyses in R. We therefore have to pay **close attention** to preparing the data correctly before we can proceed. +======= +```{block,type='rmdinfo'} +This chapter will tell you about how you can **import** your effect size data in RStudio. We will also show you a few commands which make it easier to **manipulate data** directly in R. + +Data preparation can be tedious and exhausting at times, but it is the backbone of all later steps when doing meta-analyses in R. We therefore have to pay **close attention** to preparing the data correctly before we can proceed. + + +``` + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Data preparation in Excel @@ -21,7 +35,14 @@ To conduct Meta-Analyses in R, you need to have your study data prepared. For a * If you want to have a look at differences between various study subgroups later on, you also need a **subgroup code** for each study which signifies to which subgroup it belongs. For example, if a study was conducted in children, you might give it the subgroup code "children". +<<<<<<< HEAD As per usual, such data is stored in **EXCEL spreadsheets**. We recommend to store your data there, because this makes it very easy to import data into RStudio. However, it is very important how you **name the columns of your spreadsheet**. If you name the columns of your sheet adequately in EXCEL already, you can save a lot of time because your data doesn't have to be transformed in RStudio later on. +======= +As per usual, such data is stored in **EXCEL spreadsheets**. We recommend to store your data there, because this makes it very easy to import data into RStudio. + + +However, it is very important how you **name the columns of your spreadsheet**. If you name the columns of your sheet adequately in EXCEL already, you can save a lot of time because your data doesn't have to be transformed in RStudio later on. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Here is how you should name the data columns in your EXCEL spreadheet containing your Meta-Analysis data** @@ -29,6 +50,7 @@ As per usual, such data is stored in **EXCEL spreadsheets**. We recommend to sto ```{r,echo=FALSE} library(kableExtra) Package<-c("Author","Me","Se","Mc","Sc","Ne","Nc","Subgroup") +<<<<<<< HEAD Description<-c("This signifies the column for the study label (i.e., the first author)","The Mean of the experimental/intervention group","The Standard Deviation of the experimental/intervention group","The Mean of the control group","The Standard Deviation of the control group","The number of participants in the experimental/intervention group","The number of participants in the control group","This is the label for one of your Subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g. population). In this column, each study should then be given an subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique") m<-data.frame(Package,Description) names<-c("Column", "Description") @@ -40,6 +62,22 @@ kable(m) %>% Note that it **doesn't matter how these columns are ordered in your EXCEL spreadsheet**. They just have to be labeled correctly. There's also no need to **format** the columns in any way. If you type the column name in the first line of you spreadsheet, R will automatically detect it as a column name. It's also important to know that the import **will distort letters like ä,ü,ö,á,é,ê, etc**. So be sure to transform them to "normal" letters before you proceed. +======= +Description<-c("This signifies the column for the study label (i.e., the first author)","The Mean of the experimental/intervention group","The Standard Deviation of the experimental/intervention group","The Mean of the control group","The Standard Deviation of the control group","The number of participants in the experimental/intervention group","The number of participats in the control group","This is the label for one of your Subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g. population). In this column, each study should then be given an subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique") +m<-data.frame(Package,Description) +names<-c("Column", "Description") +colnames(m)<-names +kable(m) +``` + +Note that it **doesn't matter how these columns are ordered in your EXCEL spreadsheet**. They just have to be labeled correctly. + +There's also no need to **format** the columns in any way. If you type the column name in the first line of you spreadsheet, R will automatically detect it as a column name. + +```{block,type='rmdachtung'} +It's also important to know that the import **will distort letters like ä,ü,ö,á,é,ê, etc**. So be sure to transform them to "normal" letters before you proceed. +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ### Setting the columns of your sheet if you have calculated the effect sizes of each study already @@ -52,10 +90,19 @@ Description.cma<-c("This signifies the column for the study label (i.e., the fir m.cma<-data.frame(Package.cma,Description.cma) names.cma<-c("Column", "Description") colnames(m.cma)<-names.cma +<<<<<<< HEAD kable(m.cma) %>% column_spec(2, width = "40em") ``` +======= +kable(m.cma) +``` + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Importing the Spreadsheet into Rstudio {#import_excel} @@ -135,6 +182,7 @@ madata.s$`ROB superstreng`=NULL madata.s$compensation=NULL madata.s$instruments=NULL madata.s$guidance=NULL +<<<<<<< HEAD madata.s$`intervention duration`=NULL kable(madata.s) %>% column_spec(6) @@ -144,6 +192,21 @@ kable(madata.s) %>% ## Data manipulation Now that we have the Meta-Analysis data in RStudio, let's do a **few manipulations with the data**. These functions might come in handy when were conducting analyses later on. Going back to the output of the `str()` function, we see that this also gives us details on the type of column data we have stored in our data. There a different abbreviations signifying different types of data. +======= +kable(madata.s) +``` + +

+ +--- + +## Data manipulation + +Now that we have the Meta-Analysis data in RStudio, let's do a **few manipulations with the data**. These functions might come in handy when were conducting analyses later on. + + +Going back to the output of the `str()` function, we see that this also gives us details on the type of column data we have stored in our data. There a different abbreviations signifying different types of data. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} library(kableExtra) @@ -153,25 +216,40 @@ Description<-c("This is all data stored as numbers (e.g. 1.02)","This is all dat m<-data.frame(Package,type,Description) names<-c("Abbreviation", "Type","Description") colnames(m)<-names +<<<<<<< HEAD kable(m) %>% column_spec(3, width = "30em") +======= +kable(m) +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ``` ### Converting to factors {#convertfactors} +<<<<<<< HEAD Let's say we have the subgroup **Risk of Bias** (in which the Risk of Bias rating is coded), and want it to be a factor with two different levels: "low" and "high". To do this, we need to the variable `ROB` to be a factor. However, this variable is currently stored as a character (`chr`). We can have a look at this variable by typing the name of our dataset, then adding the selector `$` and then adding the variable we want to have a look at. $~$ +======= +Let's say we have the subgroup **Risk of Bias** (in which the Risk of Bias rating is coded), and want it to be a factor with two different levels: "low" and "high". + +To do this, we need to the variable `ROB` to be a factor. However, this variable is currently stored as a character (`chr`). We can have a look at this variable by typing the name of our dataset, then adding the selector `$` and then adding the variable we want to have a look at. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} madata$ROB<-madata$`ROB streng` ``` +<<<<<<< HEAD ```{r, collapse=TRUE} +======= +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 madata$ROB str(madata$ROB) ``` +<<<<<<< HEAD $~$ We can see now that `ROB`is indeed a **character** type variable, which contains only two words: "low" and "high". We want to convert this to a **factor** variable now, which has only two levels, low and high. To do this, we use the `factor()` function. @@ -179,26 +257,39 @@ We can see now that `ROB`is indeed a **character** type variable, which contains $~$ ```{r, collapse=TRUE} +======= +We can see now that `ROB`is indeed a **character** type variable, which contains only two words: "low" and "high". We want to convert this to a **factor** variable now, which has only two levels, low and high. To do this, we use the `factor()` function. + +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 madata$ROB<-factor(madata$ROB) madata$ROB str(madata$ROB) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 We now see that the variable has been **converted to a factor with the levels "high" and "low"**. ### Converting to logicals Now lets have a look at the **intervention type** subgroup variable. This variable is currently stores as a character `chr` too. +<<<<<<< HEAD $~$ ```{r, collapse=TRUE} +======= +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 madata$`intervention type` str(madata$`intervention type`) ``` +<<<<<<< HEAD $~$ Let's say we want a variable which only contains information if a study way a mindfulness intervention or not. A logical is very well suited for this. To convert the data to logical, we use the `as.logical` function. We will create a new variable containing this information called `intervention.type.logical`. To tell R what to count as `TRUE` and what as `FALSE`, we have to define the specific intervention type using the `==` command. @@ -206,10 +297,16 @@ Let's say we want a variable which only contains information if a study way a mi $~$ ```{r, collapse=TRUE} +======= +Let's say we want a variable which only contains information if a study way a mindfulness intervention or not. A logical is very well suited for this. To convert the data to logical, we use the `as.logical` function. We will create a new variable containing this information called `intervention.type.logical`. To tell R what to count as `TRUE` and what as `FALSE`, we have to define the specific intervention type using the `==` command. + +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 intervention.type.logical<-as.logical(madata$`intervention type`=="mindfulness") intervention.type.logical ``` +<<<<<<< HEAD $~$ We see that R has converted the character information into trues and falses for us. To check if this was done correctly, let's compare the original and the new variable @@ -220,19 +317,46 @@ names<-c("New", "Original") colnames(n)<-names kable(n) %>% column_spec(2, width = "10em") +======= +We see that R has converted the character information into trues and falses for us. To check if this was done correctly, let's compare the original and the new variable + +```{r} +n<-data.frame(intervention.type.logical,madata$`intervention type`) +names<-c("New", "Original") +colnames(n)<-names +kable(n) +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ``` ### Selecting specific studies {#select} +<<<<<<< HEAD It may often come in handy to **select certain studies for further analyses**, or to **exclude some studies in further analyses** (e.g., if they are outliers). To do this, we can use the `filter` function in the `dplyr`package, which is part of the `tidyverse` package we installed before. So, let's load the package first. +======= +It may often come in handy to **select certain studies for further analyses**, or to **exclude some studies in further analyses** (e.g., if they are outliers). + +To do this, we can use the `filter` function in the `dplyr`package, which is part of the `tidyverse` package we installed before. + +So, let's load the package first. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r, eval=FALSE,warning=FALSE} library(dplyr) ``` +<<<<<<< HEAD Let's say we want to do a Meta-Analysis with three studies in our dataset only. To do this, we need to create a new dataset containing only these studies using the `dplyr::filter()` function. The `dplyr::` part is necessary as there is more than one ``filter` function in R, and we want to use to use the one of the `dplyr`package. Let's say we want to have the studies by **Cavanagh et al.**, **Frazier et al.** and **Phang et al.** stored in another dataset, so we can conduct analyses only for these studies. The R code to store these three studies in a new dataset called `madata.new` looks like this: ```{r, message=FALSE, warning=FALSE} +======= +Let's say we want to do a Meta-Analysis with three studies in our dataset only. To do this, we need to create a new dataset containing only these studies using the `dplyr::filter()` function. The `dplyr::` part is necessary as there is more than one ``filter` function in R, and we want to use to use the one of the `dplyr`package. + +Let's say we want to have the studies by **Cavanagh et al.**, **Frazier et al.** and **Phang et al.** stored in another dataset, so we can conduct analyses only for these studies. + +The R code to store these three studies in a new dataset called `madata.new` looks like this: + +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 madata.new<-dplyr::filter(madata,Author %in% c("Cavanagh et al.", "Frazier et al.", "Phang et al.")) @@ -248,6 +372,7 @@ kable(madata.new) Note that the function can also be used for any other type of data and variable. We can also use it to e.g., only select studies which were coded as being a mindfulness study +<<<<<<< HEAD $~$ ```{r, message=FALSE, warning=FALSE} @@ -260,6 +385,14 @@ We can also use the `dplyr::filter()` function to exclude studies from our datas $~$ +======= +```{r} +madata.new.mf<-dplyr::filter(madata,`intervention type` %in% c("mindfulness")) +``` + +We can also use the `dplyr::filter()` function to exclude studies from our dataset. To do this, we only have to add `!` in front of the variable we want to use for filtering. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} madata.new.excl<-dplyr::filter(madata,!Author %in% c("Cavanagh et al.", "Frazier et al.", @@ -269,25 +402,41 @@ madata.new.excl<-dplyr::filter(madata,!Author %in% c("Cavanagh et al.", ### Changing cell values +<<<<<<< HEAD Sometimes, even when preparing your data in EXCEL, you might want to **change values in RStudio once you have imported your data**. To do this, we have to select a cell in our data frame in RStudio. This can be done by adding `[x,y]` to our dataset name, where **x** signifies the number of the **row** we want to select, and **y** signifies the number of the **column**. To see how this works, let's select a variable using this command first: $~$ +======= +Sometimes, even when preparing your data in EXCEL, you might want to **change values in RStudio once you have imported your data**. + +To do this, we have to select a cell in our data frame in RStudio. This can be done by adding `[x,y]` to our dataset name, where **x** signifies the number of the **row** we want to select, and **y** signifies the number of the **column**. + +To see how this works, let's select a variable using this command first: +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} madata[6,1] ``` +<<<<<<< HEAD $~$ We now see the **6th study** in our dataframe, and the value of this study for **Column 1 (the author name)** is displayed. Let's say we had a typo in this name and want to have it changed. In this case, we have to give this exact cell a new value. $~$ +======= +We now see the **6th study** in our dataframe, and the value of this study for **Column 1 (the author name)** is displayed. Let's say we had a typo in this name and want to have it changed. In this case, we have to give this exact cell a new value. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} madata[6,1]<-"Frogelli et al." ``` +<<<<<<< HEAD $~$ Let's check if the name has changed. $~$ +======= +Let's check if the name has changed. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} madata[6,1] @@ -295,5 +444,11 @@ madata[6,1] You can also use this function to change any other type of data, including numericals and logicals. Only for characters, you have to put the values you want to insert in `""`. +<<<<<<< HEAD + +======= +

+--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/03-pooling_effect_sizes.Rmd b/03-pooling_effect_sizes.Rmd index 9c209c7..f9a75de 100644 --- a/03-pooling_effect_sizes.Rmd +++ b/03-pooling_effect_sizes.Rmd @@ -2,8 +2,22 @@ ![](pooling.jpg) +<<<<<<< HEAD Now, let's get to the core of every Meta-Analysis: **pooling your effect sizes** to get one overall effect size estimate of the studies.When pooling effect sizes in Meta-Analysis, there are two approaches which we can use: the **Fixed-Effect-Model**, or the **Random-Effects-Model** [@borenstein2011]. There is an extensive debate on which model fits best in which context [@fleiss1993review], with no clear consensus in sight. Although it has been recommended to **only resort to the Random-Effects-Pooling model** in clinical psychology and the health sciences [@cuijpers2016meta], we will describe how to conduct both in R here. Both of these models only require an **effect size**, and a **dispersion (variance)** estimate for each study, of which the inverse is taken. This is why the methods are often called **generic inverse-variance methods**. We will describe in-depth how to conduct meta-analyses in R with **continuous variables** (such as effect sizes), as these are the most common ones in psychology and the health science field. Later on, we will present briefly how to do meta-analyses with **binary outcome** data too, which might be important if you're focusing on prevention trials. For these meta-analyses, we'll use the `meta` package [@schwarzer2007meta]. In [Section 2.1](#RStudio), we showed how to install the package. Now, load the package from your library to proceed. $~$ +======= +Now, let's get to the core of every Meta-Analysis: **pooling your effect sizes** to get one overall effect size estimate of the studies. + +```{block,type='rmdinfo'} +When pooling effect sizes in Meta-Analysis, there are two approaches which we can use: the **Fixed-Effect-Model**, or the **Random-Effects-Model** [@borenstein2011]. There is an extensive debate on which model fits best in which context [@fleiss1993review], with no clear consensus in sight. Although it has been recommended to **only resort to the Random-Effects-Pooling model** in clinical psychology and the health sciences [@cuijpers2016meta], we will describe how to conduct both in R here. + +Both of these models only require an **effect size**, and a **dispersion (variance)** estimate for each study, of which the inverse is taken. This is why the methods are often called **generic inverse-variance methods**. +``` + +We will describe in-depth how to conduct meta-analyses in R with **continuous variables** (such as effect sizes), as these are the most common ones in psychology and the health science field. Later on, we will present briefly how to do meta-analyses with **binary outcome** data too, which might be important if you're focusing on prevention trials. + +For these meta-analyses, we'll use the `meta` package [@schwarzer2007meta]. In [Section 2.1](#RStudio), we showed how to install the package. Now, load the package from your library to proceed. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r, warning=FALSE,message=FALSE} library(meta) @@ -14,14 +28,21 @@ library(tidyverse) library(knitr) ``` +<<<<<<< HEAD + +======= +

+--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Fixed-Effects-Model {#fixed} ### Pre-calculated effect size data {#pre.calc} +<<<<<<< HEAD @@ -29,6 +50,15 @@ library(knitr) The fixed-effects-model assumes that all studies along with their effect sizes stem from a single homogeneous population [@borenstein2011]. To calculate the overall effect, we therefore average all effect sizes, but give studies with greater precision a higher weight. In this case, greater precision means that the study has a larger **N**, which leads to a smaller **Standard Error** of its effect size estimate. For this weighing, we use the **inverse of the variance** $1/\hat\sigma^2_k$ of each study $k$. We then calculate a weighted average of all studies, our fixed effect size estimator $\hat\theta_F$: +======= +```{block, type='rmdinfo'} +**The idea behind the fixed-effects-model** + +The fixed-effects-model assumes that all studies along with their effect sizes stem from a single homogeneous population [@borenstein2011]. To calculate the overall effect, we therefore average all effect sizes, but give studies with greater precision a higher weight. In this case, greater precision means that the study has a larger **N**, which leads to a smaller **Standard Error** of its effect size estimate. + +For this weighing, we use the **inverse of the variance** $1/\hat\sigma^2_k$ of each study $k$. We then calculate a weighted average of all studies, our fixed effect size estimator $\hat\theta_F$: +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 \begin{equation} \hat\theta_F = \frac{\sum\limits_{k=1}^K \hat\theta_k/ \hat\sigma^2_k}{\sum\limits_{k=1}^K 1/\hat\sigma^2_k} @@ -43,8 +73,11 @@ In [Chapter 3.1](#excel_preparation), we have described two ways your EXCEL spre The functions to pool the results with a fixed-effect-model **differ depending on which data format you used**, but not much. First, let's assume you already have a dataset with the **calucated effects and SE** for each study. In my case, this is my `madata` dataset. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} load("Meta_Analysis_Data.RData") madata<-Meta_Analysis_Data @@ -53,12 +86,18 @@ madata<-Meta_Analysis_Data ```{r} str(madata) ``` +<<<<<<< HEAD $~$ This dataset has **continuous outcome data**. As our effect sizes are already calculated, we can use the `meta::metagen` function. For this function, we can specify loads of parameters, all of which you can accessed by typing `?metagen` in your console once the `meta` package is loaded. $~$ +======= + +This dataset has **continuous outcome data**. As our effect sizes are already calculated, we can use the `meta::metagen` function. For this function, we can specify loads of parameters, all of which you can accessed by typing `?metagen` in your console once the `meta` package is loaded. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Here is a table with the most important parameters for our code:** ```{r,echo=FALSE} @@ -73,6 +112,7 @@ ii<-c("This tells R to use the TE column to retrieve the effect sizes for each s ms<-data.frame(i,ii) names<-c("Parameter", "Function") colnames(ms)<-names +<<<<<<< HEAD kable(ms) %>% column_spec(2, width = "40em") ``` @@ -83,6 +123,13 @@ Let's code our first fixed-effects-model Meta-Analysis. We we will give the resu $~$ +======= +kable(ms) +``` + +Let's code our first fixed-effects-model Meta-Analysis. We we will give the results of this analysis the simple name `m`. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} m<-metagen(TE, seTE, @@ -96,8 +143,11 @@ m ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 We now see the results of our Meta-Analysis, including * The **individual effect sizes** for each study, and their weight @@ -107,24 +157,35 @@ We now see the results of our Meta-Analysis, including Using the `$` command, we can also have a look at various outputs directly. For example +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r, eval=FALSE} m$lower.I2 ``` +<<<<<<< HEAD $~$ Gives us the lower bound of the 95% confidence interval for *I^2^* $~$ +======= +Gives us the lower bound of the 95% confidence interval for *I^2^* + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r, echo=FALSE} m$lower.I2 ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 We can **save the results of the meta-analysis** to our working directory as a .txt-file using this command ```{r,eval=FALSE} @@ -153,6 +214,7 @@ ii<-c("The number of participants (N) in the intervention group", ms2<-data.frame(i,ii) names<-c("Parameter", "Function") colnames(ms2)<-names +<<<<<<< HEAD kable(ms2) %>% column_spec(2, width = "40em") ``` @@ -160,6 +222,11 @@ kable(ms2) %>% $~$ +======= +kable(ms2) +``` + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE,warning=FALSE} load("metacont_data.RData") metacont$Ne<-as.numeric(metacont$Ne) @@ -170,18 +237,26 @@ metacont$Sc<-as.numeric(metacont$Sc) ``` For this purpose, i will use my dataset `metacont`, which contains the raw data of all studies i want to snythesize +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} str(metacont) ``` +<<<<<<< HEAD $~$ Now, let's code the Meta-Analysis function, this time using the `meta::metacont` function, and my `metacont` dataset. I want to name my output `m.raw` now. $~$ +======= +Now, let's code the Meta-Analysis function, this time using the `meta::metacont` function, and my `metacont` dataset. I want to name my output `m.raw` now. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} m.raw<-metacont(Ne, Me, @@ -198,6 +273,7 @@ m.raw<-metacont(Ne, m.raw ``` +<<<<<<< HEAD $~$ @@ -209,12 +285,24 @@ We can **save the results of the meta-analysis** to our working directory as a . $~$ +======= + +```{block,type='rmdachtung'} +As you can see, all the calculated effect sizes are **negative** now, including the pooled effect. However, all studies report a positive outcome, meaning that the symptoms in the intervention group (e.g., of depression) were reduced. The negative orientation results from the fact that in **most clinical trials, lower scores indicate better outcomes** (e.g., less depression). It is no problem to report values like this: in fact, it is conventional. + +Some readers who are unfamiliar with meta-analysis, however, **might be confused** by this, so may consider changing the orientation of your values before you report them in your paper. +``` + +We can **save the results of the meta-analysis** to our working directory as a .txt-file using this command + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,eval=FALSE} sink("results.txt") print(m.raw) sink() ``` +<<<<<<< HEAD ## Random-Effects-Model {#random} @@ -226,33 +314,79 @@ $~$ **The Idea behind the Random-Effects-Model** In the Random-Effects-Model, we want to account for our assumption that the study effect estimates show more variance than when drawn from a single population [@schwarzer2015meta]. The random-effects-model works under the so-called **assumption of exchangeability**. This means that in Random-Effects-Model Meta-Analyses, we not only assume that effects of individual studies deviate from the true intervention effect of all studies due to sampling error, but that there is another source of variance introduced by the fact that the studies do not stem from one single population, but are drawn from a "universe" of populations. We therefore assume that there is not only one true effect size, but **a distribution of true effect sizes**. We therefore want to estimate the mean of this distribution of true effect sizes. The fixed-effect-model assumes that when the observed effect size $\hat\theta_k$ of an individual study $k$ deviates from the true effect size $\theta_F$, the only reason for this is that the estimate is burdened by (sampling) error $\epsilon_k$. +======= +

+ +--- + +## Random-Effects-Model {#random} + +Previously, we showed how to perform a fixed-effect-model meta-analysis using the `meta:metagen` and `meta:metacont` functions. + +However, we can only use the fixed-effect-model when we can assume that **all included studies come from the same population**. In practice this is hardly ever the case: interventions may vary in certain characteristics, the sample used in each study might be slightly different, or its methods. In this case, we cannot assume that all studies stem from one hypothesized "population" of studies. + +Same is the case once we detect **statistical heterogeneity** in our fixed-effect-model meta-analysis, as indicated by $I^{2}>0$. + +So, it is very likely that you will actually use a random-effects-model for your meta-analysis. Thankfully, there's not much more we have to think about when conducting a random-effects-model meta-analysis in R instead of a fixed-effect-model meta-analysis. + +```{block,type='rmdinfo'} +**The Idea behind the Random-Effects-Model** + +In the Random-Effects-Model, we want to account for our assumption that the study effect estimates show more variance than when drawn from a single population [@schwarzer2015meta]. The random-effects-model works under the so-called **assumption of exchangeability**. + +This means that in Random-Effects-Model Meta-Analyses, we not only assume that effects of individual studies deviate from the true intervention effect of all studies due to sampling error, but that there is another source of variance introduced by the fact that the studies do not stem from one single population, but are drawn from a "universe" of populations. We therefore assume that there is not only one true effect size, but **a distribution of true effect sizes**. We therefore want to estimate the mean of this distribution of true effect sizes. + +The fixed-effect-model assumes that when the observed effect size $\hat\theta_k$ of an individual study $k$ deviates from the true effect size $\theta_F$, the only reason for this is that the estimate is burdened by (sampling) error $\epsilon_k$. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 $$\hat\theta_k = \theta_F + \epsilon_k$$ While the random-effects-model assumes that, in addition, there is **a second source of error** $\zeta_k$.This second source of error is introduced by the fact that even the true effect size $\theta_k$ of our study $k$ is also only part of an over-arching distribution of true effect sizes with the mean $\mu$ [@borenstein2011]. +<<<<<<< HEAD ```{r, echo=FALSE, fig.width=6,fig.height=4,fig.align='center', fig.cap="An illustration of parameters of the random-effects-model"} +======= +``` + +```{r, echo=FALSE, fig.width=6,fig.height=4,fig.align='center'} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 library(png) library(grid) img <- readPNG("density.png") grid.raster(img) ``` +<<<<<<< HEAD +======= +

+*An illustration of parameters of the random-effects-model* +

+ +```{block,type='rmdinfo'} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 The formula for the random-effects-model therefore looks like this: $$\hat\theta_k = \mu + \epsilon_k + \zeta_k$$ When calculating a random-effects-model meta-analysis, where therefore also have to take the error $\zeta_k$ into account. To do this, we have to **estimate the variance of the distribution of true effect sizes**, which is denoted by $\tau^{2}$, or *tau^2^*. There are several estimators for $\tau^{2}$, all of which are implemented in `meta`. We will give you more details about them in the next section. +<<<<<<< HEAD Even though it is **conventional** to use random-effects-model meta-analyses in psychological outcome research, applying this model is **not undisputed**. The random-effects-model pays **more attention to small studies** when pooling the overall effect in a meta-analysis [@schwarzer2015meta]. Yet, small studies in particular are often fraught with **bias** (see [Chapter 8.1](#smallstudyeffects)). This is why some have argued that the fixed-effects-model should be nearly always preferred [@poole1999; @furukawa2003]. +======= +``` + +```{block,type='rmdachtung'} +Even though it is **conventional** to use random-effects-model meta-analyses in psychological outcome research, applying this model is **not undisputed**. The random-effects-model pays **more attention to small studies** when pooling the overall effect in a meta-analysis [@schwarzer2015meta]. Yet, small studies in particular are often fraught with **bias** (see [Chapter 8.1](#smallstudyeffects)). This is why some have argued that the fixed-effects-model should be nearly always preferred [@poole1999; @furukawa2003]. +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ### Estimators for *tau^2^* in the random-effects-model {#tau2} @@ -267,6 +401,7 @@ colnames(ms)<-names kable(ms) ``` +<<<<<<< HEAD $~$ **Which estimator should i use?** @@ -286,6 +421,31 @@ $~$ It should be noted, however, that the HKSJ method is not uncontroversial. Some authors argue that other (standard) pooling models should also be used **in addition** to the HKSJ as a **sensitivity analysis** [@wiksten2016hartung]. Jackson and colleagues [@jackson2017hartung] present four residual concerns with this method, which you may take into account before selecting your meta-analytic method. The paper can be read [here](https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.7411). +======= +```{block,type='rmdinfo'} +**Which estimator should i use?** + +All of these estimators derive $\tau^{2}$ using a slightly different approach, leading to somewhat different pooled effect size estimates and confidence intervals. If one of these approaches is more or less biased often depends on the context, and parameters such as the number of studies $k$, the number of participants $n$ in each study, how much $n$ varies from study to study, and how big $\tau^{2}$ is. + +An overview paper by Veroniki and colleagues [@veroniki2016methods] provides an excellent summary on current evidence which estimator might be more or less biased in which situation. The article is openly accessible, and you can read it [here](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4950030/). + +Especially in medical and psychological research, the by far most often used estimator is the **DerSimonian-Laird estimator** [@dersimonian1986meta]. Part of this widespread use might be attributable to the fact that programs such as *RevMan* or *Comprehensive Meta-Analysis* (older versions) only use this estimator. It is also the default option in our `meta` package in R. Simulation studies, however, have shown that the **Maximum-Likelihood**, **Sidik-Jonkman**, and **Empirical Bayes** estimators have better properties in estimating the between-study variance [@sidik2007comparison;@viechtbauer2005bias]. +``` + +```{block,type='rmdinfo'} +**The Hartung-Knapp-Sidik-Jonkman method** + +Another criticism of the **DerSimonian-Laird** method is that when estimating the variance of our pooled effect $var(\hat\theta_F)$, this method is very prone to producing false positives [@inthout2014hartung]. This is especially the case when the **number of studies** is small, and when there is substantial **heterogeneity** [@hartung1999alternative;@hartung2001refined;@hartung2001tests;@follmann1999valid;@makambi2004effect]. Unfortunately, this is very often the case in when we do meta-analysis in the medical field or in psychology. This is quite a problem, as we don't want to find pooled effects to be statistically significant when in fact they are not! + +The **Hartung-Knapp-Sidik-Jonkman (HKSJ) method** was thus proposed a way to produce more robust estimates of $var(\hat\theta_F)$. It has been shown that this method substantially outperforms the DerSimonian-Laird method in many cases [@inthout2014hartung]. The HKSJ method can also be very easily applied in R, while other programs don't have this option yet. This is another big plus of doing meta-analysis in R. The HKSJ usually leads to more **conservative** results, indicated by wider confidence intervals. +``` + +```{block,type='rmdachtung'} +**Residual concerns with the Hartung-Knapp-Sidik-Jonkman method** + +It should be noted, however, that the HKSJ method is not uncontroversial. Some authors argue that other (standard) pooling models should also be used **in addition** to the HKSJ as a **sensitivity analysis** [@wiksten2016hartung]. Jackson and colleagues [@jackson2017hartung] present four residual concerns with this method, which you may take into account before selecting your meta-analytic method. The paper can be read [here](https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.7411). +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ### Pre-calculated effect size data {#random.precalc} @@ -307,6 +467,7 @@ ii<-c("This tells R to use the TE column to retrieve the effect sizes for each s ms<-data.frame(i,ii) names<-c("Parameter", "Function") colnames(ms)<-names +<<<<<<< HEAD kable(ms) %>% column_spec(2, width = "40em") ``` @@ -315,18 +476,29 @@ $~$ I will use my `madata` dataset again to do the meta-analysis. For illustrative purposes, let's use the Sidik-Jonkman estimator ("SJ") and the HKSJ method. To do this analysis, make sure that `meta` as well as `metafor` are loaded in R. $~$ +======= +kable(ms) +``` + +I will use my `madata` dataset again to do the meta-analysis. For illustrative purposes, let's use the Sidik-Jonkman estimator ("SJ") and the HKSJ method. To do this analysis, make sure that `meta` as well as `metafor` are loaded in R. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,eval=FALSE,warning=FALSE} library(meta) library(metafor) ``` +<<<<<<< HEAD $~$ Now, let's code our random-effects-model meta-analysis. Remember, as our effect size data are precalculated, i'll use the `meta::metagen()` function. $~$ +======= +Now, let's code our random-effects-model meta-analysis. Remember, as our effect size data are precalculated, i'll use the `meta::metagen()` function. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} load("Meta_Analysis_Data.RData") madata<-Meta_Analysis_Data @@ -346,11 +518,19 @@ m.hksj<-metagen(TE, m.hksj ``` +<<<<<<< HEAD $~$ The output shows that our estimated effect is $g=0.5935$, and the 95% confidence interval stretches from $g=0.39$ to $0.80$ (rounded). It also becomes clear that this effect is different (and larger) than the one we found in the fixed-effects-model meta-analysis in [Chapter 4.1](#fixed) ($g=0.48$). Let's compare this to the output using the **DerSimonian-Laird** estimator, and when setting `hakn=FALSE`. As this estimator is the **default**, i don't have to define `method.tau` this time. $~$ +======= +The output shows that our estimated effect is $g=0.5935$, and the 95% confidence interval stretches from $g=0.39$ to $0.80$ (rounded). + +It also becomes clear that this effect is different (and larger) than the one we found in the fixed-effects-model meta-analysis in [Chapter 4.1](#fixed) ($g=0.48$). + +Let's compare this to the output using the **DerSimonian-Laird** estimator, and when setting `hakn=FALSE`. As this estimator is the **default**, i don't have to define `method.tau` this time. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} m.dl<-metagen(TE, @@ -365,11 +545,17 @@ m.dl<-metagen(TE, m.dl ``` +<<<<<<< HEAD $~$ We see that the overall effect size estimate using this estimator is similar to the previous one ($g=0.57$), but the confidence intervals **is narrower because we did not adjust them** using the HKSJ method. ```{r,echo=FALSE,fig.height=2, fig.width=10} +======= +We see that the overall effect size estimate using this estimator is similar to the previous one ($g=0.57$), but the confidence intervals **is narrower because we did not adjust them** using the HKSJ method. + +```{r,echo=FALSE,fig.height=2} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 TE<-c(m.hksj$TE.random,m.dl$TE.random) seTE<-c(m.hksj$seTE.random,m.dl$seTE.random) Method<-c("Knapp-Hartung-Sidik-Jonkman","DerSimonian-Laird") @@ -390,8 +576,11 @@ forest(m.frst,xlim = c(0.34,0.85)) I we use raw effect size data, such as the one stored in my `metacont` dataset, we can use the `meta::metacont` function again. The parameters stay the same as before. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE,warning=FALSE} load("metacont_data.RData") metacont$Ne<-as.numeric(metacont$Ne) @@ -420,11 +609,21 @@ m.hksj.raw ``` +<<<<<<< HEAD ## Meta-Analysis with binary outcomes {#binary} +======= +

+ +--- + +## Meta-Analysis with binary outcomes {#binary} + +![](compare.jpg) +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ### Event rate data @@ -435,8 +634,11 @@ Here, have two options again: * **The effect sizes are already calculated**. In this case, we can use the `metagen` function as we did before (see [Chapter 4.1](#fixed) and [Chapter 4.2](#random)). The calculated effect `TE` then describes the Odds Ratio, or whatever binary outcome we calculated previously for our data. * **We only have the raw outcome data**. If this is the case, we will have to use the `meta::metabin` function instead. We'll show you how to do this now. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **For meta-analyses of binary outcomes, we need our data in the following format:** ```{r,echo=FALSE} @@ -459,33 +661,48 @@ Description<-c( m<-data.frame(Package,Description) names<-c("Column", "Description") colnames(m)<-names +<<<<<<< HEAD kable(m) %>% column_spec(2, width = "40em") ``` $~$ +======= +kable(m) +``` + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 I'll use my dataset `binarydata`, which also has this format ```{r} load("binarydata.RData") str(binarydata) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 The other parameters are like the ones we used in the meta-analyses with continuous outcome data, with two exceptions: * **sm**: As we want to have a pooled effect for binary data, we have to choose another summary measure now. We can choose from **"OR"** (Odds Ratio), **"RR"** (Risk Ratio), or **RD** (Risk Difference), among other things. * **incr**. This lets us define if and how we want **conitinuity correction** to be performed. Such a correction is necessary in cases where one of the cells in your data is zero (e.g., because no one in the intervention arm died). This can be a frequent phenomenon in some contexts, and **distorts our effect size estimates**. By default, the `metabin` function adds the value **0.5** in all cells were N is zero [@gart1967bias]. This value can be changed using the `incr`-parameter (e.g., `incr=0.1`). If your trial arms are very uneven in terms of their total $n$, we can also use the **treatment arm continuity correction** [@j2004add]. This can be done by using `incr="TACC"`. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Here's the code for a meta-analysis with raw binary data** I have decided to run a random-effect-model meta-analysis. I want the summary measure to be the Risk Ratio (RR). +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} m.bin<-metabin(Ee, Ne, @@ -503,7 +720,11 @@ m.bin<-metabin(Ee, m.bin ``` +<<<<<<< HEAD $~$ +======= +

+>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **L'Abbé Plots** @@ -515,8 +736,11 @@ So-called **L'Abbé plots** [@labbe] are a good way to visualize data based on e The results of the `metabin` function can be easily used to generate L'Abbé plots using the `labbe.metabin` function included in the `meta` package. We can specify the following parameters: +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} library(kableExtra) Package<-c("x","bg","col","studlab","col.fixed","col.random") @@ -538,22 +762,36 @@ Description<-c( m<-data.frame(Package,Description) names<-c("Parameter", "Description") colnames(m)<-names +<<<<<<< HEAD kable(m) %>% column_spec(2, width = "40em") +======= +kable(m) +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ``` For this example, i'll use the `m.bin` output i previously generated using the `metabin` function. +<<<<<<< HEAD $~$ ```{r, fig.width=5} +======= +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 labbe.metabin(x = m.bin, bg = "blue", studlab = TRUE, col.random = "red") ``` +<<<<<<< HEAD Works like a charm! We see that the **dashed red line** signifying the **pooled effect estimate** of my meta-analysis is running trough the bottom-right sector of my L'Abbé plot, meaning that the overall effect size is positive (i.e., that the intervention has a preventive effect). However, it also becomes clear that **all studies** clearly follow this trend: we see that most studies lie tightly in the bottom-left corner of the plot, meaning that these studies had **small event rates** (i.e., the event in these studies was very rare irrespective of group assignment). We also see that two of our included studies don't fall into this pattern: **Schmidthauer** and **van der Zee**. Those two studies have higher event rates, and both favor the intervention group more clearly than the others did. +======= +Works like a charm! We see that the **dashed red line** signifying the **pooled effect estimate** of my meta-analysis is running trough the bottom-right sector of my L'Abbé plot, meaning that the overall effect size is positive (i.e., that the intervention has a preventive effect). + +However, it also becomes clear that **all studies** clearly follow this trend: we see that most studies lie tightly in the bottom-left corner of the plot, meaning that these studies had **small event rates** (i.e., the event in these studies was very rare irrespective of group assignment). We also see that two of our included studies don't fall into this pattern: **Schmidthauer** and **van der Zee**. Those two studies have higher event rates, and both favor the intervention group more clearly than the others did. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ### Incidence rates @@ -563,14 +801,21 @@ To conduct meta-analyses using incidence rate data, so-called **person-time** da As an example, i will use my `IRR.data.RData` dataset, in which the person-time (in my case, person-year) is stored in the `time.e` and `time.c` column for the experimental and control group, respectively. The `event.e` and `event.c` column contains the number of **events** (in my case, depression onsets) for both groups. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r, echo=FALSE} load("IRR.data.RData") IRR.data ``` +<<<<<<< HEAD $~$ +======= + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 To pool the data, we use the `metainc` function included in the `meta` package. We can set the following parameters: @@ -591,6 +836,7 @@ ii<-c("The number of events in the intervention group", ms<-data.frame(i,ii) names<-c("Parameter", "Function") colnames(ms)<-names +<<<<<<< HEAD kable(ms) %>% column_spec(2, width = "40em") @@ -598,6 +844,11 @@ kable(ms) %>% $~$ +======= +kable(ms) +``` + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} metainc(event.e, time.e, @@ -612,6 +863,10 @@ metainc(event.e, hakn = TRUE) ``` +<<<<<<< HEAD +======= +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/04-forest_plots.Rmd b/04-forest_plots.Rmd index a1c503e..d3e0dae 100644 --- a/04-forest_plots.Rmd +++ b/04-forest_plots.Rmd @@ -2,18 +2,33 @@ ![](forest.jpg) +<<<<<<< HEAD Now that we created the **output of our meta-analysis** using the `metagen`, `metacont` or `metabin` functions in `meta` (see [Chapter 4.1](#fixed),[Chapter 4.2](#random) and [Chapter 4.3](#binary)), it is time to present the data in a more digestable way. **Forest Plots** are an easy way to do this, and it is conventional to report forest plots in meta-analysis publications. +======= +```{block,type='rmdinfo'} +Now that we created the **output of our meta-analysis** using the `metagen`, `metacont` or `metabin` functions in `meta` (see [Chapter 4.1](#fixed),[Chapter 4.2](#random) and [Chapter 4.3](#binary)), it is time to present the data in a more digestable way. + +**Forest Plots** are an easy way to do this, and it is conventional to report forest plots in meta-analysis publications. +``` + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Generating a Forest Plot To produce a forest plot, we use the meta-analysis output we just created (e.g., `m`, `m.raw`) und the `meta::forest()` function. I'll use my `m.hksj.raw` output from [Chapter 4.2.3](#random.raw) to create the forest plot +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE,warning=FALSE,message=FALSE} load("metacont_data.RData") metacont$Ne<-as.numeric(metacont$Ne) @@ -40,6 +55,7 @@ m.hksj.raw<-metacont(Ne, metacont$intervention.type<-c("PCI","PCI","Mindfulness","CBT","CBT","CBT") ``` +<<<<<<< HEAD ```{r,fig.width=11,fig.height=3,fig.align='center'} forest(m.hksj.raw) ``` @@ -47,11 +63,21 @@ forest(m.hksj.raw) $~$ Looks good so far. We see that the function plotted a forest plot with a **diamond** (i.e. the overall effect and its confidence interval) and a **prediction interval**. There are plenty of **other parameters** within the `meta::forest` function which we can use to modify the forest plot. +======= +```{r,fig.width=11,fig.height=4,fig.align='center'} +forest(m.hksj.raw) +``` + +Looks good so far. We see that the function plotted a forest plot with a **diamond** (i.e. the overall effect and its confidence interval) and a **prediction interval**. + +There are plenty of **other parameters** within the `meta::forest` function which we can use to modify the forest plot. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} library(knitr) library(grid) load("foresttable.RData") +<<<<<<< HEAD foresttable1<-foresttable[1:23,] kable(foresttable1) %>% column_spec(3, width = "30em") @@ -78,6 +104,14 @@ kable(foresttable3) %>% This is again just an overview. For all settings, type `?meta::forest` in your **console** to see more.Let's play around with the function a little now: $~$ +======= +kable(foresttable) +``` + +This is again just an overview. For all settings, type `?meta::forest` in your **console** to see more. + +Let's play around with the function a little now: +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,fig.width=9,fig.height=3.5,fig.align='center'} forest(m.hksj.raw, @@ -99,11 +133,19 @@ forest(m.hksj.raw, ``` +<<<<<<< HEAD $~$ Looks good so far! For special **layout types**, proceed to [Chapter 5.2](#layouttypes) now. +======= +Looks good so far! For special **layout types**, proceed to [Chapter 5.2](#layouttypes) now. + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Layout types {#layouttypes} @@ -115,17 +157,23 @@ The `meta::forest` function also has two **Layouts** preinstalled which we can u The **RevMan** layout looks like this: +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,fig.width=10,fig.height=4,fig.align='center'} forest(m.hksj.raw, layout = "RevMan5", digits.sd = 2) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 The **JAMA** layout looks like this: ```{r,fig.width=7,fig.height=3,fig.align='center'} @@ -136,6 +184,7 @@ forest(m.hksj.raw, colgap.forest.left = unit(15,"mm")) ``` +<<<<<<< HEAD ## Saving the forest plots @@ -144,6 +193,19 @@ Let's say i want to save the JAMA version of my Forest Plot now. To do this, i h

$~$ +======= +

+ +--- + +## Saving the forest plots + +Let's say i want to save the JAMA version of my Forest Plot now. To do this, i have to reuse the code with which i plotted my forest plot, and put it between `pdf(file='name_of_the_pdf_i_want_to_create.pdf')` and `dev.off`, both in separate lines. This saves the plot into a PDF in my Working Directory. + +This way, i can export the plot in different formats (you can find more details on the saving options [here](#saving)). + +

+>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **PDF** @@ -157,8 +219,11 @@ forest.jama<-forest(m.hksj.raw, dev.off() ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **PNG** ```{r, eval=FALSE} @@ -171,8 +236,11 @@ forest.jama<-forest(m.hksj.raw, dev.off() ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Scalable Vector Graphic** ```{r, eval=FALSE} @@ -186,5 +254,11 @@ dev.off() ``` +<<<<<<< HEAD +======= +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/06-Subgroup_Analyses.Rmd b/06-Subgroup_Analyses.Rmd index 363791f..74d25bd 100644 --- a/06-Subgroup_Analyses.Rmd +++ b/06-Subgroup_Analyses.Rmd @@ -2,15 +2,30 @@ ![](subgroup.jpg) +<<<<<<< HEAD In [Chapter 6](#heterogeneity), we discussed in depth why **between-study heterogeneity** is such an important issue in interpreting the results of our meta-analysis, and how we can **explore sources of heterogeneity** using [outlier](#outliers) and [influence analyses](#influenceanalyses). Another source of between-study heterogeneity making our effect size estimate less precise could be that **there are slight differences in the study design or intervention components between the studies**. For example, in a meta-analysis on the effects of **cognitive behavioral therapy** (CBT) for **depression** in **university students**, it could be the case that some studies delivered the intervention in a **group setting**, while others delivered the therapy to each student **individually**. In the same example, it is also possible that studies used different **criteria** to determine if a student suffers from **depression** (e.g. they either used the *ICD-10* or the *DSM-5* diagnostic manual). Many other differences of this sort are possible, and it seems plausible that such study differences may also be associated with differences in the overall effect.In **subgroup analyses**, we therefore have a look at different **subgroups within the studies of our meta-analysis** and try to determine of the **differ between these subgroups**. $~$ +======= +In [Chapter 6](#heterogeneity), we discussed in depth why **between-study heterogeneity** is such an important issue in interpreting the results of our meta-analysis, and how we can **explore sources of heterogeneity** using [outlier](#outliers) and [influence analyses](#influenceanalyses). + +Another source of between-study heterogeneity making our effect size estimate less precise could be that **there are slight differences in the study design or intervention components between the studies**. For example, in a meta-analysis on the effects of **cognitive behavioral therapy** (CBT) for **depression** in **university students**, it could be the case that some studies delivered the intervention in a **group setting**, while others delivered the therapy to each student **individually**. In the same example, it is also possible that studies used different **criteria** to determine if a student suffers from **depression** (e.g. they either used the *ICD-10* or the *DSM-5* diagnostic manual). + +Many other differences of this sort are possible, and it seems plausible that such study differences may also be associated with differences in the overall effect. + +In **subgroup analyses**, we therefore have a look at different **subgroups within the studies of our meta-analysis** and try to determine of the **differ between these subgroups**. + +```{block,type='rmdinfo'} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **The idea behind subgroup analyses** Basically, a every subgroup analysis consists of **two parts**: (1) **pooling the effect of each subgroup**, and (2) **comparing the effects of the subgroups** [@borenstein2013meta]. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **1. Pooling the effect of each subgroup** @@ -19,8 +34,11 @@ This point it rather straightforward, as the same criteria as the ones for a **s * If you assume that **all studies in subgroup** stem from the same population, and all have **one shared true effect**, you may use the **fixed-effect-model**. As we mention in [Chapter 4](#pool), many **doubt** that this assumption is ever **true in psychological** and **medical research**, even when we partition our studies into subgroups. * The alternative, therefore, is to use a **random-effect-model** which assumes that the studies within a subgroup are drawn from a **universe** of populations follwing its own distribution, for which we want to estimate the **mean**. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **2. Comparing the effects of the subgroups** After we calculated the pooled effect for each subgroup, **we can compare the size of the effects of each subgroup**. However, to know if this difference is in fact singnificant and/or meaningful, we have to calculate the **Standard Error of the differences between subgroup effect sizes** $SE_{diff}$, to calculate **confidence intervals** and conduct **significance tests**. @@ -40,8 +58,24 @@ The (simplified) formula for the estimation of $V_{Diff}$ using this model there $$V_{Diff}=V_A + V_B + \frac{\hat T^2_G}{m} $$ +<<<<<<< HEAD Where $\hat T^2_G$ is the **estimated variance between the subgroups**, and $m$ is the **number of subgroups**. Be aware that subgroup analyses should **always be based on an informed, *a priori* decision** which subgroup differences within the study might be **practically relevant**, and would lead to information gain on relevant **research questions** in your field of research. It is also **good practice** to specify your subgroup analyses **before you do the analysis**, and list them in **the registration of your analysis**. It is also important to keep in mind that **the capabilites of subgroup analyses to detect meaningful differences between studies is often limited**. Subgroup analyses also need **sufficient power**, so it makes no sense to compare two or more subgroups when your entire number of studies in the meta-analysis is smaller than $k=10$ [@higgins2004controlling]. +======= +Where $\hat T^2_G$ is the **estimated variance between the subgroups**, and $m$ is the **number of subgroups**. +``` + +```{block,type='rmdachtung'} +Be aware that subgroup analyses should **always be based on an informed, *a priori* decision** which subgroup differences within the study might be **practically relevant**, and would lead to information gain on relevant **research questions** in your field of research. It is also **good practice** to specify your subgroup analyses **before you do the analysis**, and list them in **the registration of your analysis**. + +It is also important to keep in mind that **the capabilites of subgroup analyses to detect meaningful differences between studies is often limited**. Subgroup analyses also need **sufficient power**, so it makes no sense to compare two or more subgroups when your entire number of studies in the meta-analysis is smaller than $k=10$ [@higgins2004controlling]. + +``` + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Subgroup Analyses using the Mixed-Effects-Model {#mixed} @@ -298,6 +332,7 @@ Description<-c("The output of you meta-analysis. In my case, this is 'm.hksj'", m<-data.frame(Code,Description) names<-c("Code","Description") colnames(m)<-names +<<<<<<< HEAD kable(m) %>% column_spec(2, width = "40em") ``` @@ -305,6 +340,14 @@ kable(m) %>% In my `madata` dataset, which i used previously to generate my meta-analysis output `m.hksj`, i stored the subgroup variable `Control`. This variable specifies **which control group type was employed in which study**. There are **three subgroups**: `WLC` (waitlist control), `no intervention` and `information only`. The function to do a subgroup analysis using the mixed-effects-model with these paramters looks like this. $~$ +======= +kable(m) +``` + +In my `madata` dataset, which i used previously to generate my meta-analysis output `m.hksj`, i stored the subgroup variable `Control`. This variable specifies **which control group type was employed in which study**. There are **three subgroups**: `WLC` (waitlist control), `no intervention` and `information only`. + +The function to do a subgroup analysis using the mixed-effects-model with these paramters looks like this. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,message=FALSE,warning=FALSE} subgroup.analysis.mixed.effects(data=m.hksj, @@ -315,11 +358,21 @@ subgroup.analysis.mixed.effects(data=m.hksj, subgroup3 = "information only") ``` +<<<<<<< HEAD $~$ The results of the subgroup analysis are displayed under `Results for subgroups (fixed effect model)`. We see that, while the **pooled effects of the subgroups differ quite substantially** (*g* = 0.41-0.78), this difference is **not statistically significant**. This can be seen under `Test for subgroup differences` in the `Between groups` row. We can see that $Q=3.03$ and $p=0.2196$. This information can be reported in our meta-analysis paper. Please note that the values displayed under `k` in the `Results for subgroups (fixed effects model)` section are always 1, as the pooled effect of the subgroup is treated as a single study. To determine the actual $k$ of each subgroup, you can use the `count` function from `dplyr` in R. +======= +The results of the subgroup analysis are displayed under `Results for subgroups (fixed effect model)`. We see that, while the **pooled effects of the subgroups differ quite substantially** (*g* = 0.41-0.78), this difference is **not statistically significant**. + +This can be seen under `Test for subgroup differences` in the `Between groups` row. We can see that $Q=3.03$ and $p=0.2196$. This information can be reported in our meta-analysis paper. + +```{block,type='rmdachtung'} +Please not that the values displayed under `k` in the `Results for subgroups (fixed effects model)` section are always 1, as the pooled effect of the subgroup is treated as a single study. To determine the actual $k$ of each subgroup, you can use the `count` function from `dplyr` in R. +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} load("Meta_Analysis_Data.RData") @@ -332,6 +385,12 @@ library(dplyr) dplyr::count(madata, vars=madata$Control) ``` +<<<<<<< HEAD +======= +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Subgroup Analyses using the Random-Effects-Model @@ -340,9 +399,15 @@ region<-c("Netherlands","Netherlands","Netherlands","USA","USA","USA","USA","Arg madata$region<-region ``` +<<<<<<< HEAD Now, let's assume i want to **know if intervention effects in my meta-analysis differ by region**. I use a **random-effects-model** and the selected coutries Argentina, Australia, China, and the Netherlands.Again, i use the `m.hksj` meta-analysis output object. I can perform a random-effects-model for between-subgroup-differences using the `update.meta` function. For this function, we have to **set two parameters**. $~$ +======= +Now, let's assume i want to **know if intervention effects in my meta-analysis differ by region**. I use a **random-effects-model** and the selected coutries Argentina, Australia, China, and the Netherlands. + +Again, i use the `m.hksj` meta-analysis output object. I can perform a random-effects-model for between-subgroup-differences using the `update.meta` function. For this function, we have to **set two parameters**. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} library(knitr) @@ -351,11 +416,17 @@ Description<-c("Here, we specify the variable in which the subgroup of each stud m<-data.frame(Code,Description) names<-c("Code","Description") colnames(m)<-names +<<<<<<< HEAD kable(m) %>% column_spec(2, width = "40em") ``` $~$ +======= +kable(m) +``` + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} m.hksj<-metagen(TE, seTE, data=madata, method.tau = "SJ", hakn = TRUE, studlab = paste(Author), comb.random = TRUE) @@ -369,6 +440,7 @@ region.subgroup<-update.meta(m.hksj, region.subgroup ``` +<<<<<<< HEAD $~$ Here, we get the **pooled effect for each subgroup** (country). Under `Test for subgroup differences (random effects model)`, we can see the **test for subgroup differences using the random-effects-model**, which is **not significant** ($Q=4.52$,$p=0.3405$). This means that we did not find differences in the overall effect between different regions, represented by the country in which the study was conducted. @@ -380,5 +452,18 @@ $~$ To use a fixed-effect-model in combination with a fixed-effects-model, we can also use the `update.meta` function again. The procedure is the same as the one we described before, but we have to set `comb.random` as `FALSE` and `comb.fixed` as `TRUE`. +======= +Here, we get the **pooled effect for each subgroup** (country). Under `Test for subgroup differences (random effects model)`, we can see the **test for subgroup differences using the random-effects-model**, which is **not significant** ($Q=4.52$,$p=0.3405$). This means that we did not find differences in the overall effect between different regions, represented by the country in which the study was conducted. + +```{block,type='rmdachtung'} +**Using a fixed-effect-model for within-subgroup-pooling and a fixed-effects-model for between-subgroup-differences** + +To use a fixed-effect-model in combination with a fixed-effects-model, we can also use the `update.meta` function again. The procedure is the same as the one we described before, but we have to set `comb.random` as `FALSE` and `comb.fixed` as `TRUE`. +``` + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/07-metaregression.Rmd b/07-metaregression.Rmd index 9c47199..d90ca8f 100644 --- a/07-metaregression.Rmd +++ b/07-metaregression.Rmd @@ -5,10 +5,19 @@ Conceptually, **Meta-Regression** does not differ much from a **subgroup analysis**. In fact, subgroup analyses with more than two groups are nothing more than a meta-regression with categorial covariates. However, meta-regression does also allow us to use **continuous data** as covariates and check weather values of this variable are associated with effect size. +<<<<<<< HEAD **The idea behind meta-regression** You may have already performed regressions in regular data where participants or patients are the **unit of analysis**. In typical Meta-Analyses, we do not have the individual data for each participant available, but only the **aggregated effects**, which is why we have to perform meta-regressions with covariates at the **study level**. This also means that while we conduct analyses on participant samples much larger than usual in single studies, it is still very likely that **we don't have enough studies for a meta-regression to be sensible**. In [Chapter 7](#subgroup), we told you that subgroup analyses make no sense when *k*<10. For **meta-regression**, Borenstein and colleages [@borenstein2011] recommend that **each covariate should at least contain ten studies**, although this should not be seen as clear rule. In a conventional regression, we want to estimate a parameter $y$ using a covariate $x_i$ with $n$ regression coefficients $\beta$. A standard regression equation therefore looks like this: +======= +```{block,type='rmdinfo'} +**The idea behind meta-regression** + +You may have already performed regressions in regular data where participants or patients are the **unit of analysis**. In typical Meta-Analyses, we do not have the individual data for each participant available, but only the **aggregated effects**, which is why we have to perform meta-regressions with covariates at the **study level**. This also means that while we conduct analyses on participant samples much larger than usual in single studies, it is still very likely that **we don't have enough studies for a meta-regression to be sensible**. In [Chapter 7](#subgroup), we told you that subgroup analyses make no sense when *k*<10. For **meta-regression**, Borenstein and colleages [@borenstein2011] recommend that **each covariate should at least contain ten studies**, although this should not be seen as clear rule. + +In a conventional regression, we want to estimate a parameter $y$ using a covariate $x_i$ with $n$ regression coefficients $\beta$. A standard regression equation therefore looks like this: +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 $$y=\beta_0 + \beta_1x_1 + ...+\beta_nx_n$$ @@ -16,7 +25,15 @@ In a meta-regression, we want to estimate the **effect size** $\theta$ for diffe $$\hat \theta_k = \theta + \beta_1x_{1k} + ... + \beta_nx_{nk} + \epsilon_k + \zeta_k$$ +<<<<<<< HEAD You might have seen that when estimating the effect size $\theta_k$ of a study $k$ in our regression model, there are two **extra terms in the equation**, $\epsilon_k$ and $\zeta_k$. The same terms can also be found in the equation for the random-effects-model in [Chapter 4.2](#random). The two terms signify two types of **independent errors** which cause our regression prediction to be **imperfect**. The first one, $\epsilon_k$, is the sampling error through which the effect size of the study deviates from its "true" effect. The second one, $\zeta_k$, denotes that even the true effect size of the study is only sampled from **an overarching distribution of effect sizes** (see the Chapter on the [Random-Effects-Model](#random) for more details). In a **fixed-effect-model**, we assume that all studies actually share the **same true effect size** and that the **between-study heterogeneity** $\tau^2 = 0$. In this case, we do not consider $\zeta_k$ in our equation, but only $\epsilon_k$. As the equation above has includes **fixed effects** (the $\beta$ coefficients) as well as **random effects** ($\zeta_k$), the model used in meta-regression is often called **a mixed-effects-model**. Mathematically, this model is identical to the **mixed-effects-model** we described in [Chapter 7](#subgroup) where we explained how **subgroup analyses** work.Indeed **subgroup analyses with more than two subgroups** are nothing else than a **meta-regression** with a **categorical predictor**. For meta-regression, these subgroups are then **dummy-coded**, e.g. +======= +You might have seen that when estimating the effect size $\theta_k$ of a study $k$ in our regression model, there are two **extra terms in the equation**, $\epsilon_k$ and $\zeta_k$. The same terms can also be found in the equation for the random-effects-model in [Chapter 4.2](#random). The two terms signify two types of **independent errors** which cause our regression prediction to be **imperfect**. The first one, $\epsilon_k$, is the sampling error through which the effect size of the study deviates from its "true" effect. The second one, $\zeta_k$, denotes that even the true effect size of the study is only sampled from **an overarching distribution of effect sizes** (see the Chapter on the [Random-Effects-Model](#random) for more details). In a **fixed-effect-model**, we assume that all studies actually share the **same true effect size** and that the **between-study heterogeneity** $\tau^2 = 0$. In this case, we do not consider $\zeta_k$ in our equation, but only $\epsilon_k$. + +As the equation above has includes **fixed effects** (the $\beta$ coefficients) as well as **random effects** ($\zeta_k$), the model used in meta-regression is often called **a mixed-effects-model**. Mathematically, this model is identical to the **mixed-effects-model** we described in [Chapter 7](#subgroup) where we explained how **subgroup analyses** work. + +Indeed **subgroup analyses with more than two subgroups** are nothing else than a **meta-regression** with a **categorical predictor**. For meta-regression, these subgroups are then **dummy-coded**, e.g. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 $$ D_k = \{\begin{array}{c}0:ACT \\1:CBT \end{array}$$ @@ -24,6 +41,10 @@ $$\hat \theta_k = \theta + \beta x_{k} + D_k \gamma + \epsilon_k + \zeta_k$$ In this case, we assume the same **regression line**, which is simply "shifted" **up or down for the different subgroups** $D_k$. +<<<<<<< HEAD +======= +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r, echo=FALSE, fig.width=6,fig.cap="Visualisation of a Meta-Regression with dummy-coded categorial predictors",fig.align='center'} library(png) @@ -32,30 +53,56 @@ img <- readPNG("dummy.PNG") grid.raster(img) ``` +<<<<<<< HEAD $~$ +======= +

+ +```{block,type='rmdinfo'} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Assessing the fit of a regression model** To evaluate the **statistical significance of a predictor**, we a **t-test** of its $\beta$-weight is performed. $$ t=\frac{\beta}{SE_{\beta}}$$ +<<<<<<< HEAD Which provides a $p$-value telling us if a variable significantly predicts effect size differences in our regression model. If we fit a regression model, our aim is to find a model **which explains as much as possible of the current variability in effect sizes** we find in our data. In conventional regression, $R^2$ is commonly used to quantify the **goodness of fit** of our model in percent (0-100%). As this measure is commonly used, and many researchers know how to to interpret it, we can also calculate a $R^2$ analog for meta-regression using this formula: +======= +Which provides a $p$-value telling us if a variable significantly predicts effect size differences in our regression model. + +If we fit a regression model, our aim is to find a model **which explains as much as possible of the current variability in effect sizes** we find in our data. + +In conventional regression, $R^2$ is commonly used to quantify the **goodness of fit** of our model in percent (0-100%). As this measure is commonly used, and many researchers know how to to interpret it, we can also calculate a $R^2$ analog for meta-regression using this formula: +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 $$R_2=\frac{\hat\tau^2_{REM}-\hat\tau^2_{MEM}}{\hat\tau^2_{REM}}$$ Where $\hat\tau^2_{REM}$ is the estimated total heterogenetiy based on the random-effects-model and $\hat\tau^2_{REM}$ the total heterogeneity of our mixed-effects regression model. +<<<<<<< HEAD +======= +``` + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Calculating meta-regressions in R Meta-regressions can be conducted in R using the `metareg` function in `meta`. To show the similarity between `subgroup` analysis and `meta-regression` with categorical predictors, i'll first conduct a meta-regression with my variable "Control" as predictor again. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} metareg(m.hksj,Control) ``` +<<<<<<< HEAD $~$ We see in the output that the `metareg` function uses the values of "Control" (i.e, the three different types of control groups) as a **moderator**. It takes **"information only"** as a dummy-coded *reference group*, and **"no intervention"** and **"WLC"** as dummy-coded **predictors**. Under `Test of Moderators`, we can see that control groups are not significantly associated with effect size differences $F_{2,15}=0.947$, $p=0.41$. Our regression model does not explain any of the variability in our effect size data ($R^2=0\%$). Below `Model Results`, we can also see the $\beta$-values (`estimate`) of both predictors, and their significance level `pval`. As we can see, both predictors were not significant. @@ -67,6 +114,19 @@ $~$ Let's assume i want to check if the **publication year** is associated with effect size. I have stored the variable `pub_year`, containing the publication year of every study in my dataset, and conducted the meta-analysis with it. I stored my meta-analysis output in the `m.pubyear` output. Now, i can use this predictor in a meta-regression. $~$ +======= +We see in the output that the `metareg` function uses the values of "Control" (i.e, the three different types of control groups) as a **moderator**. It takes **"information only"** as a dummy-coded *reference group*, and **"no intervention"** and **"WLC"** as dummy-coded **predictors**. Under `Test of Moderators`, we can see that control groups are not significantly associated with effect size differences $F_{2,15}=0.947$, $p=0.41$. Our regression model does not explain any of the variability in our effect size data ($R^2=0\%$). + +Below `Model Results`, we can also see the $\beta$-values (`estimate`) of both predictors, and their significance level `pval`. As we can see, both predictors were not significant. + +

+ +**Continuous variables** + +Let's assume i want to check if the **publication year** is associated with effect size. I have stored the variable `pub_year`, containing the publication year of every study in my dataset, and conducted the meta-analysis with it. I stored my meta-analysis output in the `m.pubyear` output. + +**Now, i can use this predictor in a meta-regression.** +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} load("Meta_Analysis_Data.RData") @@ -81,11 +141,19 @@ output.metareg<-metareg(m.pubyear,pub_year) output.metareg ``` +<<<<<<< HEAD $~$ As you can see from the output, `pub_year` was now included as a **predictor**, but it is not significantly associated with the effect size ($p=0.9412$). +======= +As you can see from the output, `pub_year` was now included as a **predictor**, but it is not significantly associated with the effect size ($p=0.9412$). + +

+ +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Plotting regressions @@ -108,8 +176,11 @@ colnames(ms)<-names kable(ms) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,eval=FALSE} bubble(output.metareg, xlab = "Publication Year", @@ -117,4 +188,10 @@ bubble(output.metareg, studlab = TRUE) ``` +<<<<<<< HEAD + +======= +

+--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/08-publication_bias.Rmd b/08-publication_bias.Rmd index 45e0ba3..7956bf9 100644 --- a/08-publication_bias.Rmd +++ b/08-publication_bias.Rmd @@ -2,6 +2,7 @@ ![](publicationbias.jpg) +<<<<<<< HEAD In the last chapters, we have showed you how to **pool effects** in meta-analysis, choose the **right pooling model**, assess the **heterogeneity** of your effect estimate, and determine sources of heterogeneity through **outlier, influence, and subgroup analyses**. Nevertheless, even the most thoroughly conducted meta-analysis can only work with the **study material at hand**. An issue commonly discussed in research, however, is the **file-drawer** or **publication bias** problem, which states that a study with high effect sizes **is more likely to be published** than a study with a low effect size [@rothstein2006publication]. Such **missing studies** with low effect sizes, it is assumed, thus never get published and therefore cannot be integrated into our meta-analysis. This leads to **publication bias**, as the pooled effect we estimate in our meta-analysis might be higher than the **true effect size** because we did not consider the missing studies with lower effects due to the simple fact that they were never published. Although this practice is gradually changing [@nelson2018psychology], weather a study is published or not heavily depends on the **statistical significance** ($p<0.05$) of its results [@dickersin2005publication]. For any sample size, a result is more likely to become **statistically significant** if its **effect size is high**. This is particularly true for **small studies**, for which very large effect sizes are needed to attain a statisitcally significant result. In the [following chapter](#smallstudyeffects), we will describe the **idea behind statistical models for publication bias** in further depth. We termed these concepts and methods **small-study effect methods**, as it is small studies that they mostly focus on. These methods assume that publication bias is primarily driven by **effect size** and because researchers **immediately put every study in the file drawer once the results are not significant**. Recently, it has been argued that these **assumptions may not be true**, and that publication bias is mostly caused by **significance levels** and **p-hacking** [@simonsohn2014p]. An alternative method called **P-Curve** has therefore been suggested to examine **publication bias**. We will present this method in the [last chapter](#pcurve) of this section. $~$ @@ -11,14 +12,43 @@ $~$ While recent research suggests that the conventional **small-study effect methods** may have substantial **limitations**, and that **p-Curve** may be able to estimate the true effect with less bias [@simonsohn2014p;@simonsohn2015better;@simonsohn2014pb], please note that both methods are based on different **theoretical assumptions** about the origin of publication bias. As we cannot ultimately decide which assumption is the **"true"** one in specific research fields, and, in practice **the true effect is unkown when doing meta-analysis**, we argue that you may use **both methods** and compare results as **sensitivity analyses** [@harrer2019internet]. **P-curve** was developed with **full-blown experimental psychological research in mind**, in which researchers often have **high degrees of "researcher freedom"** [@simmons2011false] in deleting outliers and performing statistical test on their data. We argue that this looks slightly different for **clinical psychology** and the medical field, where researchers conduct **randomized controlled trials** whith a clear **primary outcome**: the difference between the control and the intervention group after the treatment. While it is also true for **medicine and clinical psychology that statistical significance plays an important role**, the **effect size** of an intervention is often of greater interest, as **treatments are often compared in terms of their treatment effects** in this field. Furthermore, best practice for randomized controlled trials is to perform **intention-to-treat** analyses, in which all collected data in a trial has to be considered, giving researchers less space to "play around" with their data and perform p-hacking. While we certainly do not want to insinuate that **outcome research in clinical psychology** is free from p-hacking and bad data analysis practices, this should be seen as a **caveat** that the assumptions of the small-study effects methods may be more adequate for clinical psychology than other fields within psychology, especially when **the risk of bias for each study is also taken into account*. Facing this uncertainty, we think that conducting both analyses and reporting them in our research paper may be the most adequate approach until meta-scientific research gives us more certainty about which **assumption actually best reflects the field of clinical psychology**. +======= +In the last chapters, we have showed you how to **pool effects** in meta-analysis, choose the **right pooling model**, assess the **heterogeneity** of your effect estimate, and determine sources of heterogeneity through **outlier, influence, and subgroup analyses**. + +Nevertheless, even the most thoroughly conducted meta-analysis can only work with the **study material at hand**. An issue commonly discussed in research, however, is the **file-drawer** or **publication bias** problem, which states that a study with high effect sizes **is more likely to be published** than a study with a low effect size [@rothstein2006publication]. + +Such **missing studies** with low effect sizes, it is assumed, thus never get published and therefore cannot be integrated into our meta-analysis. This leads to **publication bias**, as the pooled effect we estimate in our meta-analysis might be higher than the **true effect size** because we did not consider the missing studies with lower effects due to the simple fact that they were never published. + +Although this practice is gradually changing [@nelson2018psychology], weather a study is published or not heavily depends on the **statistical significance** ($p<0.05$) of its results [@dickersin2005publication]. For any sample size, a result is more likely to become **statistically significant** if its **effect size is high**. This is particularly true for **small studies**, for which very large effect sizes are needed to attain a statisitcally significant result. + +In the [following chapter](#smallstudyeffects), we will describe the **idea behind statistical models for publication bias** in further depth. We termed these concepts and methods **small-study effect methods**, as it is small studies that they mostly focus on. These methods assume that publication bias is primarily driven by **effect size** and because researchers **immediately put every study in the file drawer once the results are not significant**. + +Recently, it has been argued that these **assumptions may not be true**, and that publication bias is mostly caused by **significance levels** and **p-hacking** [@simonsohn2014p]. An alternative method called **P-Curve** has therefore been suggested to examine **publication bias**. We will present this method in the [last chapter](#pcurve) of this section. + +```{block,type='rmdinfo'} +**Which method should i use for my meta-analysis?** + +While recent research suggests that the conventional **small-study effect methods** may have substantial **limitations**, and that **p-Curve** may be able to estimate the true effect with less bias [@simonsohn2014p;@simonsohn2015better;@simonsohn2014pb], please note that both methods are based on different **theoretical assumptions** about the origin of publication bias. As we cannot ultimately decide which assumption is the **"true"** one in specific research fields, and, in practice **the true effect is unkown when doing meta-analysis**, we argue that you may use **both methods** and compare results as **sensitivity analyses** [@harrer2019internet]. + +**P-curve** was developed with **full-blown experimental psychological research in mind**, in which researchers often have **high degrees of "researcher freedom"** [@simmons2011false] in deleting outliers and performing statistical test on their data. + +We argue that this looks slightly different for **clinical psychology** and the medical field, where researchers conduct **randomized controlled trials** whith a clear **primary outcome**: the difference between the control and the intervention group after the treatment. While it is also true for **medicine and clinical psychology that statistical significance plays an important role**, the **effect size** of an intervention is often of greater interest, as **treatments are often compared in terms of their treatment effects** in this field. Furthermore, best practice for randomized controlled trials is to perform **intention-to-treat** analyses, in which all collected data in a trial has to be considered, giving researchers less space to "play around" with their data and perform p-hacking. While we certainly do not want to insinuate that **outcome research in clinical psychology** is free from p-hacking and bad data analysis practices, this should be seen as a **caveat** that the assumptions of the small-study effects methods may be more adequate for clinical psychology than other fields within psychology, especially when **the risk of bias for each study is also taken into account*. +Facing this uncertainty, we think that conducting both analyses and reporting them in our research paper may be the most adequate approach until meta-scientific research gives us more certainty about which **assumption actually best reflects the field of clinical psychology**. + +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ## Small-study effect methods {#smallstudyeffects} The **small-study effect methods** we present here have been conventional for many years. Thus various methods to assess and control for publication bias have been developed, but we will only focus on the most important ones here. +<<<<<<< HEAD +======= +```{block,type='rmdinfo'} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **The model behind small-study effects methods** According to Borenstein et al. [@borenstein2011]. The model behind the most common small-study effects methods has these core **assumptions**: @@ -29,6 +59,7 @@ According to Borenstein et al. [@borenstein2011]. The model behind the most comm In accordance with these assumptions, the methods we present here particularly focus **on small studies with small effect sizes, and wheather they are missing**. +<<<<<<< HEAD ### Funnel plots @@ -36,6 +67,15 @@ In accordance with these assumptions, the methods we present here particularly f The best way to visualize weather **small studies with small effect sizes are missing** is through **funnel plots**. We can generate a funnel plot for our `m.hksj` meta-analysis output using the `funnel()` function in `meta`. $~$ +======= +``` + +### Funnel plots + +The best way to visualize weather **small studies with small effect sizes are missing** is through **funnel plots**. + +We can generate a funnel plot for our `m.hksj` meta-analysis output using the `funnel()` function in `meta`. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE,message=FALSE} library(meta) @@ -60,6 +100,7 @@ m.hksj<-metagen(TE, funnel(m.hksj,xlab = "Hedges' g") ``` +<<<<<<< HEAD $~$ The **funnel plot** basically consists of a **funnel** and two **axes**: the y-axis showing the **Standard Error** $SE$ of each study, with larger studies (which thus have a smaller $SE$) plotted **on top of the y-axis**; and the x-axis showing the **effect size** of each study. Given our assumptions, and in the case when there is **no publication bias**, all studies would lie **symmetrically around our pooled effect size (the striped line)** within the form of the funnel. When **publication bias is present**, we would assume that the funnel would look asymmetrical, because only the small studies with a large effect size very published, **while small studies without a significant, large effect would be missing**.We see from the plot that in the case of our meta-anlysis `m.hksj`, the latter is probably true. We see that the plot is highly asymmetrical, with exactly the small studies with low effect size missing in the bottom-left corner of our plot. We can also display the name of each study using the `studlab` parameter. @@ -67,6 +108,17 @@ The **funnel plot** basically consists of a **funnel** and two **axes**: the y-a $~$ ```{r, fig.width=10, fig.height=10} +======= +The **funnel plot** basically consists of a **funnel** and two **axes**: the y-axis showing the **Standard Error** $SE$ of each study, with larger studies (which thus have a smaller $SE$) plotted **on top of the y-axis**; and the x-axis showing the **effect size** of each study. + +Given our assumptions, and in the case when there is **no publication bias**, all studies would lie **symmetrically around our pooled effect size (the striped line)** within the form of the funnel. When **publication bias is present**, we would assume that the funnel would look asymmetrical, because only the small studies with a large effect size very published, **while small studies without a significant, large effect would be missing**. + +We see from the plot that in the case of our meta-anlysis `m.hksj`, the latter is probably true. We see that the plot is highly asymmetrical, with exactly the small studies with low effect size missing in the bottom-left corner of our plot. + +We can also display the name of each study using the `studlab` parameter. + +```{r} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 funnel(m.hksj,xlab = "g",studlab = TRUE) ``` @@ -88,13 +140,24 @@ img <- readPNG("funnel.PNG") grid.raster(img) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 We can see in the plot that while **some studies have statistically significant effect sizes** (blue background), other do not (white background). We also see a trend that while the moderately sized studies are partly significant and non-significant, with slightly more significant studies, the asymmetry is much larger for small studies. This gives us a hint that publication bias might indeed be present in our analysis. ### Testing for funnel plot asymmetry using Egger's test +<<<<<<< HEAD **Egger's test of the intercept** [@egger1997bias] quantifies the funnel plot asymmetry and performs a statistical test. We have prepared a function called `eggers.test` for you, which can be found below. The function is a wrapper for the `metabias` function in `meta`. Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. +======= +**Egger's test of the intercept** [@egger1997bias] quantifies the funnel plot asymmetry and performs a statistical test. + +We have prepared a function called `eggers.test` for you, which can be found below. The function is a wrapper for the `metabias` function in `meta`. + +Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} eggers.test<-function(data){ @@ -121,24 +184,34 @@ print(eggers.output) } ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 Now we can use the `eggers.test` function. We only have to specify our meta-analysis output `m.hksj` as the `data` the function should use. ```{r} eggers.test(data=m.hksj) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 The function returns the `intercept` along with its confidence interval. We can see that the `p-value` of Egger's test is **significant** ($p<0.05$), which means that there is substanital asymmetry in the Funnel plot. This asymmetry could have been caused by publication bias. ### Duval & Tweedie's trim-and-fill procedure {#dant} **Duval & Tweedie's trim-and-fill procedure** [@duval2000trim] is also based the funnel plot and its symmetry/asymmetry. When **Egger's test is significant**, we can use this method to estimate what **the actaul effect size would be had the "missing" small studies been published**. The procedure **imputes** missing studies into the funnel plot until symmetry is reached again. +<<<<<<< HEAD $~$ +======= +```{block,type='rmdinfo'} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **The trim-and-fill procedure includes the following five steps** [@schwarzer2015meta]: 1. Estimating the number of studies in the outlying (right) part of the funnel plot @@ -146,6 +219,7 @@ $~$ 3. This pooled effect is then taken as the center of all effect sizes 4. For each trimmed/removed study, a additional study is imputed, mirroring the effect of the study on the left side of the funnel plot 5. Pooling the results with the imputed studies and the trimmed studies included +<<<<<<< HEAD @@ -153,12 +227,21 @@ The **trim-and-fill-procedure** can be performed using the `trimfill` function i $~$ +======= +``` + +The **trim-and-fill-procedure** can be performed using the `trimfill` function in `meta`, and specifying our meta-analysis output. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} trimfill(m.hksj) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 We see that the procedure identified and trimmed **eight studies** `(with 8 added studies)`). The overall effect estimated by the procedure is $g = 0.34$. Let's compare this to our initial results. @@ -167,9 +250,15 @@ Let's compare this to our initial results. m.hksj$TE.random ``` +<<<<<<< HEAD The initial pooled effect size was $g = 0.59$, which is substantially larger than the bias-corrected effect. In our case, if we assume that **publication bias** was a problem in the analyses, the **trim-and-fill procedure** lets us assume that our initial results were **overestimated** due to publication bias, and the "true" effect when controlling for selective publication might be $g = 0.34$ rather than $g = 0.59$. If we store the results of the `trimfill` function in an **object**, we can also create **funnel plots including the imputed studies**. $~$ +======= +The initial pooled effect size was $g = 0.59$, which is substantially larger than the bias-corrected effect. In our case, if we assume that **publication bias** was a problem in the analyses, the **trim-and-fill procedure** lets us assume that our initial results were **overestimated** due to publication bias, and the "true" effect when controlling for selective publication might be $g = 0.34$ rather than $g = 0.59$. + +If we store the results of the `trimfill` function in an **object**, we can also create **funnel plots including the imputed studies**. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} m.hksj.trimfill<-trimfill(m.hksj) @@ -178,6 +267,7 @@ funnel(m.hksj.trimfill,xlab = "Hedges' g") ## P-Curve {#pcurve} +<<<<<<< HEAD In the last chapter, we showed you how you can apply **Egger's test of the intercept**, **Duval & Tweedie's trim and fill procedure**, and inspect **Funnel plots** in R. As we have mentioned before, recent research has shown **that the assumptions of the small-effect study methods may be inaccurate in many cases**. The **Duval & Tweedie trim-and-fill procedure** in particular has been shown to be prone to providing **inaccurate effect size estimates** [@simonsohn2014pb]. **P-curve Analysis** has been proposed as an alternative way to assess publication bias and estimate the true effect behind our collected data. P-Curve assumes that publication bias is not primarily generated because researchers **do not publish non-significant results**, but because the **"play" around with their data (e.g., selectively removing outliers, choosing different outcomes, controlling for different variables) until a non-significant finding becomes significant**. This (bad) practice is called **p-hacking**, and has been shown to be extremely frequent among researchers [@head2015extent]. $~$ @@ -186,22 +276,43 @@ $~$ You can find a great video describing the idea behing P-Curve here: https://www.youtube.com/embed/V7pvYLZkcK4 +======= +In the last chapter, we showed you how you can apply **Egger's test of the intercept**, **Duval & Tweedie's trim and fill procedure**, and inspect **Funnel plots** in R. + +As we have mentioned before, recent research has shown **that the assumptions of the small-effect study methods may be inaccurate in many cases**. The **Duval & Tweedie trim-and-fill procedure** in particular has been shown to be prone to providing **inaccurate effect size estimates** [@simonsohn2014pb]. + +**P-curve Analysis** has been proposed as an alternative way to assess publication bias and estimate the true effect behind our collected data. + +P-Curve assumes that publication bias is not primarily generated because researchers **do not publish non-significant results**, but because the **"play" around with their data (e.g., selectively removing outliers, choosing different outcomes, controlling for different variables) until a non-significant finding becomes significant**. This (bad) practice is called **p-hacking**, and has been shown to be extremely frequent among researchers [@head2015extent]. + +```{block,type='rmdinfo'} +**The idea behind P-Curve** + + +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ### Preparing RStudio and our data To use **P-curve**, we need our data in the same format as the one we used for the `metacont` function in [Chapter 3.1.1](#excel_preparation). We need to specify `Me`, `Se`, `Ne`, `Mc`, `Sc`, and `Nc`. My `metacont` data already has this format. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} library(knitr) load("metacont_data.RData") kable(metacont) ``` +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 To use p-curve, we also have to install and load **four packages**. The `esc` package [@esc], the `compute.es` package [@del2013compute], the `stringr` package, and the `poibin` package [@hong2011poibin]. ```{r,message=FALSE,warning=FALSE} @@ -211,11 +322,18 @@ library(stringr) library(poibin) ``` +<<<<<<< HEAD $~$ To **prepare our data** which we stored in the format described above, we need to use the `pcurve_dataprep` function, which we prepared for you. Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. $~$ +======= +To **prepare our data** which we stored in the format described above, we need to use the `pcurve_dataprep` function, which we prepared for you. + +Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} pcurve_dataprep<-function(data){ @@ -247,16 +365,22 @@ write(totoutput,ncolumns=1, file="input.txt") ``` +<<<<<<< HEAD $~$ To use the `pcurve_dataprep` function, we simply have to specify our dataset. In my case, this is `metacont`. $~$ +======= +To use the `pcurve_dataprep` function, we simply have to specify our dataset. In my case, this is `metacont`. + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} pcurve_dataprep(data=metacont) ``` +<<<<<<< HEAD $~$ The function gives us **the correct format of our effect sizes (t-values)** which we need to conduct the p-curve analysis. The function also **automatically stores a .txt-File (input.txt) in your working directory folder on your computer**. If you forgot where your working directory is, use the `getwd` function. @@ -269,11 +393,29 @@ getwd() $~$ +======= +The function gives us **the correct format of our effect sizes (t-values)** which we need to conduct the p-curve analysis. + +The function also **automatically stores a .txt-File (input.txt) in your working directory folder on your computer**. + +If you forgot where your working directory is, use the `getwd` function. + +```{r} +getwd() +``` + +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 This tells you the path to your working directory, where the **input.txt** file should be stored. ### Creating P-Curves +<<<<<<< HEAD To create p-curves, we have to use the `pcurve_app` function. The code for this function is very long, so it is not displayed here. The code for the function has been made publically available online by **Uri Simonsohn, Leif Nelson, and Joseph Simmons** and can be found [here](http://p-curve.com/app4/pcurve_app4.06.r). Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. +======= +To create p-curves, we have to use the `pcurve_app` function. The code for this function is very long, so it is not displayed here. The code for the function has been made publically available online by **Uri Simonsohn, Leif Nelson, and Joseph Simmons** and can be found [here](http://p-curve.com/app4/pcurve_app4.06.r). + +Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} @@ -988,14 +1130,30 @@ prop33=function(pc) } ``` +<<<<<<< HEAD To use the `pcurve_app` function, we have to specify that the function should use the **input.txt** in which the data we created using the `pcurve_dataprep` function is stored. We also have to provide the function with the **path to our folder/working directory where the file is stored**. Please note that while the standard way that **Windows** provides you with paths is using **backslashes**, we need to use **normal slashes** (**/**) for our designated path. ```{r,eval=FALSE} pcurve_app("input.txt","C://my_folder") +======= +To use the `pcurve_app` function, we have to specify that the function should use the **input.txt** in which the data we created using the `pcurve_dataprep` function is stored. We also have to provide the function with the **path to our folder/working directory where the file is stored**. + +```{block,type='rmdachtung'} +Please not that while the standard way that **Windows** provides you with paths is using **backslashes**, we need to use **normal slashes** (**/**) for our designated path. +``` + +```{r,eval=FALSE} +pcurve_app("input.txt","C://Users/Admin/Documents/R/WORKING_DIRECTORY/Meta-Analyse Buch/bookdown-demo-master") +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ``` The function automatically **creates and stores several files and figures into our folder/working directory**. Here the most important ones: +<<<<<<< HEAD +======= +

+ +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Figure 1: "input.png"** ```{r, echo=FALSE, fig.width=7} @@ -1007,19 +1165,29 @@ grid.raster(img) This figure shows you the **p-curve for your results (in blue)**. On the bottom, you can also find the number of effect sizes with $p<0.05$ which were included in the analysis. There are two tests displayed in the plot. +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **The test for right-skewness** If there is evidential value behind our data, the p-curve should be **right-skewed**. Through eyeballing, we see that this is pretty much the case here, and the **tests for the half and full p-curve** are also both **significant** ($p_{Full}<0.001, p_{Half}<0.001$). This means that the p-curve is heavily right-skewed, indicating that **evidential value is present in our data** +<<<<<<< HEAD $~$ +======= +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **The test for flatness** If there is evidential value behind our data, the p-curve should also **not be flat**. Through eyeballing, we see that this is pretty much the case here. The **tests for flatness** are both not significant ($p_{Full}=0.9887, p_{Binomial}=0.7459$). +<<<<<<< HEAD $~$ +======= +

+>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Figure 2: "input_fit.png"** @@ -1032,7 +1200,11 @@ grid.raster(img) This plot estimates the **power** behind our data, meaning if we have sufficient studies with sufficient participants to find a true effect if it exists. A conventional threshold for optimal power is **80%**, but P-curve can even assess evidential value if studies are **underpowered**. In our case, the the power estimate is **90%**. +<<<<<<< HEAD $~$ +======= +

+>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **Figure 3: "input_cumulative.png"** @@ -1047,7 +1219,15 @@ This plot provides **sensitivity analyses** in which the **highest and lowest p- ### Estimating the "true" effect of our data +<<<<<<< HEAD To estimate the **true effect of our data** with p-curve (much like the [Duval & Tweedie trim-and-fill procedure](#dant)), we can use the `plotloss` function described and made openly available by Simonsohn et al. [@simonsohn2014pb]. The code for this function is quite long, so it is not displayed here. It can accessed using this [link](https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/true_effect_estimation.R). Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. +======= +To estimate the **true effect of our data** with p-curve (much like the [Duval & Tweedie trim-and-fill procedure](#dant)), we can use the `plotloss` function described and made openly available by Simonsohn et al. [@simonsohn2014pb]. + +The code for this function is quite long, so it is not displayed here. It can accessed using this [link](https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/true_effect_estimation.R). + +Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r,echo=FALSE} @@ -1196,13 +1376,33 @@ df_obs<-(n1+n2)-2 } ``` +<<<<<<< HEAD For the `plotloss` function, we only have to provide the **data** to be used to find the "true" effect size underlying the data, and a **range of effect sizes in which the function should search for the true effect**, delineated by `dmin` and `dmax`. I will use my `metacont` data again here, and will search for the true effect between $d=0$ to $d=1.0$. $~$ +======= +For the `plotloss` function, we only have to provide the **data** to be used to find the "true" effect size underlying the data, and a **range of effect sizes in which the function should search for the true effect**, delineated by `dmin` and `dmax`. + +I will use my `metacont` data again here, and will search for the true effect between $d=0$ to $d=1.0$. +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{r} plotloss(data=metacont,dmin=0,dmax=1) ``` +<<<<<<< HEAD The function provides an effect estimate of $d=0.64$. It should be noted that this chapter should only be seen as **an introduction into p-Curve**, which should not be seen as comprehensive. Simonsohn et al. [@simonsohn2015better] also stress that P-Curve should only be used for **outcome data which was actually of interest for the Authors of the specific article**, because those are the one's likely to get p-hacked. They also ask meta-researchers to provide a **detailed table in which the reported results of each outcome data used in the p-Curve is documented** (a guide can be found [here](http://p-curve.com/guide.pdf)). It has also been shown that P-Curve's effect estimate are **not robust when the heterogeneity of a meta-analyis is high** (*I*^2^ > 50%). Van Aert et al. [@van2016conducting] propose **not to determine the "true" effect using P-Curve when heterogeneity is high** (defined as I^2^ > 50%). A poosible solution for this problem might be to **reduce the overall heterogeneity using outlier removal**, or to p-Curve results in **more homogeneous subgroups**. +======= +The function provides an effect estimate of $d=0.64$. + +```{block,type='rmdachtung'} +It should be noted that this chapter should only be seen as **an introduction into p-Curve**, which should not be seen as comprehensive. + +Simonsohn et al. [@simonsohn2015better] also stress that P-Curve should only be used for **outcome data which was actually of interest for the Authors of the specific article**, because those are the one's likely to get p-hacked. They also ask meta-researchers to provide a **detailed table in which the reported results of each outcome data used in the p-Curve is documented** (a guide can be found [here](http://p-curve.com/guide.pdf)). + +It has also been shown that P-Curve's effect estimate are **not robust when the heterogeneity of a meta-analyis is high** (*I*^2^ > 50%). Van Aert et al. [@van2016conducting] propose **not to determine the "true" effect using P-Curve when heterogeneity is high** (defined as I^2^ > 50%). + +A poosible solution for this problem might be to **reduce the overall heterogeneity using outlier removal**, or to p-Curve results in **more homogeneous subgroups**. +``` +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/09-risk_of_bias_summary.Rmd b/09-risk_of_bias_summary.Rmd index 908e413..be1ee42 100644 --- a/09-risk_of_bias_summary.Rmd +++ b/09-risk_of_bias_summary.Rmd @@ -4,7 +4,18 @@ ## Preparing your Risk of Bias data +<<<<<<< HEAD In many Meta-Analyses, you will also want to have a look at the quality of included studies using the [**RevMan Risk of Bias assessment tool**](https://handbook-5-1.cochrane.org/chapter_8/8_6_presentation_of_assessments_of_risk_of_bias.htm). However, many researchers don't use **RevMan** to conduct Meta-Analyses, one has to put some extra effort into inserting all study quality data by hand into RevMan, which is extremely time-consuming. Furthermore, the quality of the Risk of Bias (RoB) summary figures in RevMan are of suboptimal picture quality. Many journals will require figures with better quality, or figures saved in another format (such as **.svg** or **.pdf**). To avoid all of this, you can easily plot the **RoB Summary in RStudio yourself**. To do this, we again have to prepare an EXCEL sheet in which we store our RoB data. In [Chapter 3.1.1](#excel_preparation), we already described how the preparation and import of EXCEL sheets into RStudio works in general. For this data sheet, you need to follow a few guidelines: +======= +In many Meta-Analyses, you will also want to have a look at the quality of included studies using the [**RevMan Risk of Bias assessment tool**](https://handbook-5-1.cochrane.org/chapter_8/8_6_presentation_of_assessments_of_risk_of_bias.htm). However, many researchers don't use **RevMan** to conduct Meta-Analyses, one has to put some extra effort into inserting all study quality data by hand into RevMan, which is extremely time-consuming. + +Furthermore, the quality of the Risk of Bias (RoB) summary figures in RevMan are of suboptimal picture quality. Many journals will require figures with better quality, or figures saved in another format (such as **.svg** or **.pdf**). + +![This is the output created by RevMan](robsummaryrevman.png) + +To avoid all of this, you can easily plot the **RoB Summary in RStudio yourself**. +To do this, we again have to prepare an EXCEL sheet in which we store our RoB data. In [Chapter 3.1.1](#excel_preparation), we already described how the preparation and import of EXCEL sheets into RStudio works in general. For this data sheet, you need to follow a few guidelines: +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 * Name the first column **Author**, and put all the study names in this column (e.g., Frogeli et al.) * Give the other **columns a name signifying the RoB criterion** you assessed. Do this for all criteria you want to have included in your plot. **Important**: when naming your column, do not use spaces between word, but use underscores or points instead (e.g. allocation_concealment) diff --git a/10-network_metanalysis.Rmd b/10-network_metanalysis.Rmd new file mode 100644 index 0000000..3dda0e1 --- /dev/null +++ b/10-network_metanalysis.Rmd @@ -0,0 +1,111 @@ +# Network Meta-Analysis + +![](networks.jpg) + +Often when doing meta-analysis on the effectiveness of certain interventions, we are less interested if **one particular intervention is effective** (e.g., because it is quite well established that the intervention can be efficacious), but weather **one intervention is more or less effective than another type of intervention for some condition**. Yet, once we are interested in **head-to-head** comparisons between two treatments, we often face the problem that **only very few, if any, randomized controlled trials have compared the effects of two interventions directly**. This makes it very hard, if not impossible for us to conduct conventional meta-analyses to answer questions on the comparative effects of two or more interventions for one indication or outcome (e.g., different types of psychotherapy for major depression). + +Nevertheless, while direct comparisons between two or more interventions may often not exist, it is often the case that the interventions were evaluated in separate randomized controlled trials in which the intervention effects were compared to similar **control groups** (e.g., waitlist control groups, or placebos). This means that we do have **indirect comparisons** of the effects of different interventions, because they were compared to the same control condition. **Multiple-treatments meta-analysis (MTM)** is an extension of conventional meta-analysis which allows us to **incorporate indirect comparisons**, and thus the simultaneous analysis of several interventions. + +These meta-analysis methods are also referred to as **network meta-analyses**, as such methods allow for **multiple interventions comparisons to be integrated into our analysis**, which can be formalized as a **"network"** of comparisons. + +

+ +```{block,type='rmdinfo'} +**The idea behind network meta-analysis** + +Let's assume you have the results of two randomized controlled trials. One trial evaluated the effect of cognitive behavioral therapy (CBT) for depression to a control group. The second trial evaluted the effects of short-term psychodynamic therapy (STPP) on depression compared to control. We know the effect size $\hat\theta$ of both interventions which was found in the trial compared to control at post-test. These studies produce **indirect evidence** for the comparative effect of CBT versus STPP [@schwarzer2015meta]: + +$$\hat\theta_{CBT vs. STPP}^{indirect}=\hat\theta_{CBT vs. Control}^{direct} -\hat\theta_{STPP vs.Control}^{direct} $$ + +On the other hand, it may also be the case that we found one study **in which the effects of CBT were directly compared to the ones of STPP**. We will denote this effect as $\hat\theta_{CBT vs. STPP}^{direct}$. In network-meta-analysis, we want to integrate the **direct** as well as the **indirect** evidence to get the most precise effect estimate of the comparative effects. + +According to Schwarzer et al. [@schwarzer2015meta], there are two conditions which have to be met to conduct network meta-analyses: + +* The studies are **independent** +* The effect sizes are **consistent**. This means that effect sizes of interventions comparisons we attain through direct evidence should be similar to the one we get from indirect evidence (e.g., $\theta_{CBT vs. STPP}^{direct}=\theta_{CBT vs. STPP}^{indirect}$). Inconsistency, on the other hand, is $\theta_{CBT vs. STPP}^{direct}-\theta_{CBT vs. STPP}^{indirect0} \not= 0$. Assessing and dealing with inconsistency is highly important in network meta-analysis. + +Below, you can see a simple first network of the comparisons between the control condition and the two interventions. We could also think of a network where **some comparisons are simply not available**, as is the case in the second network. +``` + + + + +```{r,echo=FALSE,fig.width=4, fig.height=3, fig.align='center',message=FALSE,fig.cap="A simple network"} +library(sna) +library(ggplot2) +library(GGally) +library(network) + # random graph +net = rgraph(3, mode = "graph", tprob = 1) +net = network(net, directed = FALSE) + +# vertex names +network.vertex.names(net) = c("CBT","STPP","Control") +ggnet<-ggnet2(net, size=12, node.color = "lightblue", label = c("CBT","STPP","Control"),label.size=3,node.size = 8, label.color = "black") +ggnet + +``` + +```{r, echo=FALSE, fig.width=4,fig.height=3,fig.cap="A simple network with one missing comparison",fig.align='center'} +library(png) +library(grid) +img <- readPNG("networkgraph.png") +grid.raster(img) +``` + + + + + + + +```{block,type='rmdinfo'} +**Work in progress** + +A full version of this Chapter will be available soon. + +``` + + +```{r,message=FALSE} +library(netmeta) +library(plyr) +``` + + +```{r,echo=FALSE} +load("netmetdata.RData") +``` + +```{r,eval=FALSE} +netmetdata +``` + +```{r,echo=FALSE} +library(knitr) +kable(netmetdata) +``` + + +Results + +```{r} +netmet <- netmeta(TE, seTE, treat1, treat2, studlab=paste(Author), data=netmetdata, sm="SMD",reference.group = "Waitlist") +netmet +``` + +```{r} +netgraph(netmet,seq = c("Waitlist","Gestalt","Psychodynamic","MCT","MBCT","Placebo","CBT","System","Psychoanalysis","ACT")) + +netgraph(netmet, start="circle", iterate=TRUE, col="darkgray", cex=1.5, multiarm=TRUE, points=TRUE, col.points="blue", cex.points=3) +``` + +```{r} +forest(netmet, xlim=c(-1.5, 0.5), ref="Waitlist", leftlabs="Contrast to Waitlist", xlab="Effect on Depression (SMD)",sortvar = TE, smlab = "") + +forest(netmet, xlim=c(-1.5, 0.5), ref="Placebo", leftlabs="Contrast to Placebo", xlab="Effect on Depression (SMD)",sortvar = TE, smlab = "") +``` + + + + diff --git a/11-multilevel-meta-analysis.Rmd b/11-multilevel-meta-analysis.Rmd new file mode 100644 index 0000000..67ec7b9 --- /dev/null +++ b/11-multilevel-meta-analysis.Rmd @@ -0,0 +1,289 @@ +# "Multilevel" Meta-Analysis + +![](multilevel.jpg) + +$$ $$ + +This chapter deals with the topic of pooling effect sizes in **"multilevel" meta-analyses**. You probably wonder why we put the word "multilevel" into **quotation marks**. There is an increasing number of studies describing themselves as "multilevel" meta-analyses, which insinuates that a "multilevel" meta-analysis would be something special or extraordinary compared to "standard" meta-analyses. This, however, is not true, as every meta-analytic model presupposes a multilevel structure within the data to pool results [@pastor2018multilevel]. This means that basically, once you've reached this chapter in the guide, you will have already fitted a multilevel meta-analytic model to data several times (maybe even without knowing). When people talk about multilevel meta-analysis, what they often think of are **three-level meta-analytic models**, which are indeed somewhat different to the [fixed-effect](#fixed) and [random-effects models](#random) we presented before. + +```{block, type='rmdinfo'} +**The multilevel nature of Meta-Analysis** + +To see why meta-analysis is by nature multileveled, let us go back to the formula for the random-effects model we discussed in [Chapter 4.2](#random): + +$$\hat\theta_k = \mu + \epsilon_k + \zeta_k$$ + +We discussed that the terms $\epsilon_k$ and $\zeta_k$ are introduced in a random-effects model because we assume that there are two sources of variability. The first one is caused by the **sampling error** ($\epsilon_k$) of individual studies, which causes their effect size estimates to deviate from the true effect size $\theta_k$. The second one, $\zeta_k$ is the between-study heterogeneity caused by the fact that the true effect size of a study $k$ itself is again only part of an **overarching distribution of true effect sizes**, from which the individual true effect size $\theta_k$ was drawn. Our aim in the random-effects model is therefore to estimate the mean of this distribution of true effect sizes, $\mu$. + +The two error terms correspond with the two levels within our meta-analysis data: the **"participant" level** (level 1) and the **"study" level** (level 2). The figure below symbolizes this structure. In the lower level (level 1), we have the participants (and/or patients). These participants are, of course, part of a number of larger units, the studies (level 2), from which we retrieved the data for our meta-analysis. While the data on level 1 usually already reaches us, the meta-analysts, in a "pooled" form (e.g., the authors of the paper provide us with the mean and standard deviation of their studied sample instead of the raw data), pooling on level 2, the study level, has to be performed in meta-analysis. In the tradition of multilevel modeling, such data is called **nested data**: in most meta-analyses, one can say that **participants are "nested" within studies**. + +![A simplified illustration of the multilevel structure of conventional meta-analytic models](multilevel-model.png) + +Thus, if we split the formula from above into the components corresponding to the two levels, we get the following formulae: + +**Level 1 (participants) model:** + +\begin{equation} + \label{eq:1} + \hat\theta_k = \theta_k + \epsilon_k +\end{equation} + +**Level 2 (studies) model:** + +\begin{equation} + \label{eq:2} + \theta_k = \mu + \zeta_k +\end{equation} + +You might have already detected that we can substitute $\theta_k$ in the first equation with its definition in the second equation. What we then get is **exactly the generic formula for the meta-analytic model from before** (even a fixed-effects model can be defined as having a multilevel structure, yet with $\zeta_k$ being zero). Thus, it becomes evident that the way we define a meta-analytic model already has multilevel properties "built-in" given that we assume that participants are nested within studies in our data. + +``` + +```{block, type='rmdinfo'} +**Three-level meta-analytic models** + +It has become clear that meta-analysis naturally possesses a multilevel structure. This allows us to expand this structure even further to better reflect our data. **Three-level models** [@cheung2014modeling; @assink2016fitting] are a good way to do this. + +**Statistical independence** is one of the core assumptions of meta-analytic pooling [@hedges2009statistical]. If there is a dependency between effect sizes (i.e., the effect sizes are correlated), this may artificially reduce heterogeneity and thus lead to false-positive results. + +Dependence may stem from different sources [@cheung2014modeling]: + +1. **Dependence introduced by the authors of the individual studies**. For example, scientists conducting the study might have collected data from multiple sites, or they might have compared [multiple interventions to one single control group](#i), or they might have used different questionnaires to measure an outcome of interest (e.g., they used the BDI as well as the PHQ-9 to assess depression in patients). For all of these scenarios, we can assume that some kind of dependency is introduced within the reported data. +2. **Dependence introduced by the meta-analyst herself**. For example, a meta-analysis could focus on a specific psychological mechanism and include studies which were conducted in different cultural regions of the world. It seems reasonable that the reported results are more similar when studies were conducted in the same cultural context. + +We can take such dependencies into account by integrating a **third layer** into the structure of our meta-analysis model. For example, one could model that different questionnaires are nested within studies. Or one could create a model in which studies are nested within cultural regions. This creates a three-level meta-analytic model, as illustrated by the figure below. + +![](multilevel-model2.png) + +The generic mathematical formulae for a three-level meta-analytic model look like this: + +**Level 1 model** + +$$\hat\theta_{ij} = \theta_{ij} + \epsilon_{ij}$$ + +**Level 2 model** + +$$\theta_{ij} = \kappa_{j} + \zeta_{(2)ij}$$ + +**Level 3 model** + +$$\kappa_{j} = \beta_{0} + \zeta_{(3)j}$$ + +Where $\theta_{ij}$ is the **true effect size**, $\hat\theta_{ij}$ its estimator in the $i$th effect size in cluster $j$, $\kappa_{j}$ the **average effect size** in $j$ and $\beta_0$ the average population effect. Again, we can piece these formulae together and get this: + +$$\hat\theta_{ij} = \beta_{0} + \zeta_{(2)ij} + \zeta_{(3)j} + \epsilon_{ij}$$ + + +``` + +The `metafor` package is particularly well suited for fitting various three-level models in meta-analyses. When setting up meta-analytic models, we have previously used the `meta` function primarly, because we think that this package's code is a little less technical. `metafor`, however, is also fairly easy to use once you prepared the data correctly. We'll get to this in the next chapter. + +## Fitting a three-level model + +### Data preparation + +Before we start with fitting the model, we need the `metafor` package to be loaded from our library. + +```{r, message=FALSE, warning=FALSE} +library(metafor) +``` + +For this chapter, i will use the `mlm.data` dataset. It has already been prepared for model fitting using `metafor`. Let's have a peek at the dataset: + +```{r, echo=FALSE} +load("mlm.data.RData") +``` + +```{r} +mlm.data +``` + +We see that the dataset has 80 rows and 5 columns (i.e., 5 variables). The first four are the most important ones: + +* **ID_1** and **ID_2**. In these two columns, we stored the identifiers corresponding to the individual effect sizes and clusters on different levels of our multilevel model. We have left these columns quite generic to make clear that such multilevel data can accommodate various types of data structures. For example, one can think of a scenario where ID_2 corresponds with the different effect sizes we find within a study (e.g., different effect sizes for different measures of depression). In this example, the effect sizes in ID_2 would then be nested within the larger clusters defined in ID_1, the studies. Another possible example would be that ID_2 in fact symbolizes the unique identifier for single studies, which are then nested in larger clusters (e.g., cultural regions) shown in ID_1. +* **y** and **v**. In these columns, the effect size data is stored. The format is similar to the one we know from the `metagen` function in `meta`. The column `y` contains the effect size ($d$, $g$, $r$, $z$, or other continuous effect size metrics). The `v` column contains the sampling variance which can be obtained by **squaring the standard error**. + +There are two additional columns which contain data needed for **subgroup analyses**. To conduct subgroup analyses with `metafor`, however, we need to store the data a little differently than in [Chapter 7](#subgroup). This time, we do not use a factor vector containing the subgroup levels as strings, but **recode all subgroup levels** as individuals columns in the data. In our case, we are interested if the publication status has a moderating effect on the pooled results. We want to compare studies published in peer-review journals to dissertations. To do this, we need to create **two variables/columns for each subgroup level** (peer-review, dissertation). In each column we provide a code if the effect size belongs to this subgroup, defined by `1 = yes` and `0 = no`. Of course, the subgroup codings now have to be **mutually exclusive**, as we cannot calculate subgroup effects if a study is both part of the "peer-review" group and the "dissertation" group. + +### Model fitting + +We are now prepared to fit our model. I will save the model to the object `full.model`. For model fitting, we use the `rma.mv` function in `metafor`. The function has plenty of parameters one can specify (type `?metafor::rma.mv` in the console to see them all). For now, we will only focus on the really essential ones. + +```{r,echo=FALSE, warning=FALSE, message=FALSE} +library(knitr) +Code<-c("y", "v", "random", "tdist", "data", "method") +Description<-c("The variable in our dataset containing the effect size data", "The variable in our dataset containing the sampling variance data corresponding to each y", "The model specifications for our multilevel model. We describe this in more detail below" , "This is an old friend in disguise, the Knapp-Hartung adjustment for our confidence intervals (if set to TRUE)", "The dataframe containing our data", "The tau-squared estimator. Our options here are limited, and it is advised to use the restricted maximum likelihood ('REML') method, which is the default") +m<-data.frame(Code,Description) +names<-c("Code","Description") +colnames(m)<-names +kable(m) +``` + +**Setting up the formula** + +Under the `random` parameter, we feed `rma.mv` with the formula for our model. As we actually have two formulas in three-level models, we have to bundle them together in a `list()`. The notation for `rma.mv` is very similar to other packages specialized for mixed-effects models such as `lme4` [@lme4]. + +Within the `list()`, formulae are separated with commas. Within the formula, we first define the random effects variance as `~ 1`, followed by a **vertical bar** `|`. Behind the vertical bar, we assign this random effect to a **grouping variable** such as studies, measures, regions and so forth. We only have to define the formulae for **level 2** and **level 3**, as the sampling error in level 1 is assumed to be known. We type in the formulae in the order of our multilevel model. As in our example, ID_2 is nested in ID_1, we first define level 2 as `~ 1 | ID_2` and then level 3 as `~ 1 | ID_1`. Now, let's put it all together: + + +```{r} +full.model<-rma.mv(y, + v, + random = list(~ 1 | ID_2, + ~ 1 | ID_1), + tdist = TRUE, + data = mlm.data, + method = "REML") + +``` + +To see the results of our model, we can use the `summary()` function. + +```{r} +summary(full.model) +``` + +Let's go through the output one by one. First, let's have a look at the `Variance Components`. Here, we see the variance calculated for each level of our model, $\sigma^2_1$ (level 2) and $\sigma^2_2$ (level 3). Under `nlvls`, we see the number of levels (i.e., subgroups) on each level. For `ID_2`, this is 80 because each effect size has its own ID. + +Under `Model Results`, we see the `estimate` of our pooled effect, which is 0.4345. As our `y` column contained effect sizes expressed as Hedges' g, the effect is therefore $g = 0.4545$, with a confidence interval of $0.20-0.67$. + +Under `Test for Heterogeneity`, we see that the variance of effect sizes within our data overall was highly significant ($p < 0.0001$). This however, is not very informative as we are interested how variance can be attributed to the different levels in our model. + + +### Distribution of total variance + +As mentioned before, what we're actually interested in is the **distribution of variance over the three levels of our model**. Cheung (2014) provides a formula to estimate the sampling variance, which is needed to calculate the variance proportions for each level. We have have programmed a function called `variance.distribution.3lm`, which implements this formula. The code for the function can be seen below. Parts of this code were adapted from Assink and Wibbelink [@assink2016fitting]. + +R doesn't know this function yet, so we have to let R learn it copying and pasting the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. + +```{r, fig.height=3} +variance.distribution.3lm <- function(data, m){ + + # Calculate estimated sampling variance and proportions across levels + data <- data + m <- m + n <- length(data$v) + vector.inv.var <- 1/(data$v) + sum.inv.var <- sum(vector.inv.var) + sum.sq.inv.var <- (sum.inv.var)^2 + vector.inv.var.sq <- 1/(data$v^2) + sum.inv.var.sq <- sum(vector.inv.var.sq) + num <- (n-1)*sum.inv.var + den <- sum.sq.inv.var - sum.inv.var.sq + est.samp.var <- num/den + level1<-((est.samp.var)/(m$sigma2[1]+m$sigma2[2]+est.samp.var)*100) + level2<-((m$sigma2[1])/(m$sigma2[1]+m$sigma2[2]+est.samp.var)*100) + level3<-((m$sigma2[2])/(m$sigma2[1]+m$sigma2[2]+est.samp.var)*100) + Level<-c("level 1", "level 2", "level 3") + Variance<-c(level1, level2, level3) + df<-data.frame(Level, Variance) + df1<-df + colnames(df1) <- c("Level", "% of total variance") + + # Generate plot + df$distribution<-"Distribution" + df$Level<-factor(df$Level, levels(df$Level)[c(3,2,1)]) + g <- ggplot(df, aes(fill=Level, y=Variance, x=distribution)) + + geom_bar(stat="identity", position="fill", width = 0.3) + coord_flip(ylim = c(1,0)) + scale_y_continuous(labels = scales::percent)+ + theme(axis.title.x=element_blank(), + axis.title.y=element_blank(), + axis.ticks.y=element_blank(), + axis.text.y = element_blank(), + axis.line.x = element_line(colour = "black", + size = 0.5, linetype = "solid"), + legend.position = "bottom", + panel.grid.major = element_blank(), + panel.grid.minor = element_blank(), + panel.background = element_blank(), + legend.background = element_rect(linetype="solid", + colour ="black"), + legend.title = element_blank(), + legend.key.size = unit(0.75,"cm")) + guides(fill = guide_legend(reverse = TRUE)) + return(list(g, df1)) +} +``` + +Please be aware that the function needs `ggplot2` installed and loaded to work. There are only two parameters: `data`, the data we used for our three-level model (in which `v` signifies the effect size), and `m`, the model we just fitted using `rma.mv`. Let's have a look: + +```{r, fig.height=3} +library(ggplot2) +variance.distribution.3lm(data = mlm.data, + m = full.model) +``` + + +From the outputs, we see that about 8.94% of the overall variance can be attributed to level 1, 23.92% to level 2, and as much as 67.15% to level 3. + +### Comparing the fit + +Of course, when fitting a three-level model, we are interested if this model actually **captures the variability in our data better than a two-level model**. `metafor` allows us to easily do this by comparing models **in which one of the levels is removed**. + +To do this, we can use the `rma.mv` function again, but this time, we hold the variance component $\sigma^2$ of one level constant. This can be done by specifying the `sigma2` parameter. The parameter can be specified by providing a vector telling the function which level to hold constant, with the generic version being `c(level 2, level3)`. So, if we want to hold level 2 constant while leaving the rest as is, we use `sigma2 = c(0,NA)`, and vice verse for the third level. + +**Model without level 2** + +```{r} +model.l2.removed<-rma.mv(y, + v, + random = list(~ 1 | ID_2, + ~ 1 | ID_1), + tdist = TRUE, + data = mlm.data, + method = "REML", + sigma2 = c(0, NA)) +summary(model.l2.removed) +``` + +We see that the `Model Results` have changed, but does the second level in fact capture a significant amount of variability in the data? To check, we can use the `anova` function to compare the `full.model` to the `model.l2.removed`. + +```{r} +anova(full.model, model.l2.removed) +``` + +We see that the `Full` model compared to the `Reduced` model does indeed have a better fit, with the **Akaike Information Criterion** (`AIC`) and the **Bayesian Information Criterion** (`BIC`) being lower for this model. The difference is significant ($p<0.0001$), suggesting that it was a good idea to include this level into our analysis. Now, let's do the same when holding level 3 constant. + +**Model without level 3** + +```{r} +model.l3.removed<-rma.mv(y, + v, + random = list(~ 1 | ID_2, + ~ 1 | ID_1), + tdist = TRUE, + data = mlm.data, + method = "REML", + sigma2 = c(NA, 0)) +summary(model.l3.removed) + +anova(full.model, model.l2.removed) + +``` + +We see the same pattern here, suggesting that overall, our three-level model is very useful. + +## Subgroup Analyses in Three-Level Models + +Once our **three-level meta-analytic model** is set, it is easy to also check for moderators of the overall effect, i.e. conduct subgroup analyses. To do this, we can use the `mods` parameter in `rma.mv`. The subgroup/moderator can be specified by putting a tilde (`~`) in front of it. In our case, we want to check if dissertations report higher or lower effect sizes, so we use `mods = ~ dissertation`. Here's the full code: + +```{r} +model.mods<-rma.mv(y, + v, + random = list(~ 1 | ID_2, + ~ 1 | ID_1), + tdist = TRUE, + data = mlm.data, + method = "REML", + mods = ~ dissertation) +summary(model.mods) + +``` + +The first important output is the `Test of Moderators`. We see that $F_{1,78}=0.0226$ with $p=0.8808$, meaning that there is no significant difference between subgroups. The `Model Results` are printed within a meta-regression framework, so we **cannot simply extract the estimates directly to attain the summary effect size within subgroups**. The first row, the `intrcpt` (intercept) is the value of $g$ when dissertation = 0, i.e., the value for peer-review articles. The predictor `dissertation`'s estimate is expressed as a **slope**, meaning that we have to add its value to the intercept to get the actual summary effect. This is: + +$$g_{dissertation} = \beta_0 + \beta_{dissertation} = 0.4297+0.0339=0.4636$$ + +$$ $$ + + + + diff --git a/12-Effectsizeconverter.Rmd b/12-Effectsizeconverter.Rmd new file mode 100644 index 0000000..25ef60c --- /dev/null +++ b/12-Effectsizeconverter.Rmd @@ -0,0 +1,338 @@ +# Effect size calculators + +![](effect.jpg) + +Although the `meta` package can calculate all **individual effect sizes for every study** if we use the `metabin` or `metacont` function, a frequent scenario is that **some papers do not report the effect size data in the right format**. Especially older articles may often only report results of $t$**-tests**, **ANOVAs**, or $\chi^2$**-tests**. If enough data is reported, we can also use **such outcome formats to calculate effect sizes**. This way, we can calculate the **effect size (e.g., Hedges' g)** and the **Standard Error (SE)**, which we can then use in a meta-analysis with **pre-calculated effect sizes** using the `metagen` function (see [Chapter 4.1.1](#pre.calc)). + +```{block,type='rmdinfo'} +**Hedges' g** + +When dealing with **continuous outcome data**, it is conventional to calculate the **Standardized Mean Difference** (SMD) as an outcome for each study, and as your **summary measure** [@borenstein2011]. + +A common format to to calculate the SMD in single trials is **Cohen's d** [@cohen1988statistical]. Yet, this summary measure has been **shown to have a slight bias in small studies, for which it overestimates the effect** [@hedges1981distribution]. + +**Hedges *g* ** is a similar summary measure, but it **controls for this bias**. It uses a slightly different formula to calculate the pooled variance $s_{pooled}$, $s*_{pooled}$. The transformation from *d* to *g* is often performed using the formula by Hedges and Olkin [@hedges1985statistical]. + +$$g \simeq d\times(1-\frac{3}{4(n_1+n_2)-9}) $$ + +``` + +```{block,type='rmdachtung'} +Hedges' g is **commonly used in meta-analysis**, and it's the standard output format in **RevMan**. Therefore, we highly recommend that you also use this measure in you meta-analysis. + +In `meta`'s `metabin` and `metacont` function, Hedges' g is automatically calculated for each study if we set `sm="SMD"`. If you use the `metgen` function, however, you should calculate Hedges' g for each study yourself first. +``` + +To calculate the effect sizes, we will use Daniel Lüdecke's extremely helpful `esc` package [@esc]. So, please **install this package first** using the `install.packages("esc")` command, and then load it in you library. + + +```{r,message=FALSE} +library(esc) +``` + +

+ +**Here's an overview of all calculators covered in this guide** + +1. [Calculating Hedges' *g* from the Mean and SD](#a) +2. [Calculating Hedges' *g* from a regression coefficient](#b) +3. [Calculating an Odd's Ratio from *Chi-square*](#c) +4. [Calculating Hedges' *g* from a one-way ANOVA](#d) +5. [Calculating Hedges' *g* from the Mean and SE](#e) +6. [Calculating Hedges' *g* from a correlation](#f) +7. [Calculating Hedges' *g* from an independent t-test](#g) +8. [Calculating Hedges' *g* from Cohen's *d*](#h) +9. [Calculating effect sizes for studies with multiple comparisons](#i) + +

+ +## Calculating Hedges' g from the Mean and SD {#a} + +To calculate Hedges' *g* from the *Mean*, *Standard Deviation*, and $n_{group}$ of both trial arms, we can use the `esc_mean_sd` function with the following parameters. + + +* `grp1m`: The **mean** of the **first group** (e.g., the intervention). +* `grp1sd`: The **standard deviation** of the **first group**. +* `grp1n`: The **sample size** of the **first group**. +* `grp2m`: The **mean** of the **second group**. +* `grp2sd`: The **standard deviation** of the **second group**. +* `grp2n`: The **sample size** of the **second group**. +* `totalsd`: The **full sample standard deviation**, if the standard deviation for each trial arm is not reported +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + +

+ +**Here's an example** + +```{r} +esc_mean_sd(grp1m = 10.3, grp1sd = 2.5, grp1n = 60, +grp2m = 12.3, grp2sd = 3.1, grp2n = 56, es.type = "g") +``` + + + + +## Calculating Hedges' *g* from a regression coefficient {#b} + +### Unstandardized regression coefficients + +It is also possible to calculate **Hedges' *g* ** from an unstandardized or standardized regression coeffiecent [@lipsey2001practical]. + +For **unstardardized coefficients**, we can use the `esc_B` function with the following parameters: + +* `b`: unstandardized coefficient $b$ (the "treatment" predictor). +* `sdy`: the standard deviation of the dependent variable $y$ (i.e., the outcome). +* `grp1n`: the number of participants in the first group. +* `grp2n`: the number of participants in the second group. +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + +

+ +**Here's an example** + +```{r} +esc_B(b=3.3,sdy=5,grp1n = 100,grp2n = 150,es.type = "g") +``` + +

+ +### Standardized regression coefficents + +Here, we can use the `esc_beta` function with the follwing parameters: + +* `beta`: standardized coefficient $\beta$ (the "treatment" predictor). +* `sdy`: the standard deviation of the dependent variable $y$ (i.e., the outcome). +* `grp1n`: the number of participants in the first group. +* `grp2n`: the number of participants in the second group. +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + +

+ +**Here's an example** + +```{r} +esc_beta(beta=0.7, sdy=3, grp1n=100, grp2n=150, es.type = "g") +``` + +## Calculating an Odd's Ratio from *Chi-square* {#c} + +To calculate the **Odd's Ratio** (or any other kind of effect size measure) from $\chi^2$ using the `esc_chisq` function with the following paramters: + +* `chisq`: The value of Chi-squared (or only `p`) +* `p`: the chi squared p or phi value (or only `chisq`) +* `totaln`: total sample size +* `es.type`: the summary measure (in our case, `"cox.or"`) + +

+**Here's an example** + +```{r} +esc_chisq(chisq=9.9,totaln=100,es.type="cox.or") +``` + + +## Calculating Hedges' *g* from a one-way ANOVA {#d} + +We can also derive the SMD from the $F$-value of a **one-way ANOVA with two groups**. Such ANOVAs can be detected if you look for the **degrees of freedom** ($df$) underneath of $F$. In a one-way ANOVA with two groups, the degrees of freedom should always start with $1$ (e.g. $F_{1,147}=5.31$). The formula for this transformation looks like this [@cohen1992power;@rosnow1996computing;@rosnow2000contrasts]: + +$$d = \sqrt{ F(\frac{n_t+n_c}{n_t n_c})(\frac{n_t+n_c}{n_t+n_c-2})}$$ + +To calculate **Hedges' g** from $F$-values, we can use the `esc_f` function with the following parameters: + + +* `f`: *F*-value of the ANOVA +* `grp1n`: Number of participants in group 1 +* `grp2n`: Number of participants in group 2 +* `totaln`: The total number of participants (if the *n* for each group is not reported) +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + + +

+ +**Here's an example** + +```{r} +esc_f(f=5.04,grp1n = 519,grp2n = 528,es.type = "g") +``` + + +## Calculating Hedges' *g* from the Mean and SE {#e} + +When calculating **Hedges' g** from the **Mean** and **Standard Error**, we simply make use of the fact that the Standard error is not much more than the **Standard Deviation** when the sample size is taken into account [@thalheimer2002calculate]: + +$$SD = SE\sqrt{n_c}$$ + +We can calculate **Hedges' g** using the `esc_mean` function with the following parameters: + +* `grp1m`: The mean of the first group. +* `grp1se`: The standard error of the first group. +* `grp1n`: The sample size of the first group. +* `grp2m`: The mean of the second group. +* `grp2se`: The standard error of the second group. +* `grp2n`: The sample size of the second group. +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + +

+ +**Here's an example** + +```{r} +esc_mean_se(grp1m = 8.5, grp1se = 1.5, grp1n = 50, + grp2m = 11, grp2se = 1.8, grp2n = 60, es.type = "g") +``` + +## Calculating Hedges' *g* from a correlation {#f} + +For **equally sized groups** ($n_1=n_2$), we can use the following formula to derive the SMD from the pointbiserial **correlation** [@rosenthal1984meta]. + +$$r_{pb} = \frac{d}{\sqrt{d^2+4}}$$ +And this formula for **unequally sized groups** [@aaron1998equating]: + +$$r_{pb} = \frac{d}{\sqrt{d^2+ \frac{(N^2-2 \times N)}{n_1 n_2} }}$$ +To convert $r_{pb}$ to **Hedges' g**, we can use the `esc_rpb` function with the following parameters: + + +* `r`: The *r*-value. Either *r* or its *p*-value must be given. +* `p`: The *p*-value of the correlation. Either *r* or its *p*-value must be given. +* `grp1n`: The sample size of group 1. +* `grp2n`: The sample size of group 2. +* `totaln`: Total sample size, if `grp1n` and `grp2n` are not given. +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + + +```{r} +esc_rpb(r = 0.25, grp1n = 99, grp2n = 120, es.type = "g") +``` + + +## Calculating Hedges' *g* from an independent t-test {#g} + +The SMD can also be derived from an **independent t-test value** with the following formula [@thalheimer2002calculate]: + +$$d = \frac {t(n_1+n_2)}{\sqrt{(n_1+n_2-2)(n_1n_2)}}$$ + +We can calculate **Hedges' g** from a **t-test** using the `esc_t` function with the following paramters: + +* `t`: The t-value of the t-test. Either *t* or its *p*-value must be given. +* `p`: The *p*-value of the t-test. Either *t* or its *p*-value must be given. +* `grp1n`: The sample size of group 1. +* `grp2n`: The sample size of group 2. +* `totaln`: Total sample size, if `grp1n` and `grp2n` are not given. +* `es.type`: the **effect measure** we want to calculate. In our case this is `"g"`. But we could also calculate Cohen's *d* using `"d"`. + +

+ +**Here's an example** + + +```{r} +esc_t(t = 3.3, grp1n = 100, grp2n = 150,es.type="g") +``` + + +## Calculating Hedges' *g* from Cohen's *d* {#h} + +We can also directly correct **Cohen's *d* ** and thus generate **Hedges' g** using the formula by Hedges and Olkin [@hedges1985statistical]: + + +$$g \simeq d\times(1-\frac{3}{4(n_1+n_2)-9}) $$ +This can be done in R using the `hedges_g` function with the following parameters: + +* `d`: The value of **Cohen's d** +* `totaln`: the total *N* in the study + + +```{r} +hedges_g(d = 0.75, totaln = 50) +``` + +## Calculating effect sizes for studies with multiple comparisons {#i} + +![](stack.jpg) + +Many randomized-controlled trials do not only include a single **intervention** and **control group**, but compare the effect of **two or more interventions** to a control group. It might be tempting in such a scenario to **simply include all the comparisons between the intervention groups and control within a study into one meta-analysis**. Yet, researchers should abstain from this practice, as this would mean that the control group is used twice for the meta-analysis, thus **"double-counting"** the participants in the control group. This results in a **unit-of-analysis** error, as the effect size are correlated, and thus not independent, but are treated as if they would stem from independent samples. + +**There are two ways to deal with this:** + +* Splitting the N of the control group: One method to control for the unit-of-analysis error to some extent would be to **split** the number of participants in the control group between the two intervention groups. So, if your control group has $N=50$ participants, you could divide the control group into two control groups with he same mean and standard deviation, and $N=25$ participants each. After this preparation step, you could calculate the effect sizes for each intervention arm. As this procedure only partially removes the unit of analysis error, it is not generally recommended. A big plus of this procedure, however, is that it makes [**investigations of hetereogeneity**](#heterogeneity) between study arms possible. +* Another option would be to **synthesize the results of the intervention arms** to obtain one single comparison to the control group. Despite its practical limitations (sometimes, this would mean synthesizing the results from extremely different types of interventions), this procedure does get rid of the unit-of-analysis error problem, and is thus recommended from a statistical standpoint. The following calculations will deal with this option. + +To synthesize the **pooled effect size data** (pooled Mean, Standard Deviation and N), we have to use the following formula: + + +$$N_{pooled}=N_1+N_2$$ + +$$M_{pooled}=\frac{N_1M_1+N_2M_2}{N_1+N_2}$$ + +$$SD_{pooled} = \sqrt{\frac{(N_1-1)SD^{2}_{1}+ (N_2-1)SD^{2}_{2}+\frac{N_1N_2}{N_1+N_2}(M^{2}_1+M^{2}_2-2M_1M_2)} {N_1+N_2-1}}$$ + +As these formulae are quite lengthy, we prepared the function `pool.groups` for you, which does the pooling for you automatically. Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. + +```{r} +pool.groups<-function(n1,n2,m1,m2,sd1,sd2){ + +n1 <- n1 +n2 <- n2 +m1 <- m1 +m2 <- m2 +sd1 <- sd1 +sd2 <- sd2 + +Npooled <- n1+n2 +Mpooled <- (n1*m1+n2*m2)/(n1+n2) +SDpooled <- sqrt(((n1-1)*sd1^2+(n2-1)*sd2^2+(((n1*n2)/(n1+n2))*(m1^2+m2^2-2*m1*m2)) )/(n1+n2-1)) + +return(data.frame(Mpooled,SDpooled,Npooled)) +} +``` + +**To use this function, we have to specifiy the following parameters:** + +* `n1`: The N in the first group +* `n2`: The N in the second group +* `m1`: The Mean of the first group +* `m2`: The Mean of the second group +* `sd1`: The Standard Deviation of the first group +* `sd2`: The Standard Deviation of the second grop + +**Here's an example** + +```{r} +pool.groups(n1=50, + n2=50, + m1=3.5, + m2=4, + sd1=3, + sd2=3.8) +``` + +```{block, type='rmdinfo'} +**What should i do when an study has more than two intervention groups** + +If a study has more than one two intervention groups you want to synthesize (e.g. four arms, with three distinct intervention arms), you can **pool the effect size data for the first two interventions**, and then **synthesize the pooled data you calculated with the data from the third group**. + +This is fairly straightforward if you save the output from `pool.groups` as an object, and then use the `$` operator: +``` + +First, pool the **first** and **second intervention group**. I will save the output as `res`. + +```{r} +res<-pool.groups(n1 = 50, + n2 = 50, + m1 = 3.5, + m2 = 4, + sd1 = 3, + sd2 = 3.8) +``` + +Then, use the pooled data saved in `res` and **pool it with the data from the third group**, using the `$` operator to access the different values saved in `res`. + +```{r} +pool.groups(n1 = res$Npooled, + n2 = 60, + m1 = res$Mpooled, + m2 = 4.1, + sd1=res$SDpooled, + sd2 = 3.8) +``` + diff --git a/13-power_analysis.Rmd b/13-power_analysis.Rmd new file mode 100644 index 0000000..df1263e --- /dev/null +++ b/13-power_analysis.Rmd @@ -0,0 +1,328 @@ +# Power Analysis + +![](poweranalysis.jpg) + +A big asset (and probably one of the reasons why meta-analysis can be helpful in practical research) of meta-analyses is that **they allow for large data to be combined** to attain a more precise pooled effect. **Lack of statistical power**, however, may still play an important role, even in meta-analysis. This is particularly true for the clinical field, where it is often the case that only **few studies are available for synthesis**. The median number of included studies in the *Cochrane Database for Systematic Reviews*, for example, is six [@borenstein2011]. This is even more grave once we consider that (1) many meta-analysts will also want to perform **subgroup analyses and meta-regression**, for which even more power is required, and (2) many meta-analyses have **high heterogeneity**, which reduces our precision, and thus our power. + +Power is directly related to the **Type II error level** ($\beta$) we defined: $Power = 1- \beta$. It is common practice to set our **Type I error level** ($\alpha$) to $\alpha=0.05$, and thus to assume that the **Type I error is four times as grave as the Type II error** (i.e., falsely finding an effect while there is no effect in reality is four times as bad as not finding an effect while there is one in reality). The **Type II error** is therefore set at $\beta=0.20$, and the power should thus be $1-\beta=1-0.20=80\%$. + +

+ +```{block,type='rmdinfo'} +**What assumptions should i make for my meta-analysis?** + +While researchers conducting primary studies can **plan the size of their sample based on the effect size they want to find**, the situation is a little different in meta-analysis, where we can only work with the published material. However, we have some **control over the number of studies we want to include in our meta-analysis** (e.g., through more leniently or strictly defined inclusion criteria). Therefore, we can change our power to some extent by including more or less studies into the meta-analysis. There are **four things we have to make assumptions about when assessing the power of our meta-analysis a priori**. + +* The **number of included or includable studies** $k$ +* The **overall size of the studies we want to include** (are the studies in the field rather small or large?) +* The **effect size we want to determine**. This is particularly important, as we have to make assumptions about how **big an effect size has to be to still be clinically meaningful**. One study calculated that for interventions against depression, even effects as small as $SMD=0.24$ may still be meaningful for patients [@cuijpers2014threshold]. If we want to study **negative effects of an intervention** (e.g., death or symptom deterioration), even **very small effect sizes are extremely important and should be detected**. +* The **heterogeneity** of our studies' effect sizes, as this also affects the precision of our meta-analysis, and thus its potential to find significant effects. + +Besides these parameters, it is also important to think about other analyses, such as the **subgroup analyses** we want to conduct. How many studies are there for each subgroup, and what effects do we want to find in the subgroups? This is particularly important if we **hypothesize that an intervention is not effective in a subgroup of patients**, because we do not want to falsely find a treatment to be ineffective simply because the power was insufficient. +``` + +```{block,type='rmdachtung'} +**Post-hoc power tests: the abuse of power** + +Please note that power analyses should always be conducted **a priori**, meaning *before* you perform the meta-analysis. + +Power analyses conducted *after* an analysis ("post hoc") are fundamentally flawed [@hoenig2001abuse], as they suffer from the so-called **"power approach paradox"**, in which an analysis yielding no significant effect is thought to show more evidence that the null hypothesis is true when the p-value is smaller, since then, the power to detect a true effect would be higher. + +``` + +--- + +## Fixed-Effect Model {#fixed.power} + +To determine the **power** of a meta-analysis **under the fixed-effect model**, we have to assume the **true value of a distribution when the alternative hypothesis is correct** (i.e., when there is an effect). For power analysis in a conventional study, this distribution is $Z$. Follwing Borenstein et al. [@borenstein2011], we will call the true value $\lambda$ here to make clear that we are dealing with a meta-analysis, and not a primary study. $\lambda$ is defined as: + +$$\lambda=\frac{\delta}{\sqrt{V_{\delta}}}$$ + +Where $\delta$ is the **true effect size** and $V_{\delta}$ its variance. + +$V_{\delta}$ can be calculated for meta-analysis using the fixed-effect model with this formula: + + +$$V_{\delta}=\frac{\frac{n_1+n_2}{n_1xn_2}+\frac{d^2}{2(n_1+n_2)}}{k}$$ + +Where $k$ are all the included studies, and $n_1$ and $n_2$ are the **average sample sizes in each trial arm we assume across our studies**. + +Assuming a normal distribution and using $\lambda$, we can calculate the Power: + +$$Power = 1- \beta$$ +$$Power = 1- \Phi(c_{\alpha}-\lambda)+\Phi(-c_{\alpha}-\lambda) $$ + +Where $c_{\alpha}$ is the critical value of a $Z$-distribution. $\Phi$ is the **standard normal density function**, which we we need to calcuate the power using this equation: +$$\Phi(Z)=\frac{1}{\sqrt {2\pi}}e^{-\frac{Z^2}{2}}$$ + +Luckily, you don't have too think about these statistical details too much, as we have prepared a **function** for you with which you can easily conduct a **power analysis** using the fixed-effect model yourself. The function is called `power.analysis.function` and its code can be found below. + +Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. + +```{r} +power.analysis.fixed<-function(d,k,n1,n2,p){ + + n1<-n1 + n2<-n2 + d<-d + k<-k + p<-p + title<-c("Power for a fixed-effect meta-analysis:") + + v.d<-((n1+n2)/(n1*n2))+((d*d)/(2*(n1+n2))) + v.m<-v.d/k + lambda<-(d/sqrt(v.m)) + plevel<-1-(p/2) + zval<-qnorm(p=plevel, 0,1) + power<-1-(pnorm(zval-lambda))+(pnorm(-zval-lambda)) + return(power) +} +``` + +--- + +**For this function, we have to specify the following parameters:** + +```{r,echo=FALSE} +library(kableExtra) +Package<-c("d","k","n1","n2","p") +Description<-c("The effect size we want to be able to detect","The number of studies we will likely be able to include into our analysis","The average number of participats we assume in the intervention arms of our included studies","The average number of participats we assume in the control arms of our included studies","The Type I error rate (p-level). It is common to use 'p=0.05'") +m<-data.frame(Package,Description) +names<-c("Parameter", "Description") +colnames(m)<-names +kable(m) +``` + +Now, let's give an example. I assume that an effect of $d=0.30$ is likely and meaningful for the field of my meta-analysis. I also assume that on average, the studies in my analysis will be rather small, with 25 participants in each trial arm, and that there will be 10 studies in my analysis. I will set the $\alpha$-level to 0.05, as is convention. + +```{r, eval=FALSE} +power.analysis.fixed(d=0.30,k=10,n1=25,n2=25,p=0.05) +``` + +The output of the function is: + +```{r,echo=FALSE} +power.analysis.fixed(d=0.30,k=10,n1=25,n2=25,p=0.05) +``` + +Meaning that my power is 91%. This is more than the desired 80%, so given that my assumptions are remotely true, my meta-analysis will have **sufficient power using the fixed-effect model to detect a clinically relevant effect if it exists**. + +So, if i assume an effect of $d = 0.30$ in this example, i am lucky. If we play around with the effect size a little, however, while holding the other paramters constant, this can look very different. + +```{r,fig.width=5,fig.align='center',echo=FALSE,message=FALSE} +library(ggplot2) +library(reshape) +k <- seq(0, 50, length=1000) +pow.vals01<-lapply(k,function(k) power.analysis.fixed(d=0.10,k=k,n1=25,n2=25,p=0.05)) +pow.vals02<-lapply(k,function(k) power.analysis.fixed(d=0.20,k=k,n1=25,n2=25,p=0.05)) +pow.vals03<-lapply(k,function(k) power.analysis.fixed(d=0.30,k=k,n1=25,n2=25,p=0.05)) +pow.vals01<-as.numeric(pow.vals01) +pow.vals02<-as.numeric(pow.vals02) +pow.vals03<-as.numeric(pow.vals03) +data<-data.frame(k,pow.vals01,pow.vals02,pow.vals03) +ggplot()+ + geom_line(data = data, aes(x = k, y = pow.vals01), color = "blue",size=2) + + geom_line(data = data, aes(x = k, y = pow.vals02), color = "red",size=2) + + geom_line(data = data, aes(x = k, y = pow.vals03), color = "green",size=2) + + xlab('Number of Studies') + + ylab('Power')+ + scale_y_continuous(labels = scales::percent)+ + theme( + axis.line= element_line(color = "black",size = 1,linetype = "solid"), + legend.position = "bottom", + panel.grid.major = element_blank(), + panel.grid.minor = element_blank(), + panel.background = element_blank(), + legend.background = element_rect(linetype="solid", + colour ="black"), + legend.title = element_blank(), + legend.key.size = unit(0.75,"cm"), + legend.text=element_text(size=14))+ +annotate("text", x = 2, y = 0.7, label = "d = 0.3",size=5)+ + annotate("text", x = 14, y = 0.6, label = "d = 0.2",size=5)+ + annotate("text", x = 20, y = 0.24, label = "d = 0.1",size=5)+ + geom_hline(yintercept=0.8,linetype="dashed") +``` + +As you can see from this plot, sufficient power (see the **dashed line**) is soon reached for $d=0.30$, even if only few studies are included. If i assume a smaller effect size of $d=0.10$, however, **even 50 studies will not be sufficient to find a true effect**. + +--- + +## Random-Effects Model + +For power analyses under the **random-effects model**, the formula to calculate the variance of my true mean effect looks slightly different: + +$$V_{\delta}^*=\frac{V_Y+\tau^2}{k}$$ + +We see that again, $tau^2$ has to be included to take the **between-study heterogeneity into account** (see [Chapter 4.2](#random) for more details). However, i **do not know the between-study heterogeneity of my analysis** before i perform it, so what value should i assume? + +According to Hedges and Pigott [@hedges2004power], the follwing formulae may be used to calculate the power in the random-effect model assuming **small, moderate or high heterogeneity**: + + +**Small heterogeneity:** + +$$V_{\delta}^*=1.33\times\frac{V_Y}{k}$$ + +**Moderate heterogeneity:** + +$$V_{\delta}^*=1.67\times\frac{V_Y}{k}$$ + +**Large heterogeneity:** + +$$V_{\delta}^*=2\times\frac{V_Y}{k}$$ + +Again, you don't have to worry about the statistical details here. We have put the entire calculations into the `power.analysis.random` function, which can be found below. + +Again, R doesn't know this function yet, so we have to let R learn it by **copying and pasting** the code underneath **in its entirety** into the **console** on the bottom left pane of RStudio, and then hit **Enter ⏎**. + +```{r} +power.analysis.random<-function(d,k,n1,n2,p,heterogeneity){ + + n1<-n1 + n2<-n2 + d<-d + k<-k + p<-p + heterogeneity<-heterogeneity + + if(heterogeneity=="low"){ + + v.d<-((n1+n2)/(n1*n2))+((d*d)/(2*(n1+n2))) + v.m<-v.d/k + v.m<-1.33*v.m + lambda<-(d/sqrt(v.m)) + plevel<-1-(p/2) + zval<-qnorm(p=plevel, 0,1) + power<-1-(pnorm(zval-lambda))+(pnorm(-zval-lambda)) + return(power) + } + + if(heterogeneity=="moderate"){ + + v.d<-((n1+n2)/(n1*n2))+((d*d)/(2*(n1+n2))) + v.m<-v.d/k + v.m<-1.67*v.m + lambda<-(d/sqrt(v.m)) + plevel<-1-(p/2) + zval<-qnorm(p=plevel, 0,1) + power<-1-(pnorm(zval-lambda))+(pnorm(-zval-lambda)) + return(power) + } + + if(heterogeneity=="high"){ + + v.d<-((n1+n2)/(n1*n2))+((d*d)/(2*(n1+n2))) + v.m<-v.d/k + v.m<-2*v.m + lambda<-(d/sqrt(v.m)) + plevel<-1-(p/2) + zval<-qnorm(p=plevel, 0,1) + power<-1-(pnorm(zval-lambda))+(pnorm(-zval-lambda)) + return(power) + } + +} +``` + +Now you are set and ready to use the function. I will assume the same parameters from [before](#fixed.power), but this time i will also have to specify the `heterogeneity` in the function, which can take the values `"low"`, `"moderate"` and `"high"`. I will choose `"moderate"` for this example. + +```{r,eval=FALSE} +power.analysis.random(d=0.30,k=10,n1=25,n2=25,p=0.05, + heterogeneity = "moderate") + +``` + +The output i get is: + +```{r,echo=FALSE} +power.analysis.random(d=0.30,k=10,n1=25,n2=25,p=0.05,heterogeneity = "moderate") + +``` + +Interestingly, we see that this value is 73%, which is smaller than the value of 91% which was calculated using the **fixed-effect model**. The value is also below 80%, meaning that i would not have optimal power to find the desired effect of $d=0.30$ to be statistically significant if it exists. + +This has to do with the **larger heterogeneity** i assume in this simulation, which decreases the precision of my effect size estimate, and thus increases my need for statistical power. + +**The graph below visualizes this relationship:** + +```{r,fig.width=5,fig.align='center',echo=FALSE,fig.cap="Power in the random-effects-model. Darker colors indicate higher heterogeneity",message=FALSE} +library(ggplot2) +library(reshape) + +k <- seq(0, 50, length=1000) +pow.vals01<-lapply(k,function(k) power.analysis.random(d=0.10,k=k,n1=25,n2=25,p=0.05,heterogeneity = "moderate")) +pow.vals02<-lapply(k,function(k) power.analysis.random(d=0.20,k=k,n1=25,n2=25,p=0.05,heterogeneity = "moderate")) +pow.vals03<-lapply(k,function(k) power.analysis.random(d=0.30,k=k,n1=25,n2=25,p=0.05,heterogeneity = "moderate")) +pow.vals01<-as.numeric(pow.vals01) +pow.vals02<-as.numeric(pow.vals02) +pow.vals03<-as.numeric(pow.vals03) +data1<-data.frame(k,pow.vals01,pow.vals02,pow.vals03) + +k <- seq(0, 50, length=1000) +pow.vals01<-lapply(k,function(k) power.analysis.random(d=0.10,k=k,n1=25,n2=25,p=0.05,heterogeneity = "low")) +pow.vals02<-lapply(k,function(k) power.analysis.random(d=0.20,k=k,n1=25,n2=25,p=0.05,heterogeneity = "low")) +pow.vals03<-lapply(k,function(k) power.analysis.random(d=0.30,k=k,n1=25,n2=25,p=0.05,heterogeneity = "low")) +pow.vals01<-as.numeric(pow.vals01) +pow.vals02<-as.numeric(pow.vals02) +pow.vals03<-as.numeric(pow.vals03) +data2<-data.frame(k,pow.vals01,pow.vals02,pow.vals03) + +k <- seq(0, 50, length=1000) +pow.vals01<-lapply(k,function(k) power.analysis.random(d=0.10,k=k,n1=25,n2=25,p=0.05,heterogeneity = "high")) +pow.vals02<-lapply(k,function(k) power.analysis.random(d=0.20,k=k,n1=25,n2=25,p=0.05,heterogeneity = "high")) +pow.vals03<-lapply(k,function(k) power.analysis.random(d=0.30,k=k,n1=25,n2=25,p=0.05,heterogeneity = "high")) +pow.vals01<-as.numeric(pow.vals01) +pow.vals02<-as.numeric(pow.vals02) +pow.vals03<-as.numeric(pow.vals03) +data3<-data.frame(k,pow.vals01,pow.vals02,pow.vals03) + + +ggplot()+ + geom_line(data = data1, aes(x = k, y = pow.vals01), color = "blue",size=2) + + geom_line(data = data1, aes(x = k, y = pow.vals02), color = "#ff0000",size=2) + + geom_line(data = data1, aes(x = k, y = pow.vals03), color = "#00cc00",size=2) + + geom_line(data = data2, aes(x = k, y = pow.vals01), color = "lightblue",size=2) + + geom_line(data = data2, aes(x = k, y = pow.vals02), color = "#ff4d4d",size=2) + + geom_line(data = data2, aes(x = k, y = pow.vals03), color = "#1aff1a",size=2) + + geom_line(data = data3, aes(x = k, y = pow.vals01), color = "darkblue",size=2) + + geom_line(data = data3, aes(x = k, y = pow.vals02), color = "#b30000",size=2) + + geom_line(data = data3, aes(x = k, y = pow.vals03), color = "#008000",size=2) + + xlab('Number of Studies') + + ylab('Power')+ + scale_y_continuous(labels = scales::percent)+ + theme( + axis.line= element_line(color = "black",size = 1,linetype = "solid"), + legend.position = "bottom", + panel.grid.major = element_blank(), + panel.grid.minor = element_blank(), + panel.background = element_blank(), + legend.background = element_rect(linetype="solid", + colour ="black"), + legend.title = element_blank(), + legend.key.size = unit(0.75,"cm"), + legend.text=element_text(size=14))+ +annotate("text", x = 3, y = 0.7, label = "d = 0.3",size=5)+ + annotate("text", x = 25, y = 0.6, label = "d = 0.2",size=5)+ + annotate("text", x = 20, y = 0.13, label = "d = 0.1",size=5)+ + geom_hline(yintercept=0.8,linetype="dashed") +``` + +--- + +## Power Calculator Tool + +If you're feeling lazy, or if you want to quickly **check for the power of your meta-analysis under varying assumptions**, you might need a tool which makes it easier for you to calculate the power without having to run the R functions we described before each time. + +We therefore built a online **Power Calculator Tool**, which you can find below. The calculations are based on the formulae and functions we described in the previous chapters. + +[View in full page mode](https://mathiasharrer.shinyapps.io/power_calculator_meta_analysis/) + +```{r,echo=FALSE} +knitr::include_app("https://mathiasharrer.shinyapps.io/power_calculator_meta_analysis/", height = "1500px") +``` + +--- + + + + diff --git a/14-references.Rmd b/14-references.Rmd new file mode 100644 index 0000000..69552bb --- /dev/null +++ b/14-references.Rmd @@ -0,0 +1,3 @@ +`r if (knitr:::is_html_output()) ' +# References {-} +'` diff --git a/Doing_Meta_Analysis_in_R.log b/Doing_Meta_Analysis_in_R.log new file mode 100644 index 0000000..9ad9da0 --- /dev/null +++ b/Doing_Meta_Analysis_in_R.log @@ -0,0 +1,1105 @@ +This is XeTeX, Version 3.14159265-2.6-0.99999 (TeX Live 2018) (preloaded format=xelatex 2018.4.16) 3 DEC 2018 18:07 +entering extended mode + restricted \write18 enabled. + %&-line parsing enabled. +**Doing_Meta_Analysis_in_R.tex +(./Doing_Meta_Analysis_in_R.tex +LaTeX2e <2018-04-01> patch level 2 +Babel <3.18> and hyphenation patterns for 84 language(s) loaded. +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/book.cls +Document Class: book 2014/09/29 v1.4h Standard LaTeX document class +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/bk10.clo +File: bk10.clo 2014/09/29 v1.4h Standard LaTeX file (size option) +) +\c@part=\count80 +\c@chapter=\count81 +\c@section=\count82 +\c@subsection=\count83 +\c@subsubsection=\count84 +\c@paragraph=\count85 +\c@subparagraph=\count86 +\c@figure=\count87 +\c@table=\count88 +\abovecaptionskip=\skip41 +\belowcaptionskip=\skip42 +\bibindent=\dimen102 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/lm/lmodern.sty +Package: lmodern 2009/10/30 v1.6 Latin Modern Fonts +LaTeX Font Info: Overwriting symbol font `operators' in version `normal' +(Font) OT1/cmr/m/n --> OT1/lmr/m/n on input line 22. +LaTeX Font Info: Overwriting symbol font `letters' in version `normal' +(Font) OML/cmm/m/it --> OML/lmm/m/it on input line 23. +LaTeX Font Info: Overwriting symbol font `symbols' in version `normal' +(Font) OMS/cmsy/m/n --> OMS/lmsy/m/n on input line 24. +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `normal' +(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 25. +LaTeX Font Info: Overwriting symbol font `operators' in version `bold' +(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 26. +LaTeX Font Info: Overwriting symbol font `letters' in version `bold' +(Font) OML/cmm/b/it --> OML/lmm/b/it on input line 27. +LaTeX Font Info: Overwriting symbol font `symbols' in version `bold' +(Font) OMS/cmsy/b/n --> OMS/lmsy/b/n on input line 28. +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `bold' +(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 29. +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `normal' +(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 31. +LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `normal' +(Font) OT1/cmss/m/n --> OT1/lmss/m/n on input line 32. +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `normal' +(Font) OT1/cmr/m/it --> OT1/lmr/m/it on input line 33. +LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `normal' +(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 34. +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `bold' +(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 35. +LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `bold' +(Font) OT1/cmss/bx/n --> OT1/lmss/bx/n on input line 36. +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `bold' +(Font) OT1/cmr/bx/it --> OT1/lmr/bx/it on input line 37. +LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `bold' +(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 38. +) (/usr/local/texlive/2018/texmf-dist/tex/latex/amsfonts/amssymb.sty +Package: amssymb 2013/01/14 v3.01 AMS font symbols +(/usr/local/texlive/2018/texmf-dist/tex/latex/amsfonts/amsfonts.sty +Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support +\@emptytoks=\toks14 +\symAMSa=\mathgroup4 +\symAMSb=\mathgroup5 +LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold' +(Font) U/euf/m/n --> U/euf/b/n on input line 106. +)) (/usr/local/texlive/2018/texmf-dist/tex/latex/amsmath/amsmath.sty +Package: amsmath 2017/09/02 v2.17a AMS math features +\@mathmargin=\skip43 +For additional information on amsmath, use the `?' option. +(/usr/local/texlive/2018/texmf-dist/tex/latex/amsmath/amstext.sty +Package: amstext 2000/06/29 v2.01 AMS text +(/usr/local/texlive/2018/texmf-dist/tex/latex/amsmath/amsgen.sty +File: amsgen.sty 1999/11/30 v2.0 generic functions +\@emptytoks=\toks15 +\ex@=\dimen103 +)) (/usr/local/texlive/2018/texmf-dist/tex/latex/amsmath/amsbsy.sty +Package: amsbsy 1999/11/29 v1.2d Bold Symbols +\pmbraise@=\dimen104 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/amsmath/amsopn.sty +Package: amsopn 2016/03/08 v2.02 operator names +) +\inf@bad=\count89 +LaTeX Info: Redefining \frac on input line 213. +\uproot@=\count90 +\leftroot@=\count91 +LaTeX Info: Redefining \overline on input line 375. +\classnum@=\count92 +\DOTSCASE@=\count93 +LaTeX Info: Redefining \ldots on input line 472. +LaTeX Info: Redefining \dots on input line 475. +LaTeX Info: Redefining \cdots on input line 596. +\Mathstrutbox@=\box26 +\strutbox@=\box27 +\big@size=\dimen105 +LaTeX Font Info: Redeclaring font encoding OML on input line 712. +LaTeX Font Info: Redeclaring font encoding OMS on input line 713. +\macc@depth=\count94 +\c@MaxMatrixCols=\count95 +\dotsspace@=\muskip10 +\c@parentequation=\count96 +\dspbrk@lvl=\count97 +\tag@help=\toks16 +\row@=\count98 +\column@=\count99 +\maxfields@=\count100 +\andhelp@=\toks17 +\eqnshift@=\dimen106 +\alignsep@=\dimen107 +\tagshift@=\dimen108 +\tagwidth@=\dimen109 +\totwidth@=\dimen110 +\lineht@=\dimen111 +\@envbody=\toks18 +\multlinegap=\skip44 +\multlinetaggap=\skip45 +\mathdisplay@stack=\toks19 +LaTeX Info: Redefining \[ on input line 2817. +LaTeX Info: Redefining \] on input line 2818. +) (/usr/local/texlive/2018/texmf-dist/tex/generic/ifxetex/ifxetex.sty +Package: ifxetex 2010/09/12 v0.6 Provides ifxetex conditional +) (/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/ifluatex.sty +Package: ifluatex 2016/05/16 v1.4 Provides the ifluatex switch (HO) +Package ifluatex Info: LuaTeX not detected. +) (/usr/local/texlive/2018/texmf-dist/tex/latex/base/fixltx2e.sty +Package: fixltx2e 2016/12/29 v2.1a fixes to LaTeX (obsolete) +Applying: [2015/01/01] Old fixltx2e package on input line 46. + +Package fixltx2e Warning: fixltx2e is not required with releases after 2015 +(fixltx2e) All fixes are now in the LaTeX kernel. +(fixltx2e) See the latexrelease package for details. + +Already applied: [0000/00/00] Old fixltx2e package on input line 53. +) (/usr/local/texlive/2018/texmf-dist/tex/xelatex/mathspec/mathspec.sty +Package: mathspec 2016/12/22 v0.2b LaTeX Package (Mathematics font selection fo +r XeLaTeX) +(/usr/local/texlive/2018/texmf-dist/tex/latex/etoolbox/etoolbox.sty +Package: etoolbox 2018/02/11 v2.5e e-TeX tools for LaTeX (JAW) +\etb@tempcnta=\count101 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/fontspec/fontspec.sty (/usr/loc +al/texlive/2018/texmf-dist/tex/latex/l3packages/xparse/xparse.sty (/usr/local/t +exlive/2018/texmf-dist/tex/latex/l3kernel/expl3.sty +Package: expl3 2018/03/05 L3 programming layer (loader) +(/usr/local/texlive/2018/texmf-dist/tex/latex/l3kernel/expl3-code.tex +Package: expl3 2018/03/05 L3 programming layer (code) +\c_max_int=\count102 +\l_tmpa_int=\count103 +\l_tmpb_int=\count104 +\g_tmpa_int=\count105 +\g_tmpb_int=\count106 +\g__intarray_font_int=\count107 +\g__prg_map_int=\count108 +\c_log_iow=\count109 +\l_iow_line_count_int=\count110 +\l__iow_line_target_int=\count111 +\l__iow_one_indent_int=\count112 +\l__iow_indent_int=\count113 +\c_zero_dim=\dimen112 +\c_max_dim=\dimen113 +\l_tmpa_dim=\dimen114 +\l_tmpb_dim=\dimen115 +\g_tmpa_dim=\dimen116 +\g_tmpb_dim=\dimen117 +\c_zero_skip=\skip46 +\c_max_skip=\skip47 +\l_tmpa_skip=\skip48 +\l_tmpb_skip=\skip49 +\g_tmpa_skip=\skip50 +\g_tmpb_skip=\skip51 +\c_zero_muskip=\muskip11 +\c_max_muskip=\muskip12 +\l_tmpa_muskip=\muskip13 +\l_tmpb_muskip=\muskip14 +\g_tmpa_muskip=\muskip15 +\g_tmpb_muskip=\muskip16 +\l_keys_choice_int=\count114 +\c__fp_leading_shift_int=\count115 +\c__fp_middle_shift_int=\count116 +\c__fp_trailing_shift_int=\count117 +\c__fp_big_leading_shift_int=\count118 +\c__fp_big_middle_shift_int=\count119 +\c__fp_big_trailing_shift_int=\count120 +\c__fp_Bigg_leading_shift_int=\count121 +\c__fp_Bigg_middle_shift_int=\count122 +\c__fp_Bigg_trailing_shift_int=\count123 +\l__sort_length_int=\count124 +\l__sort_min_int=\count125 +\l__sort_top_int=\count126 +\l__sort_max_int=\count127 +\l__sort_true_max_int=\count128 +\l__sort_block_int=\count129 +\l__sort_begin_int=\count130 +\l__sort_end_int=\count131 +\l__sort_A_int=\count132 +\l__sort_B_int=\count133 +\l__sort_C_int=\count134 +\l__tl_build_start_index_int=\count135 +\l__tl_build_index_int=\count136 +\l__tl_analysis_normal_int=\count137 +\l__tl_analysis_index_int=\count138 +\l__tl_analysis_nesting_int=\count139 +\l__tl_analysis_type_int=\count140 +\l__regex_internal_a_int=\count141 +\l__regex_internal_b_int=\count142 +\l__regex_internal_c_int=\count143 +\l__regex_balance_int=\count144 +\l__regex_group_level_int=\count145 +\l__regex_mode_int=\count146 +\c__regex_cs_in_class_mode_int=\count147 +\c__regex_cs_mode_int=\count148 +\l__regex_catcodes_int=\count149 +\l__regex_default_catcodes_int=\count150 +\c__regex_catcode_L_int=\count151 +\c__regex_catcode_O_int=\count152 +\c__regex_catcode_A_int=\count153 +\c__regex_all_catcodes_int=\count154 +\l__regex_show_lines_int=\count155 +\l__regex_min_state_int=\count156 +\l__regex_max_state_int=\count157 +\l__regex_left_state_int=\count158 +\l__regex_right_state_int=\count159 +\l__regex_capturing_group_int=\count160 +\l__regex_min_pos_int=\count161 +\l__regex_max_pos_int=\count162 +\l__regex_curr_pos_int=\count163 +\l__regex_start_pos_int=\count164 +\l__regex_success_pos_int=\count165 +\l__regex_curr_char_int=\count166 +\l__regex_curr_catcode_int=\count167 +\l__regex_last_char_int=\count168 +\l__regex_case_changed_char_int=\count169 +\l__regex_curr_state_int=\count170 +\l__regex_step_int=\count171 +\l__regex_min_active_int=\count172 +\l__regex_max_active_int=\count173 +\l__regex_replacement_csnames_int=\count174 +\l__regex_match_count_int=\count175 +\l__regex_min_submatch_int=\count176 +\l__regex_submatch_int=\count177 +\l__regex_zeroth_submatch_int=\count178 +\g__regex_trace_regex_int=\count179 +\c_empty_box=\box28 +\l_tmpa_box=\box29 +\l_tmpb_box=\box30 +\g_tmpa_box=\box31 +\g_tmpb_box=\box32 +\l__box_top_dim=\dimen118 +\l__box_bottom_dim=\dimen119 +\l__box_left_dim=\dimen120 +\l__box_right_dim=\dimen121 +\l__box_top_new_dim=\dimen122 +\l__box_bottom_new_dim=\dimen123 +\l__box_left_new_dim=\dimen124 +\l__box_right_new_dim=\dimen125 +\l__box_internal_box=\box33 +\l__coffin_internal_box=\box34 +\l__coffin_internal_dim=\dimen126 +\l__coffin_offset_x_dim=\dimen127 +\l__coffin_offset_y_dim=\dimen128 +\l__coffin_x_dim=\dimen129 +\l__coffin_y_dim=\dimen130 +\l__coffin_x_prime_dim=\dimen131 +\l__coffin_y_prime_dim=\dimen132 +\c_empty_coffin=\box35 +\l__coffin_aligned_coffin=\box36 +\l__coffin_aligned_internal_coffin=\box37 +\l_tmpa_coffin=\box38 +\l_tmpb_coffin=\box39 +\l__coffin_display_coffin=\box40 +\l__coffin_display_coord_coffin=\box41 +\l__coffin_display_pole_coffin=\box42 +\l__coffin_display_offset_dim=\dimen133 +\l__coffin_display_x_dim=\dimen134 +\l__coffin_display_y_dim=\dimen135 +\l__coffin_bounding_shift_dim=\dimen136 +\l__coffin_left_corner_dim=\dimen137 +\l__coffin_right_corner_dim=\dimen138 +\l__coffin_bottom_corner_dim=\dimen139 +\l__coffin_top_corner_dim=\dimen140 +\l__coffin_scaled_total_height_dim=\dimen141 +\l__coffin_scaled_width_dim=\dimen142 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/l3kernel/l3xdvipdfmx.def +File: l3xdvidpfmx.def 2017/03/18 v L3 Experimental driver: xdvipdfmx +\g__driver_image_int=\count180 +)) +Package: xparse 2018/02/21 L3 Experimental document command parser +\l__xparse_current_arg_int=\count181 +\g__xparse_grabber_int=\count182 +\l__xparse_m_args_int=\count183 +\l__xparse_mandatory_args_int=\count184 +\l__xparse_v_nesting_int=\count185 +) +Package: fontspec 2017/11/09 v2.6g Font selection for XeLaTeX and LuaLaTeX +(/usr/local/texlive/2018/texmf-dist/tex/latex/fontspec/fontspec-xetex.sty +Package: fontspec-xetex 2017/11/09 v2.6g Font selection for XeLaTeX and LuaLaTe +X +\l__fontspec_script_int=\count186 +\l__fontspec_language_int=\count187 +\l__fontspec_strnum_int=\count188 +\l__fontspec_tmp_int=\count189 +\l__fontspec_em_int=\count190 +\l__fontspec_emdef_int=\count191 +\l__fontspec_strong_int=\count192 +\l__fontspec_strongdef_int=\count193 +\l__fontspec_tmpa_dim=\dimen143 +\l__fontspec_tmpb_dim=\dimen144 +\l__fontspec_tmpc_dim=\dimen145 +\g__file_internal_ior=\read1 +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/fontenc.sty +Package: fontenc 2017/04/05 v2.0i Standard LaTeX package +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/tuenc.def +File: tuenc.def 2017/04/05 v2.0i Standard LaTeX file +LaTeX Font Info: Redeclaring font encoding TU on input line 82. +)) +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \fontspec with sig. 'O{}mO{}' on line 542. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setmainfont with sig. 'O{}mO{}' on line 546. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setsansfont with sig. 'O{}mO{}' on line 550. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setmonofont with sig. 'O{}mO{}' on line 554. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setmathrm with sig. 'O{}mO{}' on line 558. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setboldmathrm with sig. 'O{}mO{}' on line 562. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setmathsf with sig. 'O{}mO{}' on line 566. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setmathtt with sig. 'O{}mO{}' on line 570. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \setromanfont with sig. 'O{}mO{}' on line 574. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newfontfamily with sig. 'mO{}mO{}' on line 578. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newfontface with sig. 'mO{}mO{}' on line 582. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \defaultfontfeatures with sig. 't+om' on line 586. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \addfontfeatures with sig. 'm' on line 590. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \addfontfeature with sig. 'm' on line 594. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newfontfeature with sig. 'mm' on line 598. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newAATfeature with sig. 'mmmm' on line 602. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newopentypefeature with sig. 'mmm' on line 606. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newICUfeature with sig. 'mmm' on line 610. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \aliasfontfeature with sig. 'mm' on line 614. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \aliasfontfeatureoption with sig. 'mmm' on line 618. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newfontscript with sig. 'mm' on line 622. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newfontlanguage with sig. 'mm' on line 626. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \DeclareFontsExtensions with sig. 'm' on line 630. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \IfFontFeatureActiveTF with sig. 'mmm' on line 634. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \EncodingCommand with sig. 'mO{}m' on line 3632. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \EncodingAccent with sig. 'mm' on line 3638. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \EncodingSymbol with sig. 'mm' on line 3644. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \EncodingComposite with sig. 'mmm' on line 3650. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \EncodingCompositeCommand with sig. 'mmm' on line 3656. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \DeclareUnicodeEncoding with sig. 'mm' on line 3681. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \UndeclareSymbol with sig. 'm' on line 3687. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \UndeclareAccent with sig. 'm' on line 3693. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \UndeclareCommand with sig. 'm' on line 3699. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \UndeclareComposite with sig. 'mm' on line 3706. +................................................. +(/usr/local/texlive/2018/texmf-dist/tex/latex/fontspec/fontspec.cfg) +LaTeX Info: Redefining \itshape on input line 3891. +LaTeX Info: Redefining \slshape on input line 3896. +LaTeX Info: Redefining \scshape on input line 3901. +LaTeX Info: Redefining \upshape on input line 3906. +LaTeX Info: Redefining \em on input line 3936. +LaTeX Info: Redefining \emph on input line 3961. +LaTeX Info: Redefining \- on input line 4015. +................................................. +. LaTeX info: "xparse/redefine-command" +. +. Redefining command \oldstylenums with sig. 'm' on line 4110. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \liningnums with sig. 'm' on line 4114. +................................................. +)) (/usr/local/texlive/2018/texmf-dist/tex/latex/xkeyval/xkeyval.sty +Package: xkeyval 2014/12/03 v2.7a package option processing (HA) +(/usr/local/texlive/2018/texmf-dist/tex/generic/xkeyval/xkeyval.tex (/usr/local +/texlive/2018/texmf-dist/tex/generic/xkeyval/xkvutils.tex +\XKV@toks=\toks20 +\XKV@tempa@toks=\toks21 +(/usr/local/texlive/2018/texmf-dist/tex/generic/xkeyval/keyval.tex)) +\XKV@depth=\count194 +File: xkeyval.tex 2014/12/03 v2.7a key=value parser (HA) +)) +\c@eu@=\count195 +\c@eu@i=\count196 +\c@mkern=\count197 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/upquote/upquote.sty +Package: upquote 2012/04/19 v1.3 upright-quote and grave-accent glyphs in verba +tim +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/textcomp.sty +Package: textcomp 2017/04/05 v2.0i Standard LaTeX package +Package textcomp Info: Sub-encoding information: +(textcomp) 5 = only ISO-Adobe without \textcurrency +(textcomp) 4 = 5 + \texteuro +(textcomp) 3 = 4 + \textohm +(textcomp) 2 = 3 + \textestimated + \textcurrency +(textcomp) 1 = TS1 - \textcircled - \t +(textcomp) 0 = TS1 (full) +(textcomp) Font families with sub-encoding setting implement +(textcomp) only a restricted character set as indicated. +(textcomp) Family '?' is the default used for unknown fonts. +(textcomp) See the documentation for details. +Package textcomp Info: Setting ? sub-encoding to TS1/1 on input line 79. +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/ts1enc.def +File: ts1enc.def 2001/06/05 v3.0e (jk/car/fm) Standard LaTeX file +) +LaTeX Info: Redefining \oldstylenums on input line 334. +Package textcomp Info: Setting cmr sub-encoding to TS1/0 on input line 349. +Package textcomp Info: Setting cmss sub-encoding to TS1/0 on input line 350. +Package textcomp Info: Setting cmtt sub-encoding to TS1/0 on input line 351. +Package textcomp Info: Setting cmvtt sub-encoding to TS1/0 on input line 352. +Package textcomp Info: Setting cmbr sub-encoding to TS1/0 on input line 353. +Package textcomp Info: Setting cmtl sub-encoding to TS1/0 on input line 354. +Package textcomp Info: Setting ccr sub-encoding to TS1/0 on input line 355. +Package textcomp Info: Setting ptm sub-encoding to TS1/4 on input line 356. +Package textcomp Info: Setting pcr sub-encoding to TS1/4 on input line 357. +Package textcomp Info: Setting phv sub-encoding to TS1/4 on input line 358. +Package textcomp Info: Setting ppl sub-encoding to TS1/3 on input line 359. +Package textcomp Info: Setting pag sub-encoding to TS1/4 on input line 360. +Package textcomp Info: Setting pbk sub-encoding to TS1/4 on input line 361. +Package textcomp Info: Setting pnc sub-encoding to TS1/4 on input line 362. +Package textcomp Info: Setting pzc sub-encoding to TS1/4 on input line 363. +Package textcomp Info: Setting bch sub-encoding to TS1/4 on input line 364. +Package textcomp Info: Setting put sub-encoding to TS1/5 on input line 365. +Package textcomp Info: Setting uag sub-encoding to TS1/5 on input line 366. +Package textcomp Info: Setting ugq sub-encoding to TS1/5 on input line 367. +Package textcomp Info: Setting ul8 sub-encoding to TS1/4 on input line 368. +Package textcomp Info: Setting ul9 sub-encoding to TS1/4 on input line 369. +Package textcomp Info: Setting augie sub-encoding to TS1/5 on input line 370. +Package textcomp Info: Setting dayrom sub-encoding to TS1/3 on input line 371. +Package textcomp Info: Setting dayroms sub-encoding to TS1/3 on input line 372. + +Package textcomp Info: Setting pxr sub-encoding to TS1/0 on input line 373. +Package textcomp Info: Setting pxss sub-encoding to TS1/0 on input line 374. +Package textcomp Info: Setting pxtt sub-encoding to TS1/0 on input line 375. +Package textcomp Info: Setting txr sub-encoding to TS1/0 on input line 376. +Package textcomp Info: Setting txss sub-encoding to TS1/0 on input line 377. +Package textcomp Info: Setting txtt sub-encoding to TS1/0 on input line 378. +Package textcomp Info: Setting lmr sub-encoding to TS1/0 on input line 379. +Package textcomp Info: Setting lmdh sub-encoding to TS1/0 on input line 380. +Package textcomp Info: Setting lmss sub-encoding to TS1/0 on input line 381. +Package textcomp Info: Setting lmssq sub-encoding to TS1/0 on input line 382. +Package textcomp Info: Setting lmvtt sub-encoding to TS1/0 on input line 383. +Package textcomp Info: Setting lmtt sub-encoding to TS1/0 on input line 384. +Package textcomp Info: Setting qhv sub-encoding to TS1/0 on input line 385. +Package textcomp Info: Setting qag sub-encoding to TS1/0 on input line 386. +Package textcomp Info: Setting qbk sub-encoding to TS1/0 on input line 387. +Package textcomp Info: Setting qcr sub-encoding to TS1/0 on input line 388. +Package textcomp Info: Setting qcs sub-encoding to TS1/0 on input line 389. +Package textcomp Info: Setting qpl sub-encoding to TS1/0 on input line 390. +Package textcomp Info: Setting qtm sub-encoding to TS1/0 on input line 391. +Package textcomp Info: Setting qzc sub-encoding to TS1/0 on input line 392. +Package textcomp Info: Setting qhvc sub-encoding to TS1/0 on input line 393. +Package textcomp Info: Setting futs sub-encoding to TS1/4 on input line 394. +Package textcomp Info: Setting futx sub-encoding to TS1/4 on input line 395. +Package textcomp Info: Setting futj sub-encoding to TS1/4 on input line 396. +Package textcomp Info: Setting hlh sub-encoding to TS1/3 on input line 397. +Package textcomp Info: Setting hls sub-encoding to TS1/3 on input line 398. +Package textcomp Info: Setting hlst sub-encoding to TS1/3 on input line 399. +Package textcomp Info: Setting hlct sub-encoding to TS1/5 on input line 400. +Package textcomp Info: Setting hlx sub-encoding to TS1/5 on input line 401. +Package textcomp Info: Setting hlce sub-encoding to TS1/5 on input line 402. +Package textcomp Info: Setting hlcn sub-encoding to TS1/5 on input line 403. +Package textcomp Info: Setting hlcw sub-encoding to TS1/5 on input line 404. +Package textcomp Info: Setting hlcf sub-encoding to TS1/5 on input line 405. +Package textcomp Info: Setting pplx sub-encoding to TS1/3 on input line 406. +Package textcomp Info: Setting pplj sub-encoding to TS1/3 on input line 407. +Package textcomp Info: Setting ptmx sub-encoding to TS1/4 on input line 408. +Package textcomp Info: Setting ptmj sub-encoding to TS1/4 on input line 409. +)) (/usr/local/texlive/2018/texmf-dist/tex/latex/microtype/microtype.sty +Package: microtype 2018/01/14 v2.7a Micro-typographical refinements (RS) +\MT@toks=\toks22 +\MT@count=\count198 +LaTeX Info: Redefining \textls on input line 793. +\MT@outer@kern=\dimen146 +LaTeX Info: Redefining \textmicrotypecontext on input line 1339. +\MT@listname@count=\count199 +(/usr/local/texlive/2018/texmf-dist/tex/latex/microtype/microtype-xetex.def +File: microtype-xetex.def 2018/01/14 v2.7a Definitions specific to xetex (RS) +LaTeX Info: Redefining \lsstyle on input line 256. +) +Package microtype Info: Loading configuration file microtype.cfg. +(/usr/local/texlive/2018/texmf-dist/tex/latex/microtype/microtype.cfg +File: microtype.cfg 2018/01/14 v2.7a microtype main configuration file (RS) +)) (/usr/local/texlive/2018/texmf-dist/tex/latex/geometry/geometry.sty +Package: geometry 2018/03/24 v5.7 Page Geometry +(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/ifpdf.sty +Package: ifpdf 2017/03/15 v3.2 Provides the ifpdf switch +) (/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/ifvtex.sty +Package: ifvtex 2016/05/16 v1.6 Detect VTeX and its facilities (HO) +Package ifvtex Info: VTeX not detected. +) +\Gm@cnth=\count266 +\Gm@cntv=\count267 +\c@Gm@tempcnt=\count268 +\Gm@bindingoffset=\dimen147 +\Gm@wd@mp=\dimen148 +\Gm@odd@mp=\dimen149 +\Gm@even@mp=\dimen150 +\Gm@layoutwidth=\dimen151 +\Gm@layoutheight=\dimen152 +\Gm@layouthoffset=\dimen153 +\Gm@layoutvoffset=\dimen154 +\Gm@dimlist=\toks23 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/hyperref/hyperref.sty +Package: hyperref 2018/02/06 v6.86b Hypertext links for LaTeX +(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/hobsub-hyperref.sty +Package: hobsub-hyperref 2016/05/16 v1.14 Bundle oberdiek, subset hyperref (HO) + +(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/hobsub-generic.sty +Package: hobsub-generic 2016/05/16 v1.14 Bundle oberdiek, subset generic (HO) +Package: hobsub 2016/05/16 v1.14 Construct package bundles (HO) +Package: infwarerr 2016/05/16 v1.4 Providing info/warning/error messages (HO) +Package: ltxcmds 2016/05/16 v1.23 LaTeX kernel commands for general use (HO) +Package hobsub Info: Skipping package `ifluatex' (already loaded). +Package hobsub Info: Skipping package `ifvtex' (already loaded). +Package: intcalc 2016/05/16 v1.2 Expandable calculations with integers (HO) +Package hobsub Info: Skipping package `ifpdf' (already loaded). +Package: etexcmds 2016/05/16 v1.6 Avoid name clashes with e-TeX commands (HO) +Package etexcmds Info: Could not find \expanded. +(etexcmds) That can mean that you are not using pdfTeX 1.50 or +(etexcmds) that some package has redefined \expanded. +(etexcmds) In the latter case, load this package earlier. +Package: kvsetkeys 2016/05/16 v1.17 Key value parser (HO) +Package: kvdefinekeys 2016/05/16 v1.4 Define keys (HO) +Package: pdftexcmds 2018/01/30 v0.27 Utility functions of pdfTeX for LuaTeX (HO +) +Package pdftexcmds Info: LuaTeX not detected. +Package pdftexcmds Info: pdfTeX >= 1.30 not detected. +Package pdftexcmds Info: \pdf@primitive is available. +Package pdftexcmds Info: \pdf@ifprimitive is available. +Package pdftexcmds Info: \pdfdraftmode not found. +Package: pdfescape 2016/05/16 v1.14 Implements pdfTeX's escape features (HO) +Package: bigintcalc 2016/05/16 v1.4 Expandable calculations on big integers (HO +) +Package: bitset 2016/05/16 v1.2 Handle bit-vector datatype (HO) +Package: uniquecounter 2016/05/16 v1.3 Provide unlimited unique counter (HO) +) +Package hobsub Info: Skipping package `hobsub' (already loaded). +Package: letltxmacro 2016/05/16 v1.5 Let assignment for LaTeX macros (HO) +Package: hopatch 2016/05/16 v1.3 Wrapper for package hooks (HO) +Package: xcolor-patch 2016/05/16 xcolor patch +Package: atveryend 2016/05/16 v1.9 Hooks at the very end of document (HO) +Package: atbegshi 2016/06/09 v1.18 At begin shipout hook (HO) +Package: refcount 2016/05/16 v3.5 Data extraction from label references (HO) +Package: hycolor 2016/05/16 v1.8 Color options for hyperref/bookmark (HO) +) (/usr/local/texlive/2018/texmf-dist/tex/latex/oberdiek/auxhook.sty +Package: auxhook 2016/05/16 v1.4 Hooks for auxiliary files (HO) +) (/usr/local/texlive/2018/texmf-dist/tex/latex/oberdiek/kvoptions.sty +Package: kvoptions 2016/05/16 v3.12 Key value format for package options (HO) +) +\@linkdim=\dimen155 +\Hy@linkcounter=\count269 +\Hy@pagecounter=\count270 +(/usr/local/texlive/2018/texmf-dist/tex/latex/hyperref/pd1enc.def +File: pd1enc.def 2018/02/06 v6.86b Hyperref: PDFDocEncoding definition (HO) +) +\Hy@SavedSpaceFactor=\count271 +(/usr/local/texlive/2018/texmf-dist/tex/latex/latexconfig/hyperref.cfg +File: hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive +) +Package hyperref Info: Hyper figures OFF on input line 4509. +Package hyperref Info: Link nesting OFF on input line 4514. +Package hyperref Info: Hyper index ON on input line 4517. +Package hyperref Info: Plain pages OFF on input line 4524. +Package hyperref Info: Backreferencing OFF on input line 4529. +Package hyperref Info: Implicit mode ON; LaTeX internals redefined. +Package hyperref Info: Bookmarks ON on input line 4762. +\c@Hy@tempcnt=\count272 +(/usr/local/texlive/2018/texmf-dist/tex/latex/url/url.sty +\Urlmuskip=\muskip17 +Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc. +) +LaTeX Info: Redefining \url on input line 5115. +\XeTeXLinkMargin=\dimen156 +\Fld@menulength=\count273 +\Field@Width=\dimen157 +\Fld@charsize=\dimen158 +Package hyperref Info: Hyper figures OFF on input line 6369. +Package hyperref Info: Link nesting OFF on input line 6374. +Package hyperref Info: Hyper index ON on input line 6377. +Package hyperref Info: backreferencing OFF on input line 6384. +Package hyperref Info: Link coloring OFF on input line 6389. +Package hyperref Info: Link coloring with OCG OFF on input line 6394. +Package hyperref Info: PDF/A mode OFF on input line 6399. +LaTeX Info: Redefining \ref on input line 6439. +LaTeX Info: Redefining \pageref on input line 6443. +\Hy@abspage=\count274 +\c@Item=\count275 +\c@Hfootnote=\count276 +) +Package hyperref Info: Driver (autodetected): hxetex. +(/usr/local/texlive/2018/texmf-dist/tex/latex/hyperref/hxetex.def +File: hxetex.def 2018/02/06 v6.86b Hyperref driver for XeTeX +(/usr/local/texlive/2018/texmf-dist/tex/latex/hyperref/puenc.def +File: puenc.def 2018/02/06 v6.86b Hyperref: PDF Unicode definition (HO) +) (/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/stringenc.sty +Package: stringenc 2016/05/16 v1.11 Convert strings between diff. encodings (HO +) +) +\pdfm@box=\box43 +\c@Hy@AnnotLevel=\count277 +\HyField@AnnotCount=\count278 +\Fld@listcount=\count279 +\c@bookmark@seq@number=\count280 +(/usr/local/texlive/2018/texmf-dist/tex/latex/oberdiek/rerunfilecheck.sty +Package: rerunfilecheck 2016/05/16 v1.8 Rerun checks for auxiliary files (HO) +Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2 +82. +) +\Hy@SectionHShift=\skip52 +) +Package hyperref Info: Option `unicode' set `true' on input line 30. +(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/se-ascii-print.def +File: se-ascii-print.def 2016/05/16 v1.11 stringenc: Printable ASCII characters + +) +Package hyperref Info: Option `breaklinks' set `true' on input line 30. +(/usr/local/texlive/2018/texmf-dist/tex/latex/natbib/natbib.sty +Package: natbib 2010/09/13 8.31b (PWD, AO) +\bibhang=\skip53 +\bibsep=\skip54 +LaTeX Info: Redefining \cite on input line 694. +\c@NAT@ctr=\count281 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/graphics/color.sty +Package: color 2016/07/10 v1.1e Standard LaTeX Color (DPC) +(/usr/local/texlive/2018/texmf-dist/tex/latex/graphics-cfg/color.cfg +File: color.cfg 2016/01/02 v1.6 sample color configuration +) +Package color Info: Driver file: xetex.def on input line 147. +(/usr/local/texlive/2018/texmf-dist/tex/latex/graphics-def/xetex.def +File: xetex.def 2017/06/24 v5.0h Graphics/color driver for xetex +)) (/usr/local/texlive/2018/texmf-dist/tex/latex/fancyvrb/fancyvrb.sty +Package: fancyvrb 2008/02/07 + +Style option: `fancyvrb' v2.7a, with DG/SPQR fixes, and firstline=lastline fix +<2008/02/07> (tvz) +\FV@CodeLineNo=\count282 +\FV@InFile=\read2 +\FV@TabBox=\box44 +\c@FancyVerbLine=\count283 +\FV@StepNumber=\count284 +\FV@OutFile=\write3 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/framed/framed.sty +Package: framed 2011/10/22 v 0.96: framed or shaded text with page breaks +\OuterFrameSep=\skip55 +\fb@frw=\dimen159 +\fb@frh=\dimen160 +\FrameRule=\dimen161 +\FrameSep=\dimen162 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/tools/longtable.sty +Package: longtable 2014/10/28 v4.11 Multi-page Table package (DPC) +\LTleft=\skip56 +\LTright=\skip57 +\LTpre=\skip58 +\LTpost=\skip59 +\LTchunksize=\count285 +\LTcapwidth=\dimen163 +\LT@head=\box45 +\LT@firsthead=\box46 +\LT@foot=\box47 +\LT@lastfoot=\box48 +\LT@cols=\count286 +\LT@rows=\count287 +\c@LT@tables=\count288 +\c@LT@chunks=\count289 +\LT@p@ftn=\toks24 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/booktabs/booktabs.sty +Package: booktabs 2016/04/27 v1.618033 publication quality tables +\heavyrulewidth=\dimen164 +\lightrulewidth=\dimen165 +\cmidrulewidth=\dimen166 +\belowrulesep=\dimen167 +\belowbottomsep=\dimen168 +\aboverulesep=\dimen169 +\abovetopsep=\dimen170 +\cmidrulesep=\dimen171 +\cmidrulekern=\dimen172 +\defaultaddspace=\dimen173 +\@cmidla=\count290 +\@cmidlb=\count291 +\@aboverulesep=\dimen174 +\@belowrulesep=\dimen175 +\@thisruleclass=\count292 +\@lastruleclass=\count293 +\@thisrulewidth=\dimen176 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/graphics/graphicx.sty +Package: graphicx 2017/06/01 v1.1a Enhanced LaTeX Graphics (DPC,SPQR) +(/usr/local/texlive/2018/texmf-dist/tex/latex/graphics/graphics.sty +Package: graphics 2017/06/25 v1.2c Standard LaTeX Graphics (DPC,SPQR) +(/usr/local/texlive/2018/texmf-dist/tex/latex/graphics/trig.sty +Package: trig 2016/01/03 v1.10 sin cos tan (DPC) +) (/usr/local/texlive/2018/texmf-dist/tex/latex/graphics-cfg/graphics.cfg +File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration +) +Package graphics Info: Driver file: xetex.def on input line 99. +) +\Gin@req@height=\dimen177 +\Gin@req@width=\dimen178 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/oberdiek/grffile.sty +Package: grffile 2017/06/30 v1.18 Extended file name support for graphics (HO) +Package grffile Info: Option `multidot' is set to `true'. +Package grffile Info: Option `extendedchars' is set to `false'. +Package grffile Info: Option `space' is set to `true'. +Package grffile Info: \Gin@ii of package `graphicx' fixed on input line 494. +) (/usr/local/texlive/2018/texmf-dist/tex/latex/parskip/parskip.sty +Package: parskip 2001/04/09 non-zero parskip adjustments +) (/usr/local/texlive/2018/texmf-dist/tex/latex/titling/titling.sty +Package: titling 2009/09/04 v2.1d maketitle typesetting +\thanksmarkwidth=\skip60 +\thanksmargin=\skip61 +\droptitle=\skip62 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/amscls/amsthm.sty +Package: amsthm 2017/10/31 v2.20.4 +\thm@style=\toks25 +\thm@bodyfont=\toks26 +\thm@headfont=\toks27 +\thm@notefont=\toks28 +\thm@headpunct=\toks29 +\thm@preskip=\skip63 +\thm@postskip=\skip64 +\thm@headsep=\skip65 +\dth@everypar=\toks30 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/tools/array.sty +Package: array 2018/04/07 v2.4g Tabular extension package (FMi) +\col@sep=\dimen179 +\ar@mcellbox=\box49 +\extrarowheight=\dimen180 +\NC@list=\toks31 +\extratabsurround=\skip66 +\backup@length=\skip67 +\ar@cellbox=\box50 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/multirow/multirow.sty +Package: multirow 2016/11/25 v2.2 Span multiple rows of a table +\multirow@colwidth=\skip68 +\multirow@cntb=\count294 +\multirow@dima=\skip69 +\bigstrutjot=\dimen181 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/xcolor/xcolor.sty +Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK) +(/usr/local/texlive/2018/texmf-dist/tex/latex/graphics-cfg/color.cfg +File: color.cfg 2016/01/02 v1.6 sample color configuration +) +Package xcolor Info: Driver file: xetex.def on input line 225. +(/usr/local/texlive/2018/texmf-dist/tex/latex/colortbl/colortbl.sty +Package: colortbl 2012/02/13 v1.0a Color table columns (DPC) +\everycr=\toks32 +\minrowclearance=\skip70 +) +LaTeX Info: Redefining \color on input line 709. +\rownum=\count295 +Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348. +Package xcolor Info: Model `RGB' extended on input line 1364. +Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366. +Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367. +Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368. +Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369. +Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370. +Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371. +) (/usr/local/texlive/2018/texmf-dist/tex/latex/wrapfig/wrapfig.sty +\wrapoverhang=\dimen182 +\WF@size=\dimen183 +\c@WF@wrappedlines=\count296 +\WF@box=\box51 +\WF@everypar=\toks33 +Package: wrapfig 2003/01/31 v 3.6 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/float/float.sty +Package: float 2001/11/08 v1.3d Float enhancements (AL) +\c@float@type=\count297 +\float@exts=\toks34 +\float@box=\box52 +\@float@everytoks=\toks35 +\@floatcapt=\box53 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/oberdiek/pdflscape.sty +Package: pdflscape 2016/05/14 v0.11 Display of landscape pages in PDF (HO) +(/usr/local/texlive/2018/texmf-dist/tex/latex/graphics/lscape.sty +Package: lscape 2000/10/22 v3.01 Landscape Pages (DPC) +) +Package pdflscape Info: Auto-detected driver: dvipdfm (xetex) on input line 99. + +) (/usr/local/texlive/2018/texmf-dist/tex/latex/tabu/tabu.sty +Package: tabu 2011/02/26 v2.8 - flexible LaTeX tabulars (FC) +(/usr/local/texlive/2018/texmf-dist/tex/latex/varwidth/varwidth.sty +Package: varwidth 2009/03/30 ver 0.92; Variable-width minipages +\@vwid@box=\box54 +\sift@deathcycles=\count298 +\@vwid@loff=\dimen184 +\@vwid@roff=\dimen185 +) +\c@taburow=\count299 +\tabu@nbcols=\count300 +\tabu@cnt=\count301 +\tabu@Xcol=\count302 +\tabu@alloc=\count303 +\tabu@nested=\count304 +\tabu@target=\dimen186 +\tabu@spreadtarget=\dimen187 +\tabu@naturalX=\dimen188 +\tabucolX=\dimen189 +\tabu@Xsum=\dimen190 +\extrarowdepth=\dimen191 +\abovetabulinesep=\dimen192 +\belowtabulinesep=\dimen193 +\tabustrutrule=\dimen194 +\tabu@thebody=\toks36 +\tabu@footnotes=\toks37 +\tabu@box=\box55 +\tabu@arstrutbox=\box56 +\tabu@hleads=\box57 +\tabu@vleads=\box58 +\tabu@cellskip=\skip71 +) +(/usr/local/texlive/2018/texmf-dist/tex/latex/threeparttable/threeparttable.sty +Package: threeparttable 2003/06/13 v 3.0 +\@tempboxb=\box59 +) +(/usr/local/texlive/2018/texmf-dist/tex/latex/threeparttablex/threeparttablex.s +ty +Package: threeparttablex 2013/07/23 v0.3 by daleif +(/usr/local/texlive/2018/texmf-dist/tex/latex/environ/environ.sty +Package: environ 2014/05/04 v0.3 A new way to define environments +(/usr/local/texlive/2018/texmf-dist/tex/latex/trimspaces/trimspaces.sty +Package: trimspaces 2009/09/17 v1.1 Trim spaces around a token list +)) +\TPTL@width=\skip72 +) (/usr/local/texlive/2018/texmf-dist/tex/generic/ulem/ulem.sty +\UL@box=\box60 +\UL@hyphenbox=\box61 +\UL@skip=\skip73 +\UL@hook=\toks38 +\UL@height=\dimen195 +\UL@pe=\count305 +\UL@pixel=\dimen196 +\ULC@box=\box62 +Package: ulem 2012/05/18 +\ULdepth=\dimen197 +) (/usr/local/texlive/2018/texmf-dist/tex/latex/makecell/makecell.sty +Package: makecell 2009/08/03 V0.1e Managing of Tab Column Heads and Cells +\rotheadsize=\dimen198 +\c@nlinenum=\count306 +\TeXr@lab=\toks39 +) +\c@theorem=\count307 +\c@lemma=\count308 +\c@definition=\count309 +\c@corollary=\count310 +\c@proposition=\count311 +\c@example=\count312 +\c@exercise=\count313 +No file Doing_Meta_Analysis_in_R.aux. +\openout1 = `Doing_Meta_Analysis_in_R.aux'. + +LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for TU/lmr/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for TS1/cmr/m/n on input line 167. +LaTeX Font Info: Try loading font information for TS1+cmr on input line 167. + +(/usr/local/texlive/2018/texmf-dist/tex/latex/base/ts1cmr.fd +File: ts1cmr.fd 2014/09/29 v2.5h Standard LaTeX font definitions +) +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Font Info: Checking defaults for PU/pdf/m/n on input line 167. +LaTeX Font Info: ... okay on input line 167. +LaTeX Info: Redefining \microtypecontext on input line 167. +Package microtype Info: Character protrusion enabled (level 2). +Package microtype Info: Using protrusion set `basicmath'. +Package microtype Info: No adjustment of tracking. +Package microtype Info: No adjustment of spacing. +Package microtype Info: No adjustment of kerning. + +(/usr/local/texlive/2018/texmf-dist/tex/latex/microtype/mt-LatinModernRoman.cfg +File: mt-LatinModernRoman.cfg 2013/03/13 v1.0 microtype config. file: Latin Mod +ern Roman (RS) +) +*geometry* driver: auto-detecting +*geometry* detected driver: xetex +*geometry* verbose mode - [ preamble ] result: +* driver: xetex +* paper: +* layout: +* layoutoffset:(h,v)=(0.0pt,0.0pt) +* modes: twoside +* h-part:(L,W,R)=(72.26999pt, 469.75502pt, 72.26999pt) +* v-part:(T,H,B)=(72.26999pt, 650.43001pt, 72.26999pt) +* \paperwidth=614.295pt +* \paperheight=794.96999pt +* \textwidth=469.75502pt +* \textheight=650.43001pt +* \oddsidemargin=0.0pt +* \evensidemargin=0.0pt +* \topmargin=-30.06749pt +* \headheight=12.0pt +* \headsep=18.06749pt +* \topskip=10.0pt +* \footskip=25.29494pt +* \marginparwidth=125.0pt +* \marginparsep=7.0pt +* \columnsep=10.0pt +* \skip\footins=9.0pt plus 4.0pt minus 2.0pt +* \hoffset=0.0pt +* \voffset=0.0pt +* \mag=1000 +* \@twocolumnfalse +* \@twosidetrue +* \@mparswitchtrue +* \@reversemarginfalse +* (1in=72.27pt=25.4mm, 1cm=28.453pt) + +\AtBeginShipoutBox=\box63 +Package hyperref Info: Link coloring OFF on input line 167. +(/usr/local/texlive/2018/texmf-dist/tex/latex/hyperref/nameref.sty +Package: nameref 2016/05/21 v2.44 Cross-referencing by name of section +(/usr/local/texlive/2018/texmf-dist/tex/generic/oberdiek/gettitlestring.sty +Package: gettitlestring 2016/05/16 v1.5 Cleanup title references (HO) +) +\c@section@level=\count314 +) +LaTeX Info: Redefining \ref on input line 167. +LaTeX Info: Redefining \pageref on input line 167. +LaTeX Info: Redefining \nameref on input line 167. +\@outlinefile=\write4 +\openout4 = `Doing_Meta_Analysis_in_R.out'. + + +Package hyperref Warning: Rerun to get /PageLabels entry. + +[1 + + +] [2 + +] +\tf@toc=\write5 +\openout5 = `Doing_Meta_Analysis_in_R.toc'. + +[3] [4 + +] +Chapter 1. +File: coverbild.jpg Graphic file (type bmp) + + +! LaTeX Error: Environment rmdinfo undefined. + +See the LaTeX manual or LaTeX Companion for explanation. +Type H for immediate help. + ... + +l.182 \begin{rmdinfo} + +Here is how much of TeX's memory you used: + 24521 strings out of 492970 + 446183 string characters out of 6133938 + 593173 words of memory out of 5000000 + 28291 multiletter control sequences out of 15000+600000 + 530818 words of font info for 34 fonts, out of 8000000 for 9000 + 1348 hyphenation exceptions out of 8191 + 45i,6n,68p,10465b,369s stack positions out of 5000i,500n,10000p,200000b,80000s + +Output written on Doing_Meta_Analysis_in_R.pdf (4 pages). diff --git a/Doing_Meta_Analysis_in_R.pdf b/Doing_Meta_Analysis_in_R.pdf new file mode 100644 index 0000000..d9daeee Binary files /dev/null and b/Doing_Meta_Analysis_in_R.pdf differ diff --git a/Doing_Meta_Analysis_in_R.tex b/Doing_Meta_Analysis_in_R.tex new file mode 100644 index 0000000..193a92a --- /dev/null +++ b/Doing_Meta_Analysis_in_R.tex @@ -0,0 +1,6535 @@ +\documentclass[]{book} +\usepackage{lmodern} +\usepackage{amssymb,amsmath} +\usepackage{ifxetex,ifluatex} +\usepackage{fixltx2e} % provides \textsubscript +\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex + \usepackage[T1]{fontenc} + \usepackage[utf8]{inputenc} +\else % if luatex or xelatex + \ifxetex + \usepackage{mathspec} + \else + \usepackage{fontspec} + \fi + \defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase} +\fi +% use upquote if available, for straight quotes in verbatim environments +\IfFileExists{upquote.sty}{\usepackage{upquote}}{} +% use microtype if available +\IfFileExists{microtype.sty}{% +\usepackage{microtype} +\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts +}{} +\usepackage[margin=1in]{geometry} +\usepackage{hyperref} +\hypersetup{unicode=true, + pdftitle={Doing Meta Analysis in R}, + pdfauthor={Mathias Harrer, B.Sc. \& Dr.~habil. David Ebert}, + pdfborder={0 0 0}, + breaklinks=true} +\urlstyle{same} % don't use monospace font for urls +\usepackage{natbib} +\bibliographystyle{apalike} +\usepackage{color} +\usepackage{fancyvrb} +\newcommand{\VerbBar}{|} +\newcommand{\VERB}{\Verb[commandchars=\\\{\}]} +\DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}} +% Add ',fontsize=\small' for more characters per line +\usepackage{framed} +\definecolor{shadecolor}{RGB}{248,248,248} +\newenvironment{Shaded}{\begin{snugshade}}{\end{snugshade}} +\newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}} +\newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{#1}} +\newcommand{\DecValTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}} +\newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}} +\newcommand{\FloatTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}} +\newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}} +\newcommand{\CharTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}} +\newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}} +\newcommand{\StringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}} +\newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}} +\newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}} +\newcommand{\ImportTok}[1]{#1} +\newcommand{\CommentTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}} +\newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}} +\newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}} +\newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}} +\newcommand{\OtherTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{#1}} +\newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}} +\newcommand{\VariableTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}} +\newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}} +\newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.81,0.36,0.00}{\textbf{#1}}} +\newcommand{\BuiltInTok}[1]{#1} +\newcommand{\ExtensionTok}[1]{#1} +\newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}} +\newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.77,0.63,0.00}{#1}} +\newcommand{\RegionMarkerTok}[1]{#1} +\newcommand{\InformationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}} +\newcommand{\WarningTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}} +\newcommand{\AlertTok}[1]{\textcolor[rgb]{0.94,0.16,0.16}{#1}} +\newcommand{\ErrorTok}[1]{\textcolor[rgb]{0.64,0.00,0.00}{\textbf{#1}}} +\newcommand{\NormalTok}[1]{#1} +\usepackage{longtable,booktabs} +\usepackage{graphicx,grffile} +\makeatletter +\def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth\else\Gin@nat@width\fi} +\def\maxheight{\ifdim\Gin@nat@height>\textheight\textheight\else\Gin@nat@height\fi} +\makeatother +% Scale images if necessary, so that they will not overflow the page +% margins by default, and it is still possible to overwrite the defaults +% using explicit options in \includegraphics[width, height, ...]{} +\setkeys{Gin}{width=\maxwidth,height=\maxheight,keepaspectratio} +\IfFileExists{parskip.sty}{% +\usepackage{parskip} +}{% else +\setlength{\parindent}{0pt} +\setlength{\parskip}{6pt plus 2pt minus 1pt} +} +\setlength{\emergencystretch}{3em} % prevent overfull lines +\providecommand{\tightlist}{% + \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}} +\setcounter{secnumdepth}{5} +% Redefines (sub)paragraphs to behave more like sections +\ifx\paragraph\undefined\else +\let\oldparagraph\paragraph +\renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}} +\fi +\ifx\subparagraph\undefined\else +\let\oldsubparagraph\subparagraph +\renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}} +\fi + +%%% Use protect on footnotes to avoid problems with footnotes in titles +\let\rmarkdownfootnote\footnote% +\def\footnote{\protect\rmarkdownfootnote} + +%%% Change title format to be more compact +\usepackage{titling} + +% Create subtitle command for use in maketitle +\newcommand{\subtitle}[1]{ + \posttitle{ + \begin{center}\large#1\end{center} + } +} + +\setlength{\droptitle}{-2em} + + \title{Doing Meta Analysis in R} + \pretitle{\vspace{\droptitle}\centering\huge} + \posttitle{\par} + \author{Mathias Harrer, B.Sc. \& Dr.~habil. David Ebert} + \preauthor{\centering\large\emph} + \postauthor{\par} + \predate{\centering\large\emph} + \postdate{\par} + \date{Friedrich-Alexander-University Erlangen-Nuremberg} + +\usepackage{booktabs} +\usepackage{amsthm} +\makeatletter +\def\thm@space@setup{% + \thm@preskip=8pt plus 2pt minus 4pt + \thm@postskip=\thm@preskip +} +\makeatother +\usepackage{booktabs} +\usepackage{longtable} +\usepackage{array} +\usepackage{multirow} +\usepackage[table]{xcolor} +\usepackage{wrapfig} +\usepackage{float} +\usepackage{colortbl} +\usepackage{pdflscape} +\usepackage{tabu} +\usepackage{threeparttable} +\usepackage{threeparttablex} +\usepackage[normalem]{ulem} +\usepackage{makecell} + +\usepackage{amsthm} +\newtheorem{theorem}{Theorem}[chapter] +\newtheorem{lemma}{Lemma}[chapter] +\theoremstyle{definition} +\newtheorem{definition}{Definition}[chapter] +\newtheorem{corollary}{Corollary}[chapter] +\newtheorem{proposition}{Proposition}[chapter] +\theoremstyle{definition} +\newtheorem{example}{Example}[chapter] +\theoremstyle{definition} +\newtheorem{exercise}{Exercise}[chapter] +\theoremstyle{remark} +\newtheorem*{remark}{Remark} +\newtheorem*{solution}{Solution} +\begin{document} +\maketitle + +{ +\setcounter{tocdepth}{1} +\tableofcontents +} +\chapter{About this Guide}\label{about-this-guide} + +\begin{figure} +\centering +\includegraphics{coverbild.jpg} +\caption{} +\end{figure} + +\begin{rmdinfo} +This guide shows you how to conduct Meta-Analyses in R from scratch. The +focus of this guide is primarily on clinical outcome research in +psychology. It is designed for staff and collaborators of the +\href{https://www.protectlab.org}{\textbf{PROTECT Lab}}, which is headed +by \textbf{Dr.~David D. Ebert}. +\end{rmdinfo} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-2-1.pdf} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\textbf{The guide will show you how to:} + +\begin{itemize} +\tightlist +\item + Get \textbf{R} and \textbf{RStudio} set for your Meta-Analysis +\item + Get your data into R +\item + \textbf{Prepare your data} for the meta-analysis +\item + Perform \textbf{fixed-effect} and \textbf{random-effects} + meta-analysis using the \texttt{meta} and \texttt{metafor}packages +\item + Analyse the \textbf{heterogeneity} of your results +\item + Tackle heterogeneity using \textbf{subgroup analyses} and + \textbf{meta-regression} +\item + Check if \textbf{selective outcome reporting (publication bias)} is a + present in your data +\item + Control for selective outcome reporting and publication bias +\item + Analyse the \textbf{risk of bias} in your data +\item + Do advanced types of meta-analyses, such as + + \begin{itemize} + \tightlist + \item + \textbf{network analyses} or + \item + meta-analyses with \textbf{more than one outcome} + \end{itemize} +\end{itemize} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\textbf{What this guide will not cover} + +Although this guide will provide some information on the statistics +behind meta-analysis, it will not give you an \textbf{in-depth +introduction} into how meta-analyses are calculated statistically. + +It is also beyond the scope of this guide to advise in detail which +meta-analytical strategy is suited best in which contexts, and on how +the search, study inclusion and reporting of meta-analyses should be +conducted. The \href{http://handbook-5-1.cochrane.org/}{\emph{Cochrane +Handbook for Systematic Reviews of Interventions}}, however, should be a +great source to find more information on these topics. + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\textbf{Generally, there a two other sources to recommended when +conducting Meta-Analyses:} + +\begin{itemize} +\tightlist +\item + If you're looking for a easily digestable, hands-on introduction on + how Meta-Analyses are conducted, we can recommend \textbf{Pim + Cuijpers' online courses on Meta-Analysis}. The courses are freely + available on YouTube. To have a look, click + \href{https://www.youtube.com/watch?v=pP7_VBrG_TY\&list=PL-h5cI5Bkvt0J-O0kq_9J9_aksWFPgR7s}{here}. +\item + If you're interested in more details on how to conduct Meta-Analyses + in R, you can either have a look at Wolfgang Viechtbauer's page for + the \texttt{metafor} package + (\href{http://metafor-project.org}{Link}). Or you can consult a book + on the \texttt{meta} package which was recently published + \citep{schwarzer2015meta}. +\end{itemize} + +\begin{rmdinfo} +\textbf{How to get the R code for this guide} + +\begin{figure} +\centering +\includegraphics{githublogo.png} +\caption{} +\end{figure} + +All code behind this book is available online on \textbf{GitHub}. We +have created a website containing a \textbf{download link} for all +codes, and a \textbf{quick guide} on how to get the code running on your +computer. The site can be found +\href{https://mathiasharrer.github.io/Doing-Meta-Analysis-in-R/}{here}. +\end{rmdinfo} + +\textbf{How to cite this guide} + +Harrer, M. \& Ebert, D. D. (2018). Doing Meta-Analysis in R: A practical +Guide. \emph{PROTECT Lab Friedrich-Alexander University +Erlangen-Nuremberg}. +\url{https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\textbf{To get started, proceed to the next chapter!} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{RStudio \& Basics}\label{rstudio-basics} + +\begin{figure} +\centering +\includegraphics{chap2.jpg} +\caption{} +\end{figure} + +\begin{rmdinfo} +Before we start with our meta-analysis, we have to download and prepare +a \textbf{computer program} which allows us to use \emph{R} for +programming. + +Probably the best option for this at the moment is \textbf{RStudio}. +This program gives us a user interface which makes it easier to overlook +our data, packages and output. The best part is that RStudio is +\textbf{completely free} and can be downloaded anytime. + +In this Chapter, we'll focus on how you can install RStudio on your +computer. We'll also provide some general information on R, and how you +can get help if you get error messages. + +If you already have RStudio installed on your computer, and if you're an +experienced R user already, all of this might be nothing new for you. +You may \textbf{skip} this chapter then. + +Especially if you have \textbf{never used R before, we would like to +consider this Chapter essential}, as it gives you some input on how R +works, and how we can use it for our data analyses. +\end{rmdinfo} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{RStudio}{\section{Getting RStudio to run on your +computer}\label{RStudio}} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-7-1.pdf} + +As a prerequisite for this guide, you need to have \textbf{RStudio} and +a few essential \textbf{R packages} installed. + +\textbf{You have to follow these steps to get your RStudio set.} + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + Download RStudio on the \textbf{RStudio} Website + (\href{https://www.rstudio.com/products/rstudio/download/}{Link}). + It's free! +\item + If you do not have \textbf{R} installed yet, you will have to install + the latest R Version before you can use RStudio. You can get the + latest R version + \href{https://cran.r-project.org/bin/windows/base/}{here}. +\item + Once RStudio is running, open the \textbf{Console} on the bottom left + corner of your screen. +\item + We will now install a few packages using R Code. Here's an overview of + the packages, and why we need them: +\end{enumerate} + +\begin{verbatim} +## Warning: package 'kableExtra' was built under R version 3.4.4 +\end{verbatim} + +\begin{tabular}{l|l} +\hline +Package & Description\\ +\hline +tidyverse & This is a large package containing various functions to tidy up your data\\ +\hline +meta & This package is a so-called wrapper for many meta-analytic functions and makes it easy to code different meta-analyses\\ +\hline +metafor & This package is used by the meta package for many applications, so we need to have it installed\\ +\hline +\end{tabular} + + 5. To install these packages, we use the \texttt{install.packages()} +function in R. One package after another, our code should look like +this: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{install.packages}\NormalTok{(}\StringTok{"tidyverse"}\NormalTok{)} +\KeywordTok{install.packages}\NormalTok{(}\StringTok{"meta"}\NormalTok{)} +\KeywordTok{install.packages}\NormalTok{(}\StringTok{"metafor"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{rmdachtung} +Don't forget to put the packages in \texttt{""}. + +Otherwise, you will get an error message. +\end{rmdachtung} + +\textbf{You are now set and ready to proceed. Below, you can find some +basic information on RStudio and troubleshooting} + +\subsection{Running R Code}\label{running-r-code} + +Order to get the most out of this guide, it's helpful (but not +essential) if you have some programming experience already. If you've +never programmed before, you might find \textbf{\emph{Hands-On +Programming with R}} \citep{grolemund2014hands} to be a useful primer. + +There are three things you need to run the code: \textbf{R}, +\textbf{RStudio}, and collection of \textbf{R packages}. Packages are +the fundamental units of reproducible R code. They include reusable +functions, the documentation that describes how to use them, and sample +data. + +Gladly, once you've reached this point successfully, these three things +are set already. Nevertheless, we will have to install and load a few +new packages at some place in this guide, for which you can use the +\texttt{install.packages()} the same way as you did before. + +Throughout the guide, a consistent set of conventions is used to refer +to code: + +\begin{itemize} +\tightlist +\item + Functions are in a code font and followed by parentheses, like + \texttt{sum()} or \texttt{mean()}. +\item + Other R objects (like data or function arguments) are in a code font, + without parentheses, like \texttt{seTE} or \texttt{method.tau}. +\item + Sometimes, we'll use the package name followed by two colons, like + \texttt{meta::metagen()}. This is also valid R code. This is used so + as to not confuse the functions of different packages with the same + name. +\end{itemize} + +\subsection{Getting Help}\label{getting-help} + +As you start to apply the techniques described in this guide to your +data you will soon find questions that the guide does not answer. This +section describes a few tips on how to get help. + +\begin{itemize} +\tightlist +\item + If you get stuck, start with \textbf{Google}. Typically, adding ``R'' + to a search is enough to restrict it to relevant results: if the + search isn't useful, it often means that there aren't any R-specific + results available. Google is particularly useful for error messages. + If you get an error message and you have no idea what it means, try + googling it. Chances are that someone else has been confused by it in + the past, and there will be help somewhere on the web. (If the error + message isn't in English, run \texttt{Sys.setenv(LANGUAGE\ =\ "en")} + and re-run the code; you're more likely to find help for English error + messages.) +\item + If Google doesn't help, try + \href{https://stackoverflow.com}{stackoverflow}. Start by spending a + little time searching for an existing answer; including {[}R{]} + restricts your search to questions and answers that use R. +\item + Lastly, if you stumble upon an error (or typos!) in this guide's text + or R syntax, feel free to contact \textbf{Mathias Harrer} at + \textbf{\href{mailto:mathias.harrer@fau.de}{\nolinkurl{mathias.harrer@fau.de}}}. +\end{itemize} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{Getting your data into R}\label{getting-your-data-into-r} + +\begin{figure} +\centering +\includegraphics{chappool.jpg} +\caption{} +\end{figure} + +\begin{rmdinfo} +This chapter will tell you about how you can \textbf{import} your effect +size data in RStudio. We will also show you a few commands which make it +easier to \textbf{manipulate data} directly in R. + +Data preparation can be tedious and exhausting at times, but it is the +backbone of all later steps when doing meta-analyses in R. We therefore +have to pay \textbf{close attention} to preparing the data correctly +before we can proceed. +\end{rmdinfo} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Data preparation in Excel}\label{data-preparation-in-excel} + +\hypertarget{excel_preparation}{\subsection{Setting the columns of the +excel spreadsheet}\label{excel_preparation}} + +To conduct Meta-Analyses in R, you need to have your study data +prepared. For a standard meta-analysis, the following information is +needed for every study. + +\begin{itemize} +\tightlist +\item + The \textbf{names} of the individual studies, so that they can be + easily identified later on. Usually, the first author and publication + year of a study is used for this (e.g. ``Ebert et al., 2018'') +\item + The \textbf{Mean} of both the Intervention and the Control group at + the same assessment point +\item + The \textbf{Standard Deviation} of both the Intervention and the + Control group at the same assessment point +\item + The \textbf{number of participants (N)} in each group of the trial +\item + If you want to have a look at differences between various study + subgroups later on, you also need a \textbf{subgroup code} for each + study which signifies to which subgroup it belongs. For example, if a + study was conducted in children, you might give it the subgroup code + ``children''. +\end{itemize} + +As per usual, such data is stored in \textbf{EXCEL spreadsheets}. We +recommend to store your data there, because this makes it very easy to +import data into RStudio. + +However, it is very important how you \textbf{name the columns of your +spreadsheet}. If you name the columns of your sheet adequately in EXCEL +already, you can save a lot of time because your data doesn't have to be +transformed in RStudio later on. + +\textbf{Here is how you should name the data columns in your EXCEL +spreadheet containing your Meta-Analysis data} + +\begin{tabular}{l|l} +\hline +Column & Description\\ +\hline +Author & This signifies the column for the study label (i.e., the first author)\\ +\hline +Me & The Mean of the experimental/intervention group\\ +\hline +Se & The Standard Deviation of the experimental/intervention group\\ +\hline +Mc & The Mean of the control group\\ +\hline +Sc & The Standard Deviation of the control group\\ +\hline +Ne & The number of participants in the experimental/intervention group\\ +\hline +Nc & The number of participats in the control group\\ +\hline +Subgroup & This is the label for one of your Subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g. population). In this column, each study should then be given an subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique\\ +\hline +\end{tabular} + +Note that it \textbf{doesn't matter how these columns are ordered in +your EXCEL spreadsheet}. They just have to be labeled correctly. + +There's also no need to \textbf{format} the columns in any way. If you +type the column name in the first line of you spreadsheet, R will +automatically detect it as a column name. + +\begin{rmdachtung} +It's also important to know that the import \textbf{will distort letters +like ä,ü,ö,á,é,ê, etc}. So be sure to transform them to ``normal'' +letters before you proceed. +\end{rmdachtung} + +\subsection{Setting the columns of your sheet if you have calculated the +effect sizes of each study +already}\label{setting-the-columns-of-your-sheet-if-you-have-calculated-the-effect-sizes-of-each-study-already} + +If you have \textbf{already calculated the effect sizes for each study +on your own}, for example using \emph{Comprehensive Meta-Analysis} or +\emph{RevMan}, there's another way to prepare your data which makes +things a little easier. In this case, you only have to include the +following columns: + +\begin{tabular}{l|l} +\hline +Column & Description\\ +\hline +Author & This signifies the column for the study label (i.e., the first author)\\ +\hline +TE & The calculated effect size of the study (either Cohen's d or Hedges' g, or some other form of effect size\\ +\hline +seTE & The Standard Error (SE) of the calculated effect\\ +\hline +Subgroup & This is the label for one of your Subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g. population). In this column, each study should then be given an subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique\\ +\hline +\end{tabular} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{import_excel}{\section{Importing the Spreadsheet into +Rstudio}\label{import_excel}} + +To get our data into R, we need to \textbf{save our data in a format and +at a place where RStudio can open it}. + +\subsection{Saving the data in the right +format}\label{saving-the-data-in-the-right-format} + +Generelly, finding the right format to import EXCEL can be tricky. + +\begin{itemize} +\tightlist +\item + If you're using a PC or Mac, it is advised to save your EXCEL sheet as + a \textbf{comma-separated-values (.csv) file}. This can be done by + clicking on ``save as'' and then choosing (.csv) as the output format + in EXCEL. +\item + With some PCs, RStudios might not be able to open such files, or the + files might be distorted. In this case, you can also try to save the + sheet as a normal \textbf{.xslx} EXCEL-file and try if this works. +\end{itemize} + +\subsection{Saving the data in your working +directory}\label{saving-the-data-in-your-working-directory} + +To function properly, you have to set a working directory for RStudio +first. The working directory is a \textbf{folder on your computer from +which RStudio can use data, and in which output it saved}. + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + Therefore, create a folder on your computer and give it a meaningful + name (e.g. ``My Meta-Analysis''). +\item + Save your spreadsheet in the folder +\item + Set the folder as your working directory. This can be done in RStudio + on the \textbf{bottom-right corner of your screen}. Under the tile + \textbf{``Files''}, search for the folder on your computer, and open + it. +\item + Once you've opened your folder, the file you just saved there should + be in there. +\item + Now that you've opened the folder, click on the \textbf{little gear + wheel on top of the pane} +\end{enumerate} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-15-1.pdf} + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\setcounter{enumi}{5} +\tightlist +\item + Then click on ``\textbf{Set as working directory}'' +\end{enumerate} + +\textbf{Your file, and the working directory, are now where they should +be!} + +\subsection{Loading the data}\label{loading-the-data} + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + To import the data, simply \textbf{click on the file} in the + bottom-right pane. Then click on \textbf{import dataset\ldots{}} +\item + An \textbf{import assistant} should now pop up, which is also loading + a preview of your data. This can be time-consuming sometimes, so you + can skip this step if you want to, and klick straight on + \textbf{``import''} +\end{enumerate} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-16-1.pdf} + +As you can see, the on the top-right pane \textbf{Environment}, your +file is now listed as a data set in your RStudio environment. + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\setcounter{enumi}{2} +\tightlist +\item + I also want to give my data a shorter name (``madata''). To rename it, + i use the following code: +\end{enumerate} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata<-Meta_Analysis_Data} +\end{Highlighting} +\end{Shaded} + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\setcounter{enumi}{3} +\tightlist +\item + Now, let's have a look at the \textbf{structure of my data} using the + \texttt{str()} function +\end{enumerate} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(madata)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Classes 'tbl_df', 'tbl' and 'data.frame': 18 obs. of 17 variables: +## $ Author : chr "Call et al." "Cavanagh et al." "DanitzOrsillo" "de Vibe et al." ... +## $ TE : num 0.709 0.355 1.791 0.182 0.422 ... +## $ seTE : num 0.261 0.196 0.346 0.118 0.145 ... +## $ RoB : chr "low" "low" "high" "low" ... +## $ Control : chr "WLC" "WLC" "WLC" "no intervention" ... +## $ intervention duration: chr "short" "short" "short" "short" ... +## $ intervention type : chr "mindfulness" "mindfulness" "ACT" "mindfulness" ... +## $ population : chr "undergraduate students" "students" "undergraduate students" "undergraduate students" ... +## $ type of students : chr "psychology" "general" "general" "general" ... +## $ prevention type : chr "selective" "universal" "universal" "universal" ... +## $ gender : chr "female" "mixed" "mixed" "mixed" ... +## $ mode of delivery : chr "group" "online" "group" "group" ... +## $ ROB streng : chr "high" "low" "high" "low" ... +## $ ROB superstreng : chr "high" "high" "high" "low" ... +## $ compensation : chr "none" "none" "voucher/money" "voucher/money" ... +## $ instruments : chr "DASS" "PSS" "DASS" "other" ... +## $ guidance : chr "f2f" "self-guided" "f2f" "f2f" ... +\end{verbatim} + +Although this output looks kind of messy, it's already very informative. +It shows the structure of my data. In this case, i used data for which +the effect sizes were already calculated. This is why the variables +\textbf{TE} and \textbf{seTE} appear. I also see plenty of other +variables, which correspond to the subgroups which were coded for this +dataset. + +\textbf{Here's a (shortened) table created for my data} + +\begin{tabular}{l|r|r|l|l|l|l} +\hline +Author & TE & seTE & RoB & Control & intervention duration & intervention type\\ +\hline +Call et al. & 0.7091362 & 0.2608202 & low & WLC & short & mindfulness\\ +\hline +Cavanagh et al. & 0.3548641 & 0.1963624 & low & WLC & short & mindfulness\\ +\hline +DanitzOrsillo & 1.7911700 & 0.3455692 & high & WLC & short & ACT\\ +\hline +de Vibe et al. & 0.1824552 & 0.1177874 & low & no intervention & short & mindfulness\\ +\hline +Frazier et al. & 0.4218509 & 0.1448128 & low & information only & short & PCI\\ +\hline +Frogeli et al. & 0.6300000 & 0.1960000 & low & no intervention & short & ACT\\ +\hline +Gallego et al. & 0.7248838 & 0.2246641 & high & no intervention & long & mindfulness\\ +\hline +Hazlett-Stevens \& Oren & 0.5286638 & 0.2104609 & low & no intervention & long & mindfulness\\ +\hline +Hintz et al. & 0.2840000 & 0.1680000 & low & information only & short & PCI\\ +\hline +Kang et al. & 1.2750682 & 0.3371997 & low & no intervention & long & mindfulness\\ +\hline +Kuhlmann et al. & 0.1036082 & 0.1947275 & low & no intervention & short & mindfulness\\ +\hline +Lever Taylor et al. & 0.3883906 & 0.2307689 & low & WLC & long & mindfulness\\ +\hline +Phang et al. & 0.5407398 & 0.2443133 & low & no intervention & short & mindfulness\\ +\hline +Rasanen et al. & 0.4261593 & 0.2579379 & low & WLC & short & ACT\\ +\hline +Ratanasiripong & 0.5153969 & 0.3512737 & high & no intervention & short & mindfulness\\ +\hline +Shapiro et al. & 1.4797260 & 0.3152817 & low & WLC & long & mindfulness\\ +\hline +SongLindquist & 0.6125782 & 0.2266834 & high & WLC & long & mindfulness\\ +\hline +Warnecke et al. & 0.6000000 & 0.2490000 & low & information only & long & mindfulness\\ +\hline +\end{tabular} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Data manipulation}\label{data-manipulation} + +Now that we have the Meta-Analysis data in RStudio, let's do a +\textbf{few manipulations with the data}. These functions might come in +handy when were conducting analyses later on. + +Going back to the output of the \texttt{str()} function, we see that +this also gives us details on the type of column data we have stored in +our data. There a different abbreviations signifying different types of +data. + +\begin{tabular}{l|l|l} +\hline +Abbreviation & Type & Description\\ +\hline +num & Numerical & This is all data stored as numbers (e.g. 1.02)\\ +\hline +chr & Character & This is all data stored as words\\ +\hline +log & Logical & These are variables which are binary, meaning that they signify that a condition is either TRUE or FALSE\\ +\hline +factor & Factor & Factors are stored as numbers, with each number signifying a different level of a variable. A possible factor of a variable might be 1 = low, 2 = medium, 3 = high\\ +\hline +\end{tabular} + +\hypertarget{convertfactors}{\subsection{Converting to +factors}\label{convertfactors}} + +Let's say we have the subgroup \textbf{Risk of Bias} (in which the Risk +of Bias rating is coded), and want it to be a factor with two different +levels: ``low'' and ``high''. + +To do this, we need to the variable \texttt{ROB} to be a factor. +However, this variable is currently stored as a character +(\texttt{chr}). We can have a look at this variable by typing the name +of our dataset, then adding the selector \texttt{\$} and then adding the +variable we want to have a look at. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata}\OperatorTok{$}\NormalTok{ROB} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] "high" "low" "high" "low" "low" "low" "high" "low" "low" "low" +## [11] "high" "low" "low" "low" "high" "high" "high" "low" +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(madata}\OperatorTok{$}\NormalTok{ROB)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## chr [1:18] "high" "low" "high" "low" "low" "low" "high" "low" "low" ... +\end{verbatim} + +We can see now that \texttt{ROB}is indeed a \textbf{character} type +variable, which contains only two words: ``low'' and ``high''. We want +to convert this to a \textbf{factor} variable now, which has only two +levels, low and high. To do this, we use the \texttt{factor()} function. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata}\OperatorTok{$}\NormalTok{ROB<-}\KeywordTok{factor}\NormalTok{(madata}\OperatorTok{$}\NormalTok{ROB)} +\NormalTok{madata}\OperatorTok{$}\NormalTok{ROB} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] high low high low low low high low low low high low low low +## [15] high high high low +## Levels: high low +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(madata}\OperatorTok{$}\NormalTok{ROB)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Factor w/ 2 levels "high","low": 1 2 1 2 2 2 1 2 2 2 ... +\end{verbatim} + +We now see that the variable has been \textbf{converted to a factor with +the levels ``high'' and ``low''}. + +\subsection{Converting to logicals}\label{converting-to-logicals} + +Now lets have a look at the \textbf{intervention type} subgroup +variable. This variable is currently stores as a character \texttt{chr} +too. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] "mindfulness" "mindfulness" "ACT" "mindfulness" "PCI" +## [6] "ACT" "mindfulness" "mindfulness" "PCI" "mindfulness" +## [11] "mindfulness" "mindfulness" "mindfulness" "ACT" "mindfulness" +## [16] "mindfulness" "mindfulness" "mindfulness" +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## chr [1:18] "mindfulness" "mindfulness" "ACT" "mindfulness" "PCI" ... +\end{verbatim} + +Let's say we want a variable which only contains information if a study +way a mindfulness intervention or not. A logical is very well suited for +this. To convert the data to logical, we use the \texttt{as.logical} +function. We will create a new variable containing this information +called \texttt{intervention.type.logical}. To tell R what to count as +\texttt{TRUE} and what as \texttt{FALSE}, we have to define the specific +intervention type using the \texttt{==} command. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{intervention.type.logical<-}\KeywordTok{as.logical}\NormalTok{(madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}\OperatorTok{==}\StringTok{"mindfulness"}\NormalTok{)} +\NormalTok{intervention.type.logical} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE FALSE TRUE TRUE +## [12] TRUE TRUE FALSE TRUE TRUE TRUE TRUE +\end{verbatim} + +We see that R has converted the character information into trues and +falses for us. To check if this was done correctly, let's compare the +original and the new variable + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{n<-}\KeywordTok{data.frame}\NormalTok{(intervention.type.logical,madata}\OperatorTok{$}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`}\NormalTok{)} +\NormalTok{names<-}\KeywordTok{c}\NormalTok{(}\StringTok{"New"}\NormalTok{, }\StringTok{"Original"}\NormalTok{)} +\KeywordTok{colnames}\NormalTok{(n)<-names} +\KeywordTok{kable}\NormalTok{(n)} +\end{Highlighting} +\end{Shaded} + +\begin{tabular}{l|l} +\hline +New & Original\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +FALSE & ACT\\ +\hline +TRUE & mindfulness\\ +\hline +FALSE & PCI\\ +\hline +FALSE & ACT\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +FALSE & PCI\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +FALSE & ACT\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +TRUE & mindfulness\\ +\hline +\end{tabular} + +\hypertarget{select}{\subsection{Selecting specific +studies}\label{select}} + +It may often come in handy to \textbf{select certain studies for further +analyses}, or to \textbf{exclude some studies in further analyses} +(e.g., if they are outliers). + +To do this, we can use the \texttt{filter} function in the +\texttt{dplyr}package, which is part of the \texttt{tidyverse} package +we installed before. + +So, let's load the package first. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(dplyr)} +\end{Highlighting} +\end{Shaded} + +Let's say we want to do a Meta-Analysis with three studies in our +dataset only. To do this, we need to create a new dataset containing +only these studies using the \texttt{dplyr::filter()} function. The +\texttt{dplyr::} part is necessary as there is more than one +`\texttt{filter} function in R, and we want to use to use the one of the +\texttt{dplyr}package. + +Let's say we want to have the studies by \textbf{Cavanagh et al.}, +\textbf{Frazier et al.} and \textbf{Phang et al.} stored in another +dataset, so we can conduct analyses only for these studies. + +The R code to store these three studies in a new dataset called +\texttt{madata.new} looks like this: + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata.new<-dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(madata,Author }\OperatorTok{%in%}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"Cavanagh et al."}\NormalTok{,} + \StringTok{"Frazier et al."}\NormalTok{,} + \StringTok{"Phang et al."}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Warning: package 'bindrcpp' was built under R version 3.4.4 +\end{verbatim} + +Note that the \texttt{\%in\%}-Command tells the \texttt{filter} function +to search for exactly the three cases we defined in the variable +\texttt{Author}. Now, let's have a look at the new data +\texttt{madata.new} we just created. + +\begin{tabular}{l|r|r|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l} +\hline +Author & TE & seTE & RoB & Control & intervention duration & intervention type & population & type of students & prevention type & gender & mode of delivery & ROB streng & ROB superstreng & compensation & instruments & guidance & ROB\\ +\hline +Cavanagh et al. & 0.3548641 & 0.1963624 & low & WLC & short & mindfulness & students & general & universal & mixed & online & low & high & none & PSS & self-guided & low\\ +\hline +Frazier et al. & 0.4218509 & 0.1448128 & low & information only & short & PCI & students & psychology & universal & mixed & online & low & low & credit & PSS & reminders & low\\ +\hline +Phang et al. & 0.5407398 & 0.2443133 & low & no intervention & short & mindfulness & students & medical studens & selective & mixed & group & low & low & none & PSS & f2f & low\\ +\hline +\end{tabular} + +Note that the function can also be used for any other type of data and +variable. We can also use it to e.g., only select studies which were +coded as being a mindfulness study + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata.new.mf<-dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(madata,}\StringTok{`}\DataTypeTok{intervention type}\StringTok{`} \OperatorTok{%in%}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"mindfulness"}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +We can also use the \texttt{dplyr::filter()} function to exclude studies +from our dataset. To do this, we only have to add \texttt{!} in front of +the variable we want to use for filtering. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata.new.excl<-dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(madata,}\OperatorTok{!}\NormalTok{Author }\OperatorTok{%in%}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"Cavanagh et al."}\NormalTok{,} + \StringTok{"Frazier et al."}\NormalTok{,} + \StringTok{"Phang et al."}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +\subsection{Changing cell values}\label{changing-cell-values} + +Sometimes, even when preparing your data in EXCEL, you might want to +\textbf{change values in RStudio once you have imported your data}. + +To do this, we have to select a cell in our data frame in RStudio. This +can be done by adding \texttt{{[}x,y{]}} to our dataset name, where +\textbf{x} signifies the number of the \textbf{row} we want to select, +and \textbf{y} signifies the number of the \textbf{column}. + +To see how this works, let's select a variable using this command first: + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata[}\DecValTok{6}\NormalTok{,}\DecValTok{1}\NormalTok{]} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## # A tibble: 1 x 1 +## Author +## +## 1 Frogeli et al. +\end{verbatim} + +We now see the \textbf{6th study} in our dataframe, and the value of +this study for \textbf{Column 1 (the author name)} is displayed. Let's +say we had a typo in this name and want to have it changed. In this +case, we have to give this exact cell a new value. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata[}\DecValTok{6}\NormalTok{,}\DecValTok{1}\NormalTok{]<-}\StringTok{"Frogelli et al."} +\end{Highlighting} +\end{Shaded} + +Let's check if the name has changed. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{madata[}\DecValTok{6}\NormalTok{,}\DecValTok{1}\NormalTok{]} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## # A tibble: 1 x 1 +## Author +## +## 1 Frogelli et al. +\end{verbatim} + +You can also use this function to change any other type of data, +including numericals and logicals. Only for characters, you have to put +the values you want to insert in \texttt{""}. + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{Pooling Effect Sizes}\label{pool} + +\begin{figure} +\centering +\includegraphics{pooling.jpg} +\caption{} +\end{figure} + +Now, let's get to the core of every Meta-Analysis: \textbf{pooling your +effect sizes} to get one overall effect size estimate of the studies. + +\begin{rmdinfo} +When pooling effect sizes in Meta-Analysis, there are two approaches +which we can use: the \textbf{Fixed-Effect-Model}, or the +\textbf{Random-Effects-Model} {[}@borenstein2011{]}. There is an +extensive debate on which model fits best in which context +{[}@fleiss1993review{]}, with no clear consensus in sight. Although it +has been recommended to \textbf{only resort to the +Random-Effects-Pooling model} in clinical psychology and the health +sciences {[}@cuijpers2016meta{]}, we will describe how to conduct both +in R here. + +Both of these models only require an \textbf{effect size}, and a +\textbf{dispersion (variance)} estimate for each study, of which the +inverse is taken. This is why the methods are often called +\textbf{generic inverse-variance methods}. +\end{rmdinfo} + +We will describe in-depth how to conduct meta-analyses in R with +\textbf{continuous variables} (such as effect sizes), as these are the +most common ones in psychology and the health science field. Later on, +we will present briefly how to do meta-analyses with \textbf{binary +outcome} data too, which might be important if you're focusing on +prevention trials. + +For these meta-analyses, we'll use the \texttt{meta} package +\citep{schwarzer2007meta}. In \protect\hyperlink{RStudio}{Section 2.1}, +we showed how to install the package. Now, load the package from your +library to proceed. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(meta)} +\end{Highlighting} +\end{Shaded} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{fixed}{\section{Fixed-Effects-Model}\label{fixed}} + +\hypertarget{pre.calc}{\subsection{Pre-calculated effect size +data}\label{pre.calc}} + +\begin{rmdinfo} +\textbf{The idea behind the fixed-effects-model} + +The fixed-effects-model assumes that all studies along with their effect +sizes stem from a single homogeneous population {[}@borenstein2011{]}. +To calculate the overall effect, we therefore average all effect sizes, +but give studies with greater precision a higher weight. In this case, +greater precision means that the study has a larger \textbf{N}, which +leads to a smaller \textbf{Standard Error} of its effect size estimate. + +For this weighing, we use the \textbf{inverse of the variance} +\(1/\hat\sigma^2_k\) of each study \(k\). We then calculate a weighted +average of all studies, our fixed effect size estimator +\(\hat\theta_F\): +\end{rmdinfo} + +\begin{equation} +\hat\theta_F = \frac{\sum\limits_{k=1}^K \hat\theta_k/ \hat\sigma^2_k}{\sum\limits_{k=1}^K 1/\hat\sigma^2_k} +\end{equation} + +In \protect\hyperlink{excel_preparation}{Chapter 3.1}, we have described +two ways your EXCEL spreadsheet for your meta-analysis data can look +like: + +\begin{itemize} +\tightlist +\item + It can either be stored as the \textbf{raw data} (including the Mean, + N, and SD of every study arm) +\item + Or it only contains the \textbf{calculated effect sizes and the + standard error (SE)} +\end{itemize} + +The functions to pool the results with a fixed-effect-model +\textbf{differ depending on which data format you used}, but not much. +First, let's assume you already have a dataset with the +\textbf{calucated effects and SE} for each study. In my case, this is my +\texttt{madata} dataset. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(madata)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Classes 'tbl_df', 'tbl' and 'data.frame': 18 obs. of 17 variables: +## $ Author : chr "Call et al." "Cavanagh et al." "DanitzOrsillo" "de Vibe et al." ... +## $ TE : num 0.709 0.355 1.791 0.182 0.422 ... +## $ seTE : num 0.261 0.196 0.346 0.118 0.145 ... +## $ RoB : chr "low" "low" "high" "low" ... +## $ Control : chr "WLC" "WLC" "WLC" "no intervention" ... +## $ intervention duration: chr "short" "short" "short" "short" ... +## $ intervention type : chr "mindfulness" "mindfulness" "ACT" "mindfulness" ... +## $ population : chr "undergraduate students" "students" "undergraduate students" "undergraduate students" ... +## $ type of students : chr "psychology" "general" "general" "general" ... +## $ prevention type : chr "selective" "universal" "universal" "universal" ... +## $ gender : chr "female" "mixed" "mixed" "mixed" ... +## $ mode of delivery : chr "group" "online" "group" "group" ... +## $ ROB streng : chr "high" "low" "high" "low" ... +## $ ROB superstreng : chr "high" "high" "high" "low" ... +## $ compensation : chr "none" "none" "voucher/money" "voucher/money" ... +## $ instruments : chr "DASS" "PSS" "DASS" "other" ... +## $ guidance : chr "f2f" "self-guided" "f2f" "f2f" ... +\end{verbatim} + +This dataset has \textbf{continuous outcome data}. As our effect sizes +are already calculated, we can use the \texttt{meta::metagen} function. +For this function, we can specify loads of parameters, all of which you +can accessed by typing \texttt{?metagen} in your console once the +\texttt{meta} package is loaded. + +\textbf{Here is a table with the most important parameters for our +code:} + +\begin{tabular}{l|l} +\hline +Parameter & Function\\ +\hline +TE & This tells R to use the TE column to retrieve the effect sizes for each study\\ +\hline +seTE & This tells R to use the seTE column to retrieve the standard error for each study\\ +\hline +data= & After =, paste the name of your dataset here\\ +\hline +studlab=paste() & This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)\\ +\hline +comb.fixed= & Weather to use a fixed-effect-model\\ +\hline +comb.random & Weather to use a random-effects-model\\ +\hline +prediction= & Weather to print a prediction interval for the effect of future studies based on present evidence\\ +\hline +sm= & The summary measure we want to calculate. We can either calculate the mean difference (MD) or Hedges' g/Cohen's d (SMD)\\ +\hline +\end{tabular} + +Let's code our first fixed-effects-model Meta-Analysis. We we will give +the results of this analysis the simple name \texttt{m}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m<-}\KeywordTok{metagen}\NormalTok{(TE,} +\NormalTok{ seTE,} + \DataTypeTok{data=}\NormalTok{madata,} + \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),} + \DataTypeTok{comb.fixed =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{comb.random =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)} +\NormalTok{m} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(fixed) +## Call et al. 0.7091 [ 0.1979; 1.2203] 3.6 +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 6.3 +## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 2.0 +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 17.5 +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 11.6 +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 6.3 +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 4.8 +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 5.5 +## Hintz et al. 0.2840 [-0.0453; 0.6133] 8.6 +## Kang et al. 1.2751 [ 0.6142; 1.9360] 2.1 +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 6.4 +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 4.6 +## Phang et al. 0.5407 [ 0.0619; 1.0196] 4.1 +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 3.6 +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 2.0 +## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 2.4 +## SongLindquist 0.6126 [ 0.1683; 1.0569] 4.7 +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 3.9 +## +## Number of studies combined: k = 18 +## +## SMD 95%-CI z p-value +## Fixed effect model 0.4805 [ 0.3840; 0.5771] 9.75 < 0.0001 +## Prediction interval [-0.0344; 1.1826] +## +## Quantifying heterogeneity: +## tau^2 = 0.0752; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 45.50 17 0.0002 +## +## Details on meta-analytical method: +## - Inverse variance method +\end{verbatim} + +We now see the results of our Meta-Analysis, including + +\begin{itemize} +\tightlist +\item + The \textbf{individual effect sizes} for each study, and their weight +\item + The total \textbf{number of included studies} (k) +\item + The \textbf{overall effect} (in our case, \emph{g} = 0.4805) and its + confidence interval and p-value +\item + Measures of \textbf{between-study heterogeneity}, such as + \emph{tau\textsuperscript{2}} or \emph{I\textsuperscript{2}} and a + \emph{Q}-test of heterogeneity +\end{itemize} + +Using the \texttt{\$} command, we can also have a look at various +outputs directly. For example + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m}\OperatorTok{$}\NormalTok{lower.I2} +\end{Highlighting} +\end{Shaded} + +Gives us the lower bound of the 95\% confidence interval for +\emph{I\textsuperscript{2}} + +\begin{verbatim} +## [1] 0.3787897 +\end{verbatim} + +We can \textbf{save the results of the meta-analysis} to our working +directory as a .txt-file using this command + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{sink}\NormalTok{(}\StringTok{"results.txt"}\NormalTok{)} +\KeywordTok{print}\NormalTok{(m)} +\KeywordTok{sink}\NormalTok{()} +\end{Highlighting} +\end{Shaded} + +\subsection{Raw effect size data}\label{fixed.raw} + +To conduct a fixed-effects-model Meta-Analysis from \textbf{raw data} +(i.e, if your data has been prepared the way we describe in +\protect\hyperlink{excel_preparation}{Chapter 3.1.1}), we have to use +the \texttt{meta::metacont()} function instead. The structure of the +code however, looks quite similar. + +\begin{tabular}{l|l} +\hline +Parameter & Function\\ +\hline +Ne & The number of participants (N) in the intervention group\\ +\hline +Me & The Mean (M) of the intervention group\\ +\hline +Se & The Standard Deviation (SD) of the intervention group\\ +\hline +Nc & The number of participants (N) in the control group\\ +\hline +Mc & The Mean (M) of the control group\\ +\hline +Sc & The Standard Deviation (SD) of the control group\\ +\hline +data= & After '=', paste the name of your dataset here\\ +\hline +studlab=paste() & This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)\\ +\hline +comb.fixed= & Weather to use a fixed-effects-model\\ +\hline +comb.random & Weather to use a random-effects-model\\ +\hline +prediction= & Weather to print a prediction interval for the effect of future studies based on present evidence\\ +\hline +sm= & The summary measure we want to calculate. We can either calculate the mean difference (MD) or Hedges' g (SMD)\\ +\hline +\end{tabular} + +For this purpose, i will use my dataset \texttt{metacont}, which +contains the raw data of all studies i want to snythesize + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(metacont)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Classes 'tbl_df', 'tbl' and 'data.frame': 6 obs. of 7 variables: +## $ Author: chr "Cavanagh" "Day" "Frazier" "Gaffney" ... +## $ Ne : num 50 64 90 30 77 60 +## $ Me : num 4.5 18.3 12.5 2.34 15.21 ... +## $ Se : num 2.7 6.4 3.2 0.87 5.35 ... +## $ Nc : num 50 65 95 30 69 60 +## $ Mc : num 5.6 20.2 15.5 3.13 20.13 ... +## $ Sc : num 2.6 7.6 4.4 1.23 7.43 ... +\end{verbatim} + +Now, let's code the Meta-Analysis function, this time using the +\texttt{meta::metacont} function, and my \texttt{metacont} dataset. I +want to name my output \texttt{m.raw} now. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.raw<-}\KeywordTok{metacont}\NormalTok{(Ne,} +\NormalTok{ Me,} +\NormalTok{ Se,} +\NormalTok{ Nc,} +\NormalTok{ Mc,} +\NormalTok{ Sc,} + \DataTypeTok{data=}\NormalTok{metacont,} + \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),} + \DataTypeTok{comb.fixed =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{comb.random =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)} +\NormalTok{m.raw} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(fixed) +## Cavanagh -0.4118 [-0.8081; -0.0155] 13.8 +## Day -0.2687 [-0.6154; 0.0781] 18.0 +## Frazier -0.7734 [-1.0725; -0.4743] 24.2 +## Gaffney -0.7303 [-1.2542; -0.2065] 7.9 +## Greer -0.7624 [-1.0992; -0.4256] 19.1 +## Harrer -0.1669 [-0.5254; 0.1916] 16.9 +## +## Number of studies combined: k = 6 +## +## SMD 95%-CI z p-value +## Fixed effect model -0.5245 [-0.6718; -0.3773] -6.98 < 0.0001 +## Prediction interval [-1.1817; 0.1494] +## +## Quantifying heterogeneity: +## tau^2 = 0.0441; H = 1.51 [1.00; 2.38]; I^2 = 56.1% [0.0%; 82.3%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 11.39 5 0.0441 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Hedges' g (bias corrected standardised mean difference) +\end{verbatim} + +\begin{rmdachtung} +As you can see, all the calculated effect sizes are \textbf{negative} +now, including the pooled effect. However, all studies report a positive +outcome, meaning that the symptoms in the intervention group (e.g., of +depression) were reduced. The negative orientation results from the fact +that in \textbf{most clinical trials, lower scores indicate better +outcomes} (e.g., less depression). It is no problem to report values +like this: in fact, it is conventional. + +Some readers who are unfamiliar with meta-analysis, however, +\textbf{might be confused} by this, so may consider changing the +orientation of your values before you report them in your paper. +\end{rmdachtung} + +We can \textbf{save the results of the meta-analysis} to our working +directory as a .txt-file using this command + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{sink}\NormalTok{(}\StringTok{"results.txt"}\NormalTok{)} +\KeywordTok{print}\NormalTok{(m.raw)} +\KeywordTok{sink}\NormalTok{()} +\end{Highlighting} +\end{Shaded} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{random}{\section{Random-Effects-Model}\label{random}} + +Previously, we showed how to perform a fixed-effect-model meta-analysis +using the \texttt{meta:metagen} and \texttt{meta:metacont} functions. + +However, we can only use the fixed-effect-model when we can assume that +\textbf{all included studies come from the same population}. In practice +this is hardly ever the case: interventions may vary in certain +characteristics, the sample used in each study might be slightly +different, or its methods. In this case, we cannot assume that all +studies stem from one hypothesized ``population'' of studies. + +Same is the case once we detect \textbf{statistical heterogeneity} in +our fixed-effect-model meta-analysis, as indicated by \(I^{2}>0\). + +So, it is very likely that you will actually use a random-effects-model +for your meta-analysis. Thankfully, there's not much more we have to +think about when conducting a random-effects-model meta-analysis in R +instead of a fixed-effect-model meta-analysis. + +\begin{rmdinfo} +\textbf{The Idea behind the Random-Effects-Model} + +In the Random-Effects-Model, we want to account for our assumption that +the study effect estimates show more variance than when drawn from a +single population {[}@schwarzer2015meta{]}. The random-effects-model +works under the so-called \textbf{assumption of exchangeability}. + +This means that in Random-Effects-Model Meta-Analyses, we not only +assume that effects of individual studies deviate from the true +intervention effect of all studies due to sampling error, but that there +is another source of variance introduced by the fact that the studies do +not stem from one single population, but are drawn from a ``universe'' +of populations. We therefore assume that there is not only one true +effect size, but \textbf{a distribution of true effect sizes}. We +therefore want to estimate the mean of this distribution of true effect +sizes. + +The fixed-effect-model assumes that when the observed effect size +\(\hat\theta_k\) of an individual study \(k\) deviates from the true +effect size \(\theta_F\), the only reason for this is that the estimate +is burdened by (sampling) error \(\epsilon_k\). + +\[\hat\theta_k = \theta_F + \epsilon_k\] + +While the random-effects-model assumes that, in addition, there is +\textbf{a second source of error} \(\zeta_k\).This second source of +error is introduced by the fact that even the true effect size +\(\theta_k\) of our study \(k\) is also only part of an over-arching +distribution of true effect sizes with the mean \(\mu\) +{[}@borenstein2011{]}. +\end{rmdinfo} + +\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-54-1} \end{center} + +\emph{An illustration of parameters of the random-effects-model} + +\begin{rmdinfo} +The formula for the random-effects-model therefore looks like this: + +\[\hat\theta_k = \mu + \epsilon_k + \zeta_k\] + +When calculating a random-effects-model meta-analysis, where therefore +also have to take the error \(\zeta_k\) into account. To do this, we +have to \textbf{estimate the variance of the distribution of true effect +sizes}, which is denoted by \(\tau^{2}\), or +\emph{tau\textsuperscript{2}}. There are several estimators for +\(\tau^{2}\), all of which are implemented in \texttt{meta}. We will +give you more details about them in the next section. +\end{rmdinfo} + +\begin{rmdachtung} +Even though it is \textbf{conventional} to use random-effects-model +meta-analyses in psychological outcome research, applying this model is +\textbf{not undisputed}. The random-effects-model pays \textbf{more +attention to small studies} when pooling the overall effect in a +meta-analysis {[}@schwarzer2015meta{]}. Yet, small studies in particular +are often fraught with \textbf{bias} (see +\protect\hyperlink{smallstudyeffects}{Chapter 8.1}). This is why some +have argued that the fixed-effects-model should be nearly always +preferred {[}@poole1999; @furukawa2003{]}. +\end{rmdachtung} + +\subsection{\texorpdfstring{Estimators for \emph{tau\textsuperscript{2}} +in the +random-effects-model}{Estimators for tau2 in the random-effects-model}}\label{tau2} + +Operationally, conducting a random-effects-model meta-analysis in R is +not so different from conducting a fixed-effects-model meta-analyis. +Yet, we do have choose an estimator for \(\tau^{2}\). Here are the +estimators implemented in \texttt{meta}, which we can choose using the +\texttt{method.tau} variable in our meta-analysis code. + +\begin{tabular}{l|l} +\hline +Code & Estimator\\ +\hline +DL & DerSimonian-Laird\\ +\hline +PM & Paule-Mandel\\ +\hline +REML & Restricted Maximum-Likelihood\\ +\hline +ML & Maximum-likelihood\\ +\hline +HS & Hunter-Schmidt\\ +\hline +SJ & Sidik-Jonkman\\ +\hline +HE & Hedges\\ +\hline +EB & Empirical Bayes\\ +\hline +\end{tabular} + +\begin{rmdinfo} +\textbf{Which estimator should i use?} + +All of these estimators derive \(\tau^{2}\) using a slightly different +approach, leading to somewhat different pooled effect size estimates and +confidence intervals. If one of these approaches is more or less biased +often depends on the context, and parameters such as the number of +studies \(k\), the number of participants \(n\) in each study, how much +\(n\) varies from study to study, and how big \(\tau^{2}\) is. + +An overview paper by Veroniki and colleagues {[}@veroniki2016methods{]} +provides an excellent summary on current evidence which estimator might +be more or less biased in which situation. The article is openly +accessible, and you can read it +\href{https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4950030/}{here}. + +Especially in medical and psychological research, the by far most often +used estimator is the \textbf{DerSimonian-Laird estimator} +{[}@dersimonian1986meta{]}. Part of this widespread use might be +attributable to the fact that programs such as \emph{RevMan} or +\emph{Comprehensive Meta-Analysis} (older versions) only use this +estimator. It is also the default option in our \texttt{meta} package in +R. Simulation studies, however, have shown that the +\textbf{Maximum-Likelihood}, \textbf{Sidik-Jonkman}, and +\textbf{Empirical Bayes} estimators have better properties in estimating +the between-study variance +{[}@sidik2007comparison;@viechtbauer2005bias{]}. +\end{rmdinfo} + +\begin{rmdinfo} +\textbf{The Hartung-Knapp-Sidik-Jonkman method} + +Another criticism of the \textbf{DerSimonian-Laird} method is that when +estimating the variance of our pooled effect \(var(\hat\theta_F)\), this +method is very prone to producing false positives +{[}@inthout2014hartung{]}. This is especially the case when the +\textbf{number of studies} is small, and when there is substantial +\textbf{heterogeneity} +{[}@hartung1999alternative;@hartung2001refined;@hartung2001tests;@follmann1999valid;@makambi2004effect{]}. +Unfortunately, this is very often the case in when we do meta-analysis +in the medical field or in psychology. This is quite a problem, as we +don't want to find pooled effects to be statistically significant when +in fact they are not! + +The \textbf{Hartung-Knapp-Sidik-Jonkman (HKSJ) method} was thus proposed +a way to produce more robust estimates of \(var(\hat\theta_F)\). It has +been shown that this method substantially outperforms the +DerSimonian-Laird method in many cases {[}@inthout2014hartung{]}. The +HKSJ method can also be very easily applied in R, while other programs +don't have this option yet. This is another big plus of doing +meta-analysis in R. The HKSJ usually leads to more \textbf{conservative} +results, indicated by wider confidence intervals. +\end{rmdinfo} + +\begin{rmdachtung} +\textbf{Residual concerns with the Hartung-Knapp-Sidik-Jonkman method} + +It should be noted, however, that the HKSJ method is not +uncontroversial. Some authors argue that other (standard) pooling models +should also be used \textbf{in addition} to the HKSJ as a +\textbf{sensitivity analysis} {[}@wiksten2016hartung{]}. Jackson and +colleagues {[}@jackson2017hartung{]} present four residual concerns with +this method, which you may take into account before selecting your +meta-analytic method. The paper can be read +\href{https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.7411}{here}. +\end{rmdachtung} + +\hypertarget{random.precalc}{\subsection{Pre-calculated effect size +data}\label{random.precalc}} + +After all this input, you'll see that even random-effects-model +meta-analyses are very easy to code in R. Compared to the +fixed-effects-model \protect\hyperlink{fixed}{Chapter 4.1}, there's just +three extra parameters we have to define. Especially, as we've described +before, we have to tell R which \textbf{between-study-variance +estimator} (\(\tau^{2}\)) we want to use, and if we want to use the +\textbf{Knapp-Hartung-Sidik-Jonkman} adjustment. + +\textbf{Here's a table of all parameters we have to define in our code +to perform a random-effects-model meta-analysis with pre-calculated +effect sizes} + +\begin{tabular}{l|l} +\hline +Parameter & Function\\ +\hline +TE & This tells R to use the TE column to retrieve the effect sizes for each study\\ +\hline +seTE & This tells R to use the seTE column to retrieve the standard error for each study\\ +\hline +data= & After =, paste the name of your dataset here\\ +\hline +studlab=paste() & This tells the function were the labels for each study are stored. If you named the spreadsheet columns as advised, this should be studlab=paste(Author)\\ +\hline +comb.fixed= & Weather to use a fixed-effects-model\\ +\hline +comb.random= & Weather to use a random-effects-model. This has to be set to TRUE\\ +\hline +method.tau= & Which estimator to use for the between-study variance\\ +\hline +hakn= & Weather to use the Knapp-Hartung-Sidik-Jonkman method\\ +\hline +prediction= & Weather to print a prediction interval for the effect of future studies based on present evidence\\ +\hline +sm= & The summary measure we want to calculate. We can either calculate the mean difference (MD) or Hedges' g (SMD)\\ +\hline +\end{tabular} + +I will use my \texttt{madata} dataset again to do the meta-analysis. For +illustrative purposes, let's use the Sidik-Jonkman estimator (``SJ'') +and the HKSJ method. To do this analysis, make sure that \texttt{meta} +as well as \texttt{metafor} are loaded in R. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(meta)} +\KeywordTok{library}\NormalTok{(metafor)} +\end{Highlighting} +\end{Shaded} + +Now, let's code our random-effects-model meta-analysis. Remember, as our +effect size data are precalculated, i'll use the +\texttt{meta::metagen()} function. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj<-}\KeywordTok{metagen}\NormalTok{(TE,} +\NormalTok{ seTE,} + \DataTypeTok{data=}\NormalTok{madata,} + \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),} + \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{,} + \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)} +\NormalTok{m.hksj} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(random) +## Call et al. 0.7091 [ 0.1979; 1.2203] 5.2 +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 6.1 +## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 4.2 +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 7.1 +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 6.8 +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 6.1 +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 5.7 +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 5.9 +## Hintz et al. 0.2840 [-0.0453; 0.6133] 6.5 +## Kang et al. 1.2751 [ 0.6142; 1.9360] 4.3 +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 6.1 +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 5.6 +## Phang et al. 0.5407 [ 0.0619; 1.0196] 5.4 +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 5.3 +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 4.1 +## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 4.5 +## SongLindquist 0.6126 [ 0.1683; 1.0569] 5.7 +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 5.4 +## +## Number of studies combined: k = 18 +## +## SMD 95%-CI t p-value +## Random effects model 0.5935 [ 0.3891; 0.7979] 6.13 < 0.0001 +## Prediction interval [-0.2084; 1.3954] +## +## Quantifying heterogeneity: +## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 45.50 17 0.0002 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +\end{verbatim} + +The output shows that our estimated effect is \(g=0.5935\), and the 95\% +confidence interval stretches from \(g=0.39\) to \(0.80\) (rounded). + +It also becomes clear that this effect is different (and larger) than +the one we found in the fixed-effects-model meta-analysis in +\protect\hyperlink{fixed}{Chapter 4.1} (\(g=0.48\)). + +Let's compare this to the output using the \textbf{DerSimonian-Laird} +estimator, and when setting \texttt{hakn=FALSE}. As this estimator is +the \textbf{default}, i don't have to define \texttt{method.tau} this +time. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.dl<-}\KeywordTok{metagen}\NormalTok{(TE,} +\NormalTok{ seTE,} + \DataTypeTok{data=}\NormalTok{madata,} + \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),} + \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{hakn =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)} +\NormalTok{m.dl} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(random) +## Call et al. 0.7091 [ 0.1979; 1.2203] 5.0 +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 6.3 +## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 3.7 +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 8.0 +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 7.4 +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 6.3 +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 5.7 +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 6.0 +## Hintz et al. 0.2840 [-0.0453; 0.6133] 6.9 +## Kang et al. 1.2751 [ 0.6142; 1.9360] 3.8 +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 6.3 +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 5.6 +## Phang et al. 0.5407 [ 0.0619; 1.0196] 5.3 +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 5.1 +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 3.6 +## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 4.1 +## SongLindquist 0.6126 [ 0.1683; 1.0569] 5.7 +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 5.2 +## +## Number of studies combined: k = 18 +## +## SMD 95%-CI z p-value +## Random effects model 0.5741 [ 0.4082; 0.7399] 6.78 < 0.0001 +## Prediction interval [-0.0344; 1.1826] +## +## Quantifying heterogeneity: +## tau^2 = 0.0752; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 45.50 17 0.0002 +## +## Details on meta-analytical method: +## - Inverse variance method +## - DerSimonian-Laird estimator for tau^2 +\end{verbatim} + +We see that the overall effect size estimate using this estimator is +similar to the previous one (\(g=0.57\)), but the confidence intervals +\textbf{is narrower because we did not adjust them} using the HKSJ +method. + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-66-1.pdf} + +\hypertarget{random.raw}{\subsection{Raw effect size +data}\label{random.raw}} + +I we use raw effect size data, such as the one stored in my +\texttt{metacont} dataset, we can use the \texttt{meta::metacont} +function again. The parameters stay the same as before. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj.raw<-}\KeywordTok{metacont}\NormalTok{(Ne,} +\NormalTok{ Me,} +\NormalTok{ Se,} +\NormalTok{ Nc,} +\NormalTok{ Mc,} +\NormalTok{ Sc,} + \DataTypeTok{data=}\NormalTok{metacont,} + \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),} + \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{,} + \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{)} +\NormalTok{m.hksj.raw} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(random) +## Cavanagh -0.4118 [-0.8081; -0.0155] 15.7 +## Day -0.2687 [-0.6154; 0.0781] 17.7 +## Frazier -0.7734 [-1.0725; -0.4743] 19.7 +## Gaffney -0.7303 [-1.2542; -0.2065] 11.7 +## Greer -0.7624 [-1.0992; -0.4256] 18.1 +## Harrer -0.1669 [-0.5254; 0.1916] 17.2 +## +## Number of studies combined: k = 6 +## +## SMD 95%-CI t p-value +## Random effects model -0.5161 [-0.8043; -0.2280] -4.60 0.0058 +## Prediction interval [-1.1947; 0.1625] +## +## Quantifying heterogeneity: +## tau^2 = 0.0472; H = 1.51 [1.00; 2.38]; I^2 = 56.1% [0.0%; 82.3%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 11.39 5 0.0441 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +## - Hedges' g (bias corrected standardised mean difference) +\end{verbatim} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{binary}{\section{Meta-Analysis with binary +outcomes}\label{binary}} + +\subsection{Event rate data}\label{event-rate-data} + +In some cases, you will work with \textbf{binary outcome data} (e.g., +dead/alive, Depressive Disorder/no Depressive Disorder) instead of +continuous data. In such a case, you will probably be more interested in +outcomes like the pooled \textbf{Odd's Ratio} or the \textbf{Relative +Risk Reduction}. + +Here, have two options again: + +\begin{itemize} +\tightlist +\item + \textbf{The effect sizes are already calculated}. In this case, we can + use the \texttt{metagen} function as we did before (see + \protect\hyperlink{fixed}{Chapter 4.1} and + \protect\hyperlink{random}{Chapter 4.2}). The calculated effect + \texttt{TE} then describes the Odds Ratio, or whatever binary outcome + we calculated previously for our data. +\item + \textbf{We only have the raw outcome data}. If this is the case, we + will have to use the \texttt{meta::metabin} function instead. We'll + show you how to do this now. +\end{itemize} + +\textbf{For meta-analyses of binary outcomes, we need our data in the +following format:} + +\begin{tabular}{l|l} +\hline +Column & Description\\ +\hline +Author & This signifies the column for the study label (i.e., the first author)\\ +\hline +Ee & Number of events in the experimental treatment arm\\ +\hline +Ne & Number of participants in the experimental treatment arm\\ +\hline +Ec & Number of events in the control arm\\ +\hline +Nc & Number of participants in the control arm\\ +\hline +Subgroup & This is the label for one of your subgroup codes. It's not that important how you name it, so you can give it a more informative name (e.g. population). In this column, each study should then be given a subgroup code, which should be exactly the same for each subgroup, including upper/lowercase letters. Of course, you can also include more than one subgroup column with different subgroup codings, but the column name has to be unique\\ +\hline +\end{tabular} + +I'll use my dataset \texttt{binarydata}, which also has this format + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{load}\NormalTok{(}\StringTok{"binarydata.RData"}\NormalTok{)} +\KeywordTok{str}\NormalTok{(binarydata)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Classes 'tbl_df', 'tbl' and 'data.frame': 11 obs. of 5 variables: +## $ Author: chr "Alcorta-Fleischmann" "Craemer" "Eriksson" "Jones" ... +## $ Ee : num 2 18 6 3 0 8 12 1 7 17 ... +## $ Ne : num 279 1273 1858 297 300 ... +## $ Ec : num 1 17 5 6 1 9 12 10 8 21 ... +## $ Nc : num 70 1287 1852 314 295 ... +\end{verbatim} + +The other parameters are like the ones we used in the meta-analyses with +continuous outcome data, with two exceptions: + +\begin{itemize} +\tightlist +\item + \textbf{sm}: As we want to have a pooled effect for binary data, we + have to choose another summary measure now. We can choose from + \textbf{``OR''} (Odds Ratio), \textbf{``RR''} (Risk Ratio), or + \textbf{RD} (Risk Difference), among other things. +\item + \textbf{incr}. This lets us define if and how we want + \textbf{conitinuity correction} to be performed. Such a correction is + necessary in cases where one of the cells in your data is zero (e.g., + because no one in the intervention arm died). This can be a frequent + phenomenon in some contexts, and \textbf{distorts our effect size + estimates}. By default, the \texttt{metabin} function adds the value + \textbf{0.5} in all cells were N is zero \citep{gart1967bias}. This + value can be changed using the \texttt{incr}-parameter (e.g., + \texttt{incr=0.1}). If your trial arms are very uneven in terms of + their total \(n\), we can also use the \textbf{treatment arm + continuity correction} \citep{j2004add}. This can be done by using + \texttt{incr="TACC"}. +\end{itemize} + +\textbf{Here's the code for a meta-analysis with raw binary data} + +I have decided to run a random-effect-model meta-analysis. I want the +summary measure to be the Risk Ratio (RR). + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.bin<-}\KeywordTok{metabin}\NormalTok{(Ee,} +\NormalTok{ Ne,} +\NormalTok{ Ec,} +\NormalTok{ Nc,} + \DataTypeTok{data=}\NormalTok{binarydata,} + \DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author),} + \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{,} + \DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{prediction=}\OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{incr=}\FloatTok{0.1}\NormalTok{,} + \DataTypeTok{sm=}\StringTok{"RR"}\NormalTok{)} +\NormalTok{m.bin} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## RR 95%-CI %W(random) +## Alcorta-Fleischmann 0.5018 [0.0462; 5.4551] 3.0 +## Craemer 1.0705 [0.5542; 2.0676] 14.6 +## Eriksson 1.1961 [0.3657; 3.9124] 8.5 +## Jones 0.5286 [0.1334; 2.0945] 7.0 +## Knauer 0.0894 [0.0001; 57.7924] 0.5 +## Kracauer 0.9076 [0.3512; 2.3453] 10.9 +## La Sala 0.9394 [0.4233; 2.0847] 12.7 +## Maheux 0.0998 [0.0128; 0.7768] 3.9 +## Schmidthauer 0.7241 [0.2674; 1.9609] 10.3 +## van der Zee 0.8434 [0.4543; 1.5656] 15.1 +## Wang 0.5519 [0.2641; 1.1534] 13.5 +## +## Number of studies combined: k = 11 +## +## RR 95%-CI t p-value +## Random effects model 0.7420 [0.5202; 1.0582] -1.87 0.0905 +## Prediction interval [0.2305; 2.3882] +## +## Quantifying heterogeneity: +## tau^2 = 0.2417; H = 1.00 [1.00; 1.36]; I^2 = 0.0% [0.0%; 45.8%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 7.34 10 0.6929 +## +## Details on meta-analytical method: +## - Mantel-Haenszel method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +## - Continuity correction of 0.1 in studies with zero cell frequencies +\end{verbatim} + +\textbf{L'Abbé Plots} + +So-called \textbf{L'Abbé plots} \citep{labbe} are a good way to +visualize data based on event rates. In a L'Abbé plot, the event rate of +a study's intervention group is plotted against the event rate in the +control group, and the \(N\) of the study is signified by the size of +the bubble in the plot. Despite the simplicity in its principles, this +plot allows us to check for three important aspects of our meta-analysis +with binary outcomes: + +\begin{itemize} +\tightlist +\item + \textbf{The overall trend of our meta-analysis}. If we are expecting a + type of intervention to have a protective effect (i.e., making an + adverse outcome such as death or depression onset less likely) the + studies should mostly lie in the bottom-right corner of the L'Abbé + plot, because the control group event rate should be higher than the + intervention group event rate. If there's no effect of the + intervention compared to the control group, the event rates are + identical and the study is shown on the diagonal of the L'Abbé plot. +\item + \textbf{Heterogeneity of effect sizes}. The plot also allows us to + eyeball for single studies or groups of studies which contribute to + the heterogeneity of the effect we found. It could be the case, for + example, that most studies lie in the bottom-right part of the plot as + they report positive effects, while a few studies lie in the top-left + sector indicated negative effects. Especially if such studies have a + small precision (i.e., a small \(N\) of participants, indicated by + small bubbles in the plot), they could have distorted our pooled + effect and may contribute to the between-study heterogeneity. +\item + \textbf{Heterogeneity of event rates}. It may also be the case that + some of the heterogeneity in our meta-analysis was introduced by the + fact that the event rates ``per se'' are higher or lower in some + studies compared to the others. The L'Abbé plot provides as with this + information, as studies with higher event rates will naturally tend + towards the top-right corner of the plot. +\end{itemize} + +The results of the \texttt{metabin} function can be easily used to +generate L'Abbé plots using the \texttt{labbe.metabin} function included +in the \texttt{meta} package. We can specify the following parameters: + +\begin{tabular}{l|l} +\hline +Parameter & Description\\ +\hline +x & This signifies our metabin meta-analysis output\\ +\hline +bg & The background color of the studies\\ +\hline +col & The line color of the studies\\ +\hline +studlab & Wether the names of the studies should be printed in the plot (TRUE/FALSE)\\ +\hline +col.fixed & The color of the dashed line symbolizing the pooled effect of the meta-analysis, if the fixed-effect-model was used\\ +\hline +col.random & The color of the dashed line symbolizing the pooled effect of the meta-analysis, if the random-effects-model was used\\ +\hline +\end{tabular} + +For this example, i'll use the \texttt{m.bin} output i previously +generated using the \texttt{metabin} function. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{labbe.metabin}\NormalTok{(}\DataTypeTok{x =}\NormalTok{ m.bin,} + \DataTypeTok{bg =} \StringTok{"blue"}\NormalTok{,} + \DataTypeTok{studlab =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{col.random =} \StringTok{"red"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-73-1.pdf} + +Works like a charm! We see that the \textbf{dashed red line} signifying +the \textbf{pooled effect estimate} of my meta-analysis is running +trough the bottom-right sector of my L'Abbé plot, meaning that the +overall effect size is positive (i.e., that the intervention has a +preventive effect). + +However, it also becomes clear that \textbf{all studies} clearly follow +this trend: we see that most studies lie tightly in the bottom-left +corner of the plot, meaning that these studies had \textbf{small event +rates} (i.e., the event in these studies was very rare irrespective of +group assignment). We also see that two of our included studies don't +fall into this pattern: \textbf{Schmidthauer} and \textbf{van der Zee}. +Those two studies have higher event rates, and both favor the +intervention group more clearly than the others did. + +\subsection{Incidence rates}\label{incidence-rates} + +The \protect\hyperlink{binary}{previous chapter} primarily dealt with +raw event data. Such data usually does not contain any information on +the \textbf{time span} during which events did or did not occur. Given +that studies often have drastically different follow-up times (e.g., 8 +weeks vs.~2 years), it often makes sense to also take the time interval +during which events occured into account. In clinical epidemiology, +\textbf{incidence rates} are often used to signify how many events +occured within a \textbf{standardized timeframe} (e.g., one year). The +corresponding effect size is the \textbf{incidence rate ratio} (IRR), +which compares the incidence rate in the intervention group to the one +in the control group. + +To conduct meta-analyses using incidence rate data, so-called +\textbf{person-time} data has to be collected or calculated by hand. +What it basically needed to calculate person-time data is the +\textbf{number of events} and the \textbf{timeframe} during which they +occurred. You can find a general introduction into this topic +\href{https://sph.unc.edu/files/2015/07/nciph_ERIC4.pdf}{here}, and this +\href{https://www.cdc.gov/ophss/csels/dsepd/ss1978/lesson3/section2.html}{quick +course} by the Centers for Disease Control and Prevention gives a +hands-on introduction on how person-time data is calculated. + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{Forest Plots}\label{forest-plots} + +\begin{figure} +\centering +\includegraphics{forest.jpg} +\caption{} +\end{figure} + +\begin{rmdinfo} +Now that we created the \textbf{output of our meta-analysis} using the +\texttt{metagen}, \texttt{metacont} or \texttt{metabin} functions in +\texttt{meta} (see \protect\hyperlink{fixed}{Chapter +4.1},\protect\hyperlink{random}{Chapter 4.2} and +\protect\hyperlink{binary}{Chapter 4.3}), it is time to present the data +in a more digestable way. + +\textbf{Forest Plots} are an easy way to do this, and it is conventional +to report forest plots in meta-analysis publications. +\end{rmdinfo} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Generating a Forest Plot}\label{generating-a-forest-plot} + +To produce a forest plot, we use the meta-analysis output we just +created (e.g., \texttt{m}, \texttt{m.raw}) und the +\texttt{meta::forest()} function. I'll use my \texttt{m.hksj.raw} output +from \protect\hyperlink{random.raw}{Chapter 4.2.3} to create the forest +plot + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{forest}\NormalTok{(m.hksj.raw)} +\end{Highlighting} +\end{Shaded} + +\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-76-1} \end{center} + +Looks good so far. We see that the function plotted a forest plot with a +\textbf{diamond} (i.e.~the overall effect and its confidence interval) +and a \textbf{prediction interval}. + +There are plenty of \textbf{other parameters} within the +\texttt{meta::forest} function which we can use to modify the forest +plot. + +\begin{tabular}{l|l|l} +\hline +Type & Parameter & Description\\ +\hline +General & sortvar & A sorting variable. For example, you can sort the forest plot by effect size using 'TE', or by author name using 'Author'\\ +\hline +General & studlab & This tells the function which variable should be printed as the study label. The standard is 'Author'\\ +\hline +General & comb.fixed & Whether fixed effect estimate should be plotted. (TRUE/FALSE)\\ +\hline +General & comb.random & Whether random effects estimate should be plotted. (TRUE/FALSE)\\ +\hline +General & overall & Whether overall summaries should be plotted. This argument is useful in a meta-analysis with subgroups if summaries should only be plotted on group level.\\ +\hline +General & text.fixed & A character string used in the plot to label the pooled fixed effect estimate. Has to be put in "" (e.g. "Overall effect")\\ +\hline +General & text.random & A character string used in the plot to label the pooled random effect estimate. Has to be put in "" (e.g. "Overall effect")\\ +\hline +General & col.fixed & Line colour (pooled fixed effect estimate). E.g. "red", "blue", or hex color code ("\#2e8aff")\\ +\hline +General & col.random & Line colour (random fixed effect estimate). E.g. "red", "blue", or hex color code ("\#2e8aff")\\ +\hline +General & prediction & Whether a prediction interval should be printed.\\ +\hline +General & text.predict & A character string used in the plot to label the prediction interval. E.g. "Prediction Interval"\\ +\hline +General & subgroup & A logical indicating whether subgroup results should be shown in forest plot. This argument is useful in a meta-analysis with subgroups if summaries should not be plotted on group level. (TRUE/FALSE)\\ +\hline +General & print.subgroup.labels & A logical indicating whether subgroup label should be printed. (TRUE/FALSE)\\ +\hline +General & study.results & Whether results for individual studies should be shown in the figure (useful to only plot subgroup results). (TRUE/FALSE)\\ +\hline +General & xlab & A label for the x-axis on the bottom. Put in "".\\ +\hline +General & smlab & A label for the summary measure on top. Put in "".\\ +\hline +General & xlim & The x limits (min,max) of the plot, or the character "s" to produce symmetric forest plots. This is particularly revelant when your results deviate substantially from zero, or if you also want to have outliers depicted. (e.g. xlim=c(0,1.5) for effects from 0 to 1.5).\\ +\hline +General & ref & The reference value to be plotted as a line in the forest plot. This is interesting if you want to compare effects to common thresholds or results from previous analyses (e.g. ref=0.5)\\ +\hline +General & leftcols & Here you can specify all variables which should be printed on the left side of your plot. The variables have to be part of you meta-analysis output, so check with name\_of\_your\_output\$ which variables can be displayed. E.g. leftcols=c("TE","seTE").\\ +\hline +General & rightcols & Same as leftcols, but for the right side of your plot\\ +\hline +General & leftlabs & This specifies how the columns on the left side should be named. Always provide all labels (e.g. leftlabs=c("Author","Effect size","Standard error"))\\ +\hline +General & rightlabs & Same as leftlabs, but for the left side of you plot\\ +\hline +General & print.I2 & Whether to print the value of the I-squared statistic.\\ +\hline +General & print.I2.ci & Whether to print the confidence interval of the I-squared statistic.\\ +\hline +General & squaresize & A numeric used to increase or decrease the size of squares in the forest plot. (E.g., squaresize = 1.2)\\ +\hline +Color & col.study & The colour for individual study results and confidence limits. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.inside & The colour for individual study results and confidence limits if confidence limits are completely within squares. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.square & The colour for squares reflecting study's weight in the meta-analysis. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.square.lines & The colour for the outer lines of squares reflecting study's weight in the meta-analysis. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.diamond & The colour of diamonds representing the results for fixed effect and random effects models. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.diamond.fixed & The colour of diamonds for fixed effect estimates. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.diamond.random & The colour of diamonds for random effects estimates. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.diamond.lines & The colour of the outer lines of diamonds representing the results for fixed effect and random effects models. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.diamond.lines.fixed & The colour of the outer lines of diamond for fixed effect estimate. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.diamond.lines.random & The colour of the outer lines of diamond for random effects estimate. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.inside.fixed & The colour for result of fixed effect meta-analysis if confidence limit lies completely within square. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.inside.random & The colour for result of random effects meta-analysis if confidence limit lies completely within square. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.predict & Background colour of prediction interval. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.predict.lines & Colour of outer lines of prediction interval. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.label.right & The colour for label on right side of null effect. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Color & col.label.left & The colour for label on left side of null effect. E.g. 'red', 'blue', or hex color code ('\#2e8aff')\\ +\hline +Digits & digits & Minimal number of significant digits for treatment effects (TE)\\ +\hline +Digits & digits.se & Minimal number of significant digits for standard errors\\ +\hline +Digits & digits.zval & Minimal number of significant digits for z- or t-statistic for test of overall effect\\ +\hline +Digits & digits.tau2 & Minimal number of significant digits for between-study variance\\ +\hline +Digits & digits.pval & Minimal number of significant digits for p-value of overall treatment effect\\ +\hline +Digits & digits.pval.Q & Minimal number of significant digits for p-value of heterogeneity test\\ +\hline +Digits & digits.Q & Minimal number of significant digits for heterogeneity statistic Q\\ +\hline +Digits & digits.I2 & Minimal number of significant digits for I-squared statistic\\ +\hline +Digits & digits.weight & Minimal number of significant digits for weights\\ +\hline +Digits & digits.mean & Minimal number of significant digits for the mean\\ +\hline +Digits & digits.sd & Minimal number of significant digits for the standard deviations\\ +\hline +\end{tabular} + +This is again just an overview. For all settings, type +\texttt{?meta::forest} in your \textbf{console} to see more. + +Let's play around with the function a little now: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{forest}\NormalTok{(m.hksj.raw,} + \DataTypeTok{sortvar=}\NormalTok{TE,} + \DataTypeTok{xlim =} \KeywordTok{c}\NormalTok{(}\OperatorTok{-}\FloatTok{1.5}\NormalTok{,}\FloatTok{0.5}\NormalTok{),} + \DataTypeTok{rightlabs =} \KeywordTok{c}\NormalTok{(}\StringTok{"g"}\NormalTok{,}\StringTok{"95% CI"}\NormalTok{,}\StringTok{"weight"}\NormalTok{),} + \DataTypeTok{leftlabs =} \KeywordTok{c}\NormalTok{(}\StringTok{"Author"}\NormalTok{, }\StringTok{"N"}\NormalTok{,}\StringTok{"Mean"}\NormalTok{,}\StringTok{"SD"}\NormalTok{,}\StringTok{"N"}\NormalTok{,}\StringTok{"Mean"}\NormalTok{,}\StringTok{"SD"}\NormalTok{),} + \DataTypeTok{lab.e =} \StringTok{"Intervention"}\NormalTok{,} + \DataTypeTok{pooled.totals =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{smlab =} \StringTok{""}\NormalTok{,} + \DataTypeTok{text.random =} \StringTok{"Overall effect"}\NormalTok{,} + \DataTypeTok{print.tau2 =} \OtherTok{FALSE}\NormalTok{,} + \DataTypeTok{col.diamond =} \StringTok{"blue"}\NormalTok{,} + \DataTypeTok{col.diamond.lines =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{print.I2.ci =} \OtherTok{TRUE}\NormalTok{,} + \DataTypeTok{digits.sd =} \DecValTok{2} +\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-78-1} \end{center} + +Looks good so far! For special \textbf{layout types}, proceed to +\protect\hyperlink{layouttypes}{Chapter 5.2} now. + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{layouttypes}{\section{Layout types}\label{layouttypes}} + +The \texttt{meta::forest} function also has two \textbf{Layouts} +preinstalled which we can use. Those layouts can be accessed with the +\texttt{layout=} parameter. + +\begin{itemize} +\tightlist +\item + \textbf{``RevMan5''}. This layout is used for Cochrane reviews and + generated by \emph{Review Manager 5}. +\item + \textbf{``JAMA''}. This layout gives you a forest plot according to + the guidelines of the \emph{Journal of the American Medical + Association} as output (see details + \href{https://jamanetwork.com/journals/jama/pages/instructions-for-authors}{here}). +\end{itemize} + +The \textbf{RevMan} layout looks like this: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{forest}\NormalTok{(m.hksj.raw,} + \DataTypeTok{layout =} \StringTok{"RevMan5"}\NormalTok{,} + \DataTypeTok{digits.sd =} \DecValTok{2}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-79-1} \end{center} + +The \textbf{JAMA} layout looks like this: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{forest}\NormalTok{(m.hksj.raw,} + \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,} + \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,} + \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-80-1} \end{center} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Saving the forest plots}\label{saving-the-forest-plots} + +Let's say i want to save the JAMA version of my Forest Plot now. To do +this, i have to reuse the code with which i plotted my forest plot, and +put it between +\texttt{pdf(file=\textquotesingle{}name\_of\_the\_pdf\_i\_want\_to\_create.pdf\textquotesingle{})} +and \texttt{dev.off}, both in separate lines. This saves the plot into a +PDF in my Working Directory. + +This way, i can export the plot in different formats (you can find more +details on the saving options \protect\hyperlink{saving}{here}). + +\textbf{PDF} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{pdf}\NormalTok{(}\DataTypeTok{file=}\StringTok{'forestplot.pdf'}\NormalTok{) } +\NormalTok{forest.jama<-}\KeywordTok{forest}\NormalTok{(m.hksj.raw,} + \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,} + \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,} + \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))} +\KeywordTok{dev.off}\NormalTok{() } +\end{Highlighting} +\end{Shaded} + +\textbf{PNG} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{png}\NormalTok{(}\DataTypeTok{file=}\StringTok{'forestplot.png'}\NormalTok{) } +\NormalTok{forest.jama<-}\KeywordTok{forest}\NormalTok{(m.hksj.raw,} + \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,} + \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,} + \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))} +\KeywordTok{dev.off}\NormalTok{() } +\end{Highlighting} +\end{Shaded} + +\textbf{Scalable Vector Graphic} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{svg}\NormalTok{(}\DataTypeTok{file=}\StringTok{'forestplot.svg'}\NormalTok{) } +\NormalTok{forest.jama<-}\KeywordTok{forest}\NormalTok{(m.hksj.raw,} + \DataTypeTok{layout =} \StringTok{"JAMA"}\NormalTok{,} + \DataTypeTok{text.predict =} \StringTok{"95% PI"}\NormalTok{,} + \DataTypeTok{col.predict =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{colgap.forest.left =} \KeywordTok{unit}\NormalTok{(}\DecValTok{15}\NormalTok{,}\StringTok{"mm"}\NormalTok{))} +\KeywordTok{dev.off}\NormalTok{() } +\end{Highlighting} +\end{Shaded} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{heterogeneity}{\chapter{Between-study +Heterogeneity}\label{heterogeneity}} + +\begin{figure} +\centering +\includegraphics{schiffchen.jpg} +\caption{} +\end{figure} + +By now, we have already shown you how to pool effect sizes in a +meta-analysis. In meta-analytic pooling, we aim to \textbf{synthesize +the effects of many different studies into one single effect}. However, +this makes only sense if we aren't comparing \textbf{Apples and +Oranges}. For example, it could be the case that while the overall +effect we calculate in the meta-analysis is \textbf{small}, there are +still a few studies which report \textbf{very high} effect sizes. Such +information is lost in the aggregate effect, but it is very important to +know if all studies, or interventions, yield small effect sizes, or if +there are exceptions. + +It could also be the case that even some very \textbf{extreme effect +sizes} were included in the meta-analysis, so-called \textbf{outliers}. +Such outliers might have even distorted our overall effect, and it is +important to know how our overall effect would have looked without them. + +The extent to which effect sizes vary within a meta-analysis is called +\textbf{heterogeneity}. It is very important to assess heterogeneity in +meta-analyses, as high heterogeneity could be caused by the fact that +there are actually two or more \textbf{subgroups} of studies present in +the data, which have a different true effect. Such information could be +very valuable for \textbf{research}, because this might allow us to find +certain interventions or populations for which effects are lower or +higher. + +From a statistical standpoint, high heterogeneity is also problematic. +Very high heterogeneity could mean that the studies have nothing in +common, and that there is no \textbf{``real'' true effect behind our +data}, meaning that it makes no sense to report the pooled effect at all +\citep{borenstein2011}. + +\begin{rmdinfo} +\textbf{The idea behind heterogeneity} + +Rücker and colleagues {[}@rucker2008undue{]} name three types of +heterogeneity in meta-analyses: + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + \emph{Clinical baseline heterogeneity}. These are differences between + sample characteristics between the studies. For example, while one + study might have included rather old people into their study, another + might have recruited study participants who were mostly quite young. +\item + \emph{Statistical heterogeneity}. This is the statistical + heterogeneity we find in our collected effect size data. Such + heterogeneity migh be either important from a clinical standpoint + (e.g., when we don't know if a treatment is very or only marginally + effective because the effects vary much from study to study), or from + statistical standpoint (because it dilutes the confidence we have in + our pooled effect) +\item + \emph{Other sources of heterogeneity}, such as design-related + heterogeneity. +\end{enumerate} + +Point 1. and 3. may be controlled for to some extent by restricting the +scope of our search for studies to certain well-defined intervention +types, populations, and outcomes. + +Point 2., on the other hand, has to be assessed once we conducted the +pooling of studies. This is what this chapter focuses on. +\end{rmdinfo} + +\begin{rmdinfo} +\textbf{Heterogeneity Measures} + +There are \textbf{three types of heterogeneity measures} which are +commonly used to assess the degree of heterogeneity. In the following +examples, \(k\) denotes the individual study, \(K\) denotes all studies +in our meta-analysis, \(\hat \theta_k\) is the estimated effect of \(k\) +with a variance of \(\hat \sigma^{2}_k\), and \(w_k\) is the individual +\textbf{weight} of the study (i.e., its \emph{inverse variance}: +\(w_k = \frac{1}{\hat \sigma^{2}_k}\); see infobox in +\protect\hyperlink{fixed}{Chapter 4.1.1} for more details). + +\textbf{1. Cochran's \emph{Q} } + +Cochran's \emph{Q}-statistic is the \textbf{difference between the +observed effect sizes and the fixed-effect model estimate} of the effect +size, which is then \textbf{squared, weighted and summed}. + +\[ Q = \sum\limits_{k=1}^K w_k (\hat\theta_k - \frac{\sum\limits_{k=1}^K w_k \hat\theta_k}{\sum\limits_{k=1}^K w_k})^{2}\] + +\textbf{2. Higgin's \& Thompson's \emph{I}\textsuperscript{2} } + +\(I^{2}\) {[}@higgins2002quantifying{]} is the \textbf{percentage of +variability} in the effect sizes which is not caused by sampling error. +It is derived from \(Q\): + +\[I^{2} = max \left\{0, \frac{Q-(K-1)}{Q} \right\}\] + +\textbf{3. Tau-squared} + +\(\tau^{2}\) is the between-study variance in our meta-analysis. As we +show in \protect\hyperlink{tau2}{Chapter 4.2.1}, there are various +proposed ways to calculate \(\tau^{2}\) +\end{rmdinfo} + +\begin{rmdachtung} +\textbf{Which measure should i use?} + +Generally, when we assess and report heterogeneity in a meta-analysis, +we need a measure which is \textbf{robust, and not to easily influenced +by statistical power}. + +\textbf{Cochran's \emph{Q} } increases both when the \textbf{number of +studies} (\(k\)) increases, and when the \textbf{precision} (i.e., the +sample size \(N\) of a study) increases. Therefore, \(Q\) and weather it +is \textbf{significant} highly depends on the size of your +meta-analysis, and thus its statistical power. We should therefore not +only rely on \(Q\) when assessing heterogeneity. + +\textbf{I\textsuperscript{2}} on the other hand, is not sensitive to +changes in the number of studies in the analyses. \(I^2\) is therefore +used extensively in medical and psychological research, especially since +there is a \textbf{``rule of thumb''} to interpret it +{[}@higgins2003measuring{]}: + +\begin{itemize} +\tightlist +\item + \emph{I}\textsuperscript{2} = 25\%: \textbf{low heterogeneity} +\item + \emph{I}\textsuperscript{2} = 50\%: \textbf{moderate heterogeneity} +\item + \emph{I}\textsuperscript{2} = 75\%: \textbf{substantial heterogeneity} +\end{itemize} + +Despite its common use in the literature, \(I^2\) not always an adequate +measure for heterogeneity either, because it still heavily depends on +the \textbf{precision} of the included studies {[}@rucker2008undue; +@borenstein2017basics{]}. As said before, \(I^{2}\) is simply the amount +of variability \textbf{not caused by sampling error}. If our studies +become increasingly large, this sampling error tends to \textbf{zero}, +while at the same time, \(I^{2}\) tends to 100\% simply because the +single studies have greater \(N\). Only relying on \(I^2\) is therefore +not a good option either. + +\textbf{Tau-squared}, on the other hand, is \textbf{insensitive} to the +number of studies, \textbf{and} the precision. Yet, it is often hard to +interpret how relevant our tau-squared is from a practical standpoint. + +\textbf{Prediction intervals} (like the ones we automatically calculated +in \protect\hyperlink{pool}{Chapter 4}) are a good way to overcome this +limitation {[}@inthout2016plea{]}, as they take our between-study +variance into account. Prediction intervals give us a range for which we +can \textbf{expect the effects of future studies to fall} based on +\textbf{our present evidence in the meta-analysis}. If our prediction +interval, for example, lies completely on the positive side favoring the +intervention, we can be quite confident to say that \textbf{despite +varying effects, the intervention might be at least in some way +beneficial in all contexts we studied in the future}. If the confidence +interval includes \textbf{zero}, we can be less sure about this, +although it should be noted that \textbf{broad prediction intervals are +quite common, especially in medicine and psychology}. +\end{rmdachtung} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Assessing the heterogeneity of your pooled effect +size}\label{assessing-the-heterogeneity-of-your-pooled-effect-size} + +Thankfully, once you've already pooled your effects in meta-analysis +using the \texttt{metagen()}, \texttt{metabin()}, or \texttt{metacont} +function, it is very easy and straightforward to retrieve the +\textbf{three most common heterogeneity measures} that we described +before. + +In \protect\hyperlink{random.precalc}{Chapter 4.2.2}, we already showed +you how to conduct a \textbf{random-effect-model meta-analysis}. In this +example, we stored our \textbf{results} in the object \texttt{m.hksj}, +which we will use again here. + +One way to get heterogeneity measures of my meta-analysis is to +\textbf{print} the meta-analysis (in my case, \texttt{m.hksj}) output +again. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{print}\NormalTok{(m.hksj)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(random) +## Call et al. 0.7091 [ 0.1979; 1.2203] 5.2 +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 6.1 +## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 4.2 +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 7.1 +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 6.8 +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 6.1 +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 5.7 +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 5.9 +## Hintz et al. 0.2840 [-0.0453; 0.6133] 6.5 +## Kang et al. 1.2751 [ 0.6142; 1.9360] 4.3 +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 6.1 +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 5.6 +## Phang et al. 0.5407 [ 0.0619; 1.0196] 5.4 +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 5.3 +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 4.1 +## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 4.5 +## SongLindquist 0.6126 [ 0.1683; 1.0569] 5.7 +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 5.4 +## +## Number of studies combined: k = 18 +## +## SMD 95%-CI t p-value +## Random effects model 0.5935 [ 0.3891; 0.7979] 6.13 < 0.0001 +## Prediction interval [-0.2084; 1.3954] +## +## Quantifying heterogeneity: +## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 45.50 17 0.0002 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +\end{verbatim} + +We see that this output \textbf{already provides us with all three +heterogeneity measures} (and even one more, \emph{H}, which we will not +cover here). + +\begin{itemize} +\item + \(\tau^{2}\), as we can see from the \texttt{tau\^{}2} output, is + \textbf{0.1337}. +\item + \(I^{2}\) is printed next to \texttt{I\^{}2}, and has the value + \textbf{62.6\%}, and a 95\% confidence interval rangin from 37.9\% to + 77.5\%. +\item + The value of \(Q\) is displayed next to \texttt{Q} under + \texttt{Test\ of\ heterogeneity:}. As we can see, the value is + \textbf{45.50}. In our case, this is highly significant (\(p=0.0002\); + see \texttt{p-value}). +\item + The \textbf{prediction interval} can be found next to + \texttt{Prediction\ interval}. As we can see, the 95\% interval ranges + from \textbf{g=-0.2084} to \textbf{1.3954}. +\end{itemize} + +How can we interpret the values of this example analysis? Well, all +three of our indicators suggest that \textbf{moderate to substantial +heterogeneity is present in our data}. Given the \textbf{broad +prediction interval}, which stretches well below zero, we also cannot be +overly confident that the positive effect we found for our interventions +is robust in every context. It might be very well possible that the +intervention does not yield positive effects in some future scenarios; +even a small negative effect might be possible based on the evidence the +meta-analysis gives us. Very high effect sizes, on the other hand, are +possible too. + +\textbf{When the measures are not displayed in my output} + +Depending on how you changed the settings of the \texttt{metagen}, +\texttt{metabin}, or \texttt{metacont}, it is possible that some of the +measures are not displayed in your output. That's not a big deal, +because all measures are stored in the object, no matter if they are +immediately displayed or not. + +To directly access one of the measures, we can to use \texttt{\$} again +(see \protect\hyperlink{convertfactors}{Chapter 3.3.1}). We use this +\textbf{in combination with our meta-analysis output object} to define +which measure we want to see. + +\begin{tabular}{l|l} +\hline +Code & Measure\\ +\hline +\$Q & Cochran's Q\\ +\hline +\$pval.Q & The p-value for Cochran's Q\\ +\hline +\$I2 & I-squared\\ +\hline +\$lower.I2 & The lower bound of the I-squared 95\%CI\\ +\hline +\$upper.I2 & The upper bound of the I-squared 95\%CI\\ +\hline +\$tau\textasciicircum{}2 & Tau-squared\\ +\hline +\$lower.predict & The lower bound of the 95\% prediction interval\\ +\hline +\$upper.predict & The upper bound of the 95\% prediction interval\\ +\hline +\end{tabular} + +Here are a few exmaples for my \texttt{m.hksj} object. As you'll see, +the output is \textbf{identical} to the one before. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{Q} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 45.50257 +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{I2} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 0.6263947 +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{tau}\OperatorTok{^}\DecValTok{2} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 0.1337024 +\end{verbatim} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Detecting outliers \& influential +cases}\label{detecting-outliers-influential-cases} + +As mentioned before, \textbf{between-study heterogeneity} can also be +caused by one more studies with \textbf{extreme effect sizes} which +don't quite \textbf{fit in}. Especially when the \textbf{quality of +these studies is low}, or the \textbf{studies are very small}, this may +\textbf{distort} our pooled effect estimate, and it's a good idea to +have a look on the \textbf{pooled effect again once we remove such +outliers from the analysis}. + +On the other hand, we also want to know \textbf{if the pooled effect +estimate we found is robust}, meaning that the effect does not depend +heavily on \textbf{one single study}. Therefore, we also want to know +\textbf{whether there are studies which heavily push the effect of our +analysis into some direction}. Such studies are called +\textbf{influential cases}, and we'll devote some time to this topic in +the \protect\hyperlink{influenceanalyses}{second part} of this chapter. + +\begin{rmdachtung} +It should be noted that they are \textbf{many methods} to spot +\textbf{outliers and influential cases}, and the methods described here +are not comprehensive. If you want to read more about the underpinnings +of this topic, we can recommend the paper by Wolfgang Viechtbauer and +Mike Cheung {[}@viechtbauer2010outlier{]}. +\end{rmdachtung} + +\hypertarget{outliers}{\subsection{Searching for extreme effect sizes +(outliers)}\label{outliers}} + +A common method to detect outliers directly is to define a study as an +outlier if the \textbf{study's confidence interval does not overlap with +the confidence interval}. This means that we define a study as an +outlier when it's effect size estimate is \textbf{so extreme that we +have high certainty that the study cannot be part of the ``population'' +of effect size we determined when pooling our results} (i.e., the +individual study differs significantly from the overall effect). + +To detect such outliers in our dataset, the \texttt{filter} function in +the \texttt{dplyr} package we introduced in +\protect\hyperlink{filter}{Chapter 3.3.3} comes in handy again. + +Using this function, we can search for all studies: + +\begin{itemize} +\tightlist +\item + for which the \textbf{upper bound of the 95\% confidence interval is + lower than the lower bound of the pooled effect confidence interval} + (i.e., extremely small effects) +\item + for which the \textbf{lower bound of the 95\% confidence interval is + higher than the higher bound of the pooled effect confidence interval} + (i.e., extremely large effects) +\end{itemize} + +Here, i'll use my \texttt{m.hksj} meta-analysis output from +\protect\hyperlink{random.precalc}{Chapter 4.2.2} again. Let's see what +the \textbf{upper and lower bound of my pooled effect confidence +interval} is. As i performed a \textbf{random-effect meta-analysis in +this example}, i will use the value stored under \texttt{\$lower.random} +and \texttt{\$upper.random}. If you performed a \textbf{fixed-effect +meta-analysis}, the objects would be \texttt{\$lower.fixed} and +\texttt{\$upper.fixed}, respectively. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{lower.random} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 0.389147 +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{upper.random} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 0.7979231 +\end{verbatim} + +Here we go. I now see that my \textbf{pooled effect confidence interval} +stretches from \(g = 0.389\) to \(g = 0.798\). We can use these values +to filter out outliers now. + +To filter out outliers \textbf{automatically}, we have prepared two +\textbf{functions} for you, \texttt{spot.outliers.random} and +\texttt{spot.outliers.fixed}. Both need the \texttt{dplyr} package (see +\protect\hyperlink{select}{Chapter 3.3.3}) to function, so we need to +need to have this package \textbf{installed} and \textbf{loaded into our +library}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(dplyr)} +\end{Highlighting} +\end{Shaded} + +The function we'll use in the case of my \texttt{m.hksj} dataset is +\texttt{spot.outliers.random}, because we conducted a +\textbf{random-effect meta-analysis to get this output object}. R +doesn't know this function yet, so we have to let R learn it copying and +pasting the code underneath \textbf{in its entirety} into the +\textbf{console} on the bottom left pane of RStudio, and then hit +\textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{spot.outliers.random<-}\ControlFlowTok{function}\NormalTok{(data)\{} +\NormalTok{data<-data} +\NormalTok{Author<-data}\OperatorTok{$}\NormalTok{studlab} +\NormalTok{lowerci<-data}\OperatorTok{$}\NormalTok{lower} +\NormalTok{upperci<-data}\OperatorTok{$}\NormalTok{upper} +\NormalTok{m.outliers<-}\KeywordTok{data.frame}\NormalTok{(Author,lowerci,upperci)} +\NormalTok{te.lower<-data}\OperatorTok{$}\NormalTok{lower.random} +\NormalTok{te.upper<-data}\OperatorTok{$}\NormalTok{upper.random} +\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,upperci }\OperatorTok{<}\StringTok{ }\NormalTok{te.lower)} +\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,lowerci }\OperatorTok{>}\StringTok{ }\NormalTok{te.upper)} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +Now, the function is ready to be used. The only thing we have to tell +the \texttt{spot.outliers.random} function is the \textbf{meta-analysis +output} that we want to check for outliers, which is defined by +\texttt{data}. In my case, this is \texttt{m.hksj}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{spot.outliers.random}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj)} +\end{Highlighting} +\end{Shaded} + +This is the output we get from the function: + +\begin{verbatim} +## Author lowerci upperci +## 1 DanitzOrsillo 1.1138668 2.468473 +## 2 Shapiro et al. 0.8617853 2.097667 +\end{verbatim} + +We see that the function has detected \textbf{two outliers}. Looking at +the \texttt{lowerci} value, the lower bound of the two study's +confidence intervals, we see that both have extremely high positive +effects, because the lower bounds are both much higher than the +\textbf{higher bound of the confidence interval of our pooled effect}, +which was \(g = 0.798\). + +Thus, we can conduct a \textbf{sensitivity analysis} in which we +\textbf{exclude these two outliers}. We can do this with the +\texttt{update.meta} function in \texttt{meta}. This creates an update +of our meta-analysis output \texttt{m.hksj} without the outliers. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj.outliers<-}\KeywordTok{update.meta}\NormalTok{(m.hksj,} + \DataTypeTok{subset =}\NormalTok{ Author }\OperatorTok{!=}\StringTok{ }\KeywordTok{c}\NormalTok{(}\StringTok{"DanitzOrsillo"}\NormalTok{,} + \StringTok{"Shapiro et al."}\NormalTok{))} +\NormalTok{m.hksj.outliers} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(random) +## Call et al. 0.7091 [ 0.1979; 1.2203] 5.0 +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 6.9 +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 10.4 +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 9.1 +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 7.0 +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 6.0 +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 6.4 +## Hintz et al. 0.2840 [-0.0453; 0.6133] 8.1 +## Kang et al. 1.2751 [ 0.6142; 1.9360] 3.5 +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 7.0 +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 5.8 +## Phang et al. 0.5407 [ 0.0619; 1.0196] 5.4 +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 5.1 +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 3.3 +## SongLindquist 0.6126 [ 0.1683; 1.0569] 5.9 +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 5.3 +## +## Number of studies combined: k = 16 +## +## SMD 95%-CI t p-value +## Random effects model 0.4708 [0.3406; 0.6010] 7.71 < 0.0001 +## Prediction interval [0.0426; 0.8989] +## +## Quantifying heterogeneity: +## tau^2 = 0.0361; H = 1.15 [1.00; 1.56]; I^2 = 24.8% [0.0%; 58.7%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 19.95 15 0.1739 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +\end{verbatim} + +The entire procedure works the same if you \textbf{conducted a +fixed-effect meta-analysis}. However, you need to copy and paste the +code for the \texttt{spot.outliers.fixed} function then, which can be +found \textbf{below}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{spot.outliers.fixed<-}\ControlFlowTok{function}\NormalTok{(data)\{} +\NormalTok{data<-data} +\NormalTok{Author<-data}\OperatorTok{$}\NormalTok{studlab} +\NormalTok{lowerci<-data}\OperatorTok{$}\NormalTok{lower} +\NormalTok{upperci<-data}\OperatorTok{$}\NormalTok{upper} +\NormalTok{m.outliers<-}\KeywordTok{data.frame}\NormalTok{(Author,lowerci,upperci)} +\NormalTok{te.lower<-data}\OperatorTok{$}\NormalTok{lower.fixed} +\NormalTok{te.upper<-data}\OperatorTok{$}\NormalTok{upper.fixed} +\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,upperci }\OperatorTok{<}\StringTok{ }\NormalTok{te.lower)} +\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{filter}\NormalTok{(m.outliers,lowerci }\OperatorTok{>}\StringTok{ }\NormalTok{te.upper)} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{influenceanalyses}{\section{Influence +Analyses}\label{influenceanalyses}} + +We have now showed you how you can detect and remove \textbf{extreme +effect sizes} (outliers) in your meta-analysis. + +As we've mentioned before in \href{}{Chapter}, however, it is not only +statistical outliers which may cause concerns regarding the robustness +of our pooled effect. It is also possible that \textbf{some studies in a +meta-analysis exert a very high influence on our overall results}. For +example, it could be the case that we find that an overall effect is not +significant, when in fact, a highly significant effect is consistently +found once we remove one particular study in our analysis. Such +information is \textbf{highly important once we want to communicate the +results of our meta-analysis to the public}. + +Here, we present techniques which dig a little deeper than simple +outlier removal. To some extent, they are based on the +\textbf{Leave-One-Out}-method, in which we \textbf{recalculate the +results of our meta-analysis} \(K-1\) times, each times leaving out one +study. This way, we can more easily detect \textbf{studies which +influence the overall estimate of our meta-analysis the most}, and lets +us better assess if this \textbf{influence may distort our pooled +effect} \citep{viechtbauer2010outlier}. Thus, such analyses are called +\textbf{Influence Analyses}. + +We have created the \textbf{function} \texttt{influence.analysis} for +you through which influences can be conducted all in one. For this +function to work, you need to have the \texttt{meta} and +\texttt{metafor} packages installed and loaded in your library. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{influence.analysis<-}\ControlFlowTok{function}\NormalTok{(data,method.tau,hakn)\{} + +\NormalTok{ influence.data<-data} +\NormalTok{ TE<-data}\OperatorTok{$}\NormalTok{TE} +\NormalTok{ seTE<-data}\OperatorTok{$}\NormalTok{seTE} +\NormalTok{ method.tau<-method.tau} +\NormalTok{ hakn<-hakn} + +\ControlFlowTok{if}\NormalTok{(hakn }\OperatorTok{==}\StringTok{ }\OtherTok{TRUE}\NormalTok{)\{} +\NormalTok{ res <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{TE, }\DataTypeTok{sei=}\NormalTok{seTE, }\DataTypeTok{measure=}\StringTok{"ZCOR"}\NormalTok{, } + \DataTypeTok{data=}\NormalTok{influence.data, } + \DataTypeTok{method =} \KeywordTok{paste}\NormalTok{(method.tau),} + \DataTypeTok{test=}\StringTok{"knha"}\NormalTok{)} +\NormalTok{ res} +\NormalTok{ inf <-}\StringTok{ }\KeywordTok{influence}\NormalTok{(res)} +\NormalTok{ influence.data<-}\KeywordTok{metainf}\NormalTok{(data)} +\NormalTok{ influence.data}\OperatorTok{$}\NormalTok{I2<-}\KeywordTok{format}\NormalTok{(}\KeywordTok{round}\NormalTok{(influence.data}\OperatorTok{$}\NormalTok{I2,}\DecValTok{2}\NormalTok{),}\DataTypeTok{nsmall=}\DecValTok{2}\NormalTok{)} + \KeywordTok{plot}\NormalTok{(inf)} + \KeywordTok{baujat}\NormalTok{(data)} + \KeywordTok{forest}\NormalTok{(influence.data,} + \DataTypeTok{sortvar=}\NormalTok{I2,} + \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),} + \DataTypeTok{smlab =} \StringTok{"Sorted by I-squared"}\NormalTok{)} + \KeywordTok{forest}\NormalTok{(influence.data,} + \DataTypeTok{sortvar=}\NormalTok{TE,} + \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),} + \DataTypeTok{smlab =} \StringTok{"Sorted by Effect size"}\NormalTok{)} + +\NormalTok{\} }\ControlFlowTok{else}\NormalTok{ \{} + +\NormalTok{ res <-}\StringTok{ }\KeywordTok{rma}\NormalTok{(}\DataTypeTok{yi=}\NormalTok{TE, }\DataTypeTok{sei=}\NormalTok{seTE, }\DataTypeTok{measure=}\StringTok{"ZCOR"}\NormalTok{, } + \DataTypeTok{data=}\NormalTok{influence.data, } + \DataTypeTok{method =} \KeywordTok{paste}\NormalTok{(method.tau))} +\NormalTok{ res} +\NormalTok{ inf <-}\StringTok{ }\KeywordTok{influence}\NormalTok{(res)} +\NormalTok{ influence.data<-}\KeywordTok{metainf}\NormalTok{(data)} +\NormalTok{ influence.data}\OperatorTok{$}\NormalTok{I2<-}\KeywordTok{format}\NormalTok{(}\KeywordTok{round}\NormalTok{(influence.data}\OperatorTok{$}\NormalTok{I2,}\DecValTok{2}\NormalTok{),}\DataTypeTok{nsmall=}\DecValTok{2}\NormalTok{)} + \KeywordTok{plot}\NormalTok{(inf)} + \KeywordTok{baujat}\NormalTok{(data)} + \KeywordTok{forest}\NormalTok{(influence.data,} + \DataTypeTok{sortvar=}\NormalTok{I2,} + \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),} + \DataTypeTok{smlab =} \StringTok{"Sorted by I-squared"}\NormalTok{)} + \KeywordTok{forest}\NormalTok{(influence.data,} + \DataTypeTok{sortvar=}\NormalTok{TE,} + \DataTypeTok{rightcols =} \KeywordTok{c}\NormalTok{(}\StringTok{"TE"}\NormalTok{,}\StringTok{"ci"}\NormalTok{,}\StringTok{"I2"}\NormalTok{),} + \DataTypeTok{smlab =} \StringTok{"Sorted by Effect size"}\NormalTok{)} +\NormalTok{\}\} } +\end{Highlighting} +\end{Shaded} + +The \texttt{influence.analysis} function has \textbf{three parameters} +which we have to define in the function. + +\begin{tabular}{l|l} +\hline +Code & Description\\ +\hline +data & The output object from our meta-analysis. In my case, this is 'data=m.hksj'\\ +\hline +method.tau & The method we used to estimate tau-squared (see Chapter 4.2.1). If you haven't set the estimator 'method.tau' in your analysis, use 'DL' because the DerSimonian-Laird estimator is the default in meta\\ +\hline +hakn & Weather we used the Knapp-Hartung-Sidik-Jonkman adjustments. If yes, use hakn=TRUE. If not, use hakn=FALSE\\ +\hline +\end{tabular} + +This is how the function code looks for my \texttt{m.hksj} data: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{influence.analysis}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj,}\DataTypeTok{method.tau =} \StringTok{"SJ"}\NormalTok{, }\DataTypeTok{hakn =} \OtherTok{TRUE}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +Now, let's have a look at the output. + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-105-1} + +} + +\caption{Influence Analyses}\label{fig:unnamed-chunk-105} +\end{figure}\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-106-1} + +} + +\caption{Baujat Plot}\label{fig:unnamed-chunk-106} +\end{figure} + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-107-1} + +} + +\caption{Leave-One-Out-Analyses}\label{fig:unnamed-chunk-1071} +\end{figure}\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-107-2} + +} + +\caption{Leave-One-Out-Analyses}\label{fig:unnamed-chunk-1072} +\end{figure} + +As you can see, the \texttt{influence.analysis} function gives us +various types of \textbf{plots} as output. Let's interpret them one by +one. + +\textbf{Influence Analyses} + +In the first analysis, you can see different influence measures, for +which we can see \textbf{graphs including each individual study of our +meta-analysis}. This type of \textbf{influence analysis} has been +proposed by Viechtbauer and Cheung \citep{viechtbauer2010outlier}. We'll +discuss the most important ones here: + +\begin{itemize} +\tightlist +\item + \textbf{dffits}: The DIFFITS value of a study indicates in standard + deviations how much the predicted pooled effect changes after + excluding this study +\item + \textbf{cook.d}: The \textbf{Cook's distance} resembles the + \textbf{Mahalanobis distance} you may know from outlier detection in + conventional multivariate statistics. It is the distance between the + value once the study is included compared to when it is excluded +\item + \textbf{cov.r}: The \textbf{covariance ratio} is the determinant of + the variance-covariance matrix of the parameter estimates when the + study is removed, divided by the determinant of the + variance-covariance matrix of the parameter estimates when the full + dataset is considered. Importantly, values of cov.r \textless{} 1 + indicate that removing the study will lead to a more precise effect + size estimation (i.e., less heterogeneity). +\end{itemize} + +Usually, however, you don't have to dig this deep into the calculations +of the individual measures. As a rule of thumb, \textbf{influential +cases} are studies with \textbf{very extreme values in the graphs}. +Viechtbauer and Cheung have also proposed cut-offs when to define a a +study as an influential case, for example (with \(p\) being the number +of model coefficients and \(k\) the number of studies): + +\[ DFFITS > 3\times\sqrt{\frac{p}{k-p}}\] \[ hat > 3\times\frac{p}{k}\] + +If a case was determined being \textbf{an influential case using these +cut-offs}, its value will be displayed in \textbf{red} (in our example, +this is the case for study number 3). + +\begin{rmdachtung} +Please note, as Viechtbauer \& Cheung emphasize, that \textbf{these +cut-offs are set on somewhat arbitrary thresholds}. Therefore, you +should never only have a look on the color of the study, but the general +structure of the graph, and interpret results in context. + +In our example, we see that while only Study 3 is defined as an +influential case, there are \textbf{actually two spiked in most plots}, +while the other studies all quite have the same value. Given this +structure, we could also decide to define \textbf{Study 16} as an +influential case too, because its values are very extreme too. +\end{rmdachtung} + +Let's have a look what the \textbf{3rd and 16th study} in our +\texttt{m.hksj} meta-analysis output were. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{studlab[}\KeywordTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{,}\DecValTok{16}\NormalTok{)]} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] "DanitzOrsillo" "Shapiro et al." +\end{verbatim} + +This is an interesting finding, as we \textbf{detected the same studies +when only looking at statistical outliers}. This further corroborates +that these two studies could maybe have distorted our pooled effect +estimate, and \textbf{might cause parts of the between-group +heterogeneity we found in our meta-analysis}. + +\textbf{Baujat Plot} + +The Baujat Plot \citep{baujat2002graphical} is a diagnostic plot to +detect studies \textbf{overly contributing to the heterogeneity of a +meta-analysis}. The plot shows the contribution of each study to the +overall heterogeneity as measured by Cochran's \emph{Q} on the +\textbf{horizontal axis}, and its \textbf{influence on the pooled effect +size} on the vertical axis. As we want to assess heterogeneity and +studies contributing to it, all studies \textbf{on the right side of the +plot are important to look at}, as this means that they cause much of +the heterogeneity we observe. \textbf{This is even more important when a +study contributes much to the overall heterogeneity, while at the same +time being not very influential concerning the overall pooled effect} +(e.g., because the study had a very small sample size). Therefore, +\textbf{all studies on the right side of the Baujat plot}, especially in +the \textbf{lower part}, are important for us. + +As you might have already recognized, the only two \textbf{studies we +find in those regions of the plot are the two studies we already +detected before} (Danitz \& Orsillo, Shapiro et al.). These studies +don't have a large impact on the overall results (presumably because +they are very small), but they do \textbf{add substantially to the +heterogeneity we found in the meta-analysis}. + +\textbf{Leave-One-Out Analyses} + +In these to forest plots, we see the \textbf{pooled effect recalculated, +with one study omitted each time}. There are two plots, which provide +the same data, but are ordered by different values. + +The \textbf{first plot is ordered by heterogeneity (low to high), as +measured by \emph{I}\textsuperscript{2} }. We see in the plot that the +lowest \emph{I}\textsuperscript{2} heterogeneity is reached (as we've +seen before) by omitting the studies \textbf{Danitz \& Orsillo} and +\textbf{Shapiro et al.}. This again corroborates our finding that these +two studies were the main ``culprits'' for the between-study +heterogeneity we found in the meta-analysis. + +The \textbf{second plot is ordered by effect size (low to high)}. Here, +we see how the overall effect estimate changes with one study removed. +Again, as the two outlying studies have very high effect sizes, we find +that the overall effect is smallest when they are removed. + +All in all, the results of our \textbf{outlier and influence analysis} +in this example point in the \textbf{same direction}. The two studies +are probably \textbf{outliers which may distort the effect size +estimate}, as well as its \textbf{precision}. We should therefore also +conduct and report a \textbf{sensitivity analysis in which these studies +are excluded}. + +\begin{rmdachtung} +\textbf{The influence analysis function for fixed-effect-model +meta-analyses} + +The \texttt{influence.analysis} function we presented above can only be +used for \textbf{random-effect meta-analyses}. If you want to perform +influence analyses for meta-analyses in \textbf{which you pooled the +effects with a fixed-effect model}, you will have to use the +\texttt{influence.analysis.fixed} function, which can be found +\href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/influence_analysis_function_for_fixed_effect_model.R}{here}. + +To use this function, \textbf{you only have to set the parameter} +\texttt{data}, as \texttt{method.tau} and \texttt{hakn} only apply to +random-effect-models. +\end{rmdachtung} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{Subgroup Analyses}\label{subgroup} + +\begin{figure} +\centering +\includegraphics{subgroup.jpg} +\caption{} +\end{figure} + +In \protect\hyperlink{heterogeneity}{Chapter 6}, we discussed in depth +why \textbf{between-study heterogeneity} is such an important issue in +interpreting the results of our meta-analysis, and how we can +\textbf{explore sources of heterogeneity} using +\protect\hyperlink{outliers}{outlier} and +\protect\hyperlink{influenceanalyses}{influence analyses}. + +Another source of between-study heterogeneity making our effect size +estimate less precise could be that \textbf{there are slight differences +in the study design or intervention components between the studies}. For +example, in a meta-analysis on the effects of \textbf{cognitive +behavioral therapy} (CBT) for \textbf{depression} in \textbf{university +students}, it could be the case that some studies delivered the +intervention in a \textbf{group setting}, while others delivered the +therapy to each student \textbf{individually}. In the same example, it +is also possible that studies used different \textbf{criteria} to +determine if a student suffers from \textbf{depression} (e.g.~they +either used the \emph{ICD-10} or the \emph{DSM-5} diagnostic manual). + +Many other differences of this sort are possible, and it seems plausible +that such study differences may also be associated with differences in +the overall effect. + +In \textbf{subgroup analyses}, we therefore have a look at different +\textbf{subgroups within the studies of our meta-analysis} and try to +determine of the \textbf{differ between these subgroups}. + +\begin{rmdinfo} +\textbf{The idea behind subgroup analyses} + +Basically, a every subgroup analysis consists of \textbf{two parts}: (1) +\textbf{pooling the effect of each subgroup}, and (2) \textbf{comparing +the effects of the subgroups} {[}@borenstein2013meta{]}. + +\textbf{1. Pooling the effect of each subgroup} + +This point it rather straightforward, as the same criteria as the ones +for a \textbf{simple meta-analysis without subgroups} (see +\protect\hyperlink{pool}{Chapter 4} and +\protect\hyperlink{random}{Chapter 4.2}) apply here. + +\begin{itemize} +\tightlist +\item + If you assume that \textbf{all studies in subgroup} stem from the same + population, and all have \textbf{one shared true effect}, you may use + the \textbf{fixed-effect-model}. As we mention in + \protect\hyperlink{pool}{Chapter 4}, many \textbf{doubt} that this + assumption is ever \textbf{true in psychological} and \textbf{medical + research}, even when we partition our studies into subgroups. +\item + The alternative, therefore, is to use a \textbf{random-effect-model} + which assumes that the studies within a subgroup are drawn from a + \textbf{universe} of populations follwing its own distribution, for + which we want to estimate the \textbf{mean}. +\end{itemize} + +\textbf{2. Comparing the effects of the subgroups} + +After we calculated the pooled effect for each subgroup, \textbf{we can +compare the size of the effects of each subgroup}. However, to know if +this difference is in fact singnificant and/or meaningful, we have to +calculate the \textbf{Standard Error of the differences between subgroup +effect sizes} \(SE_{diff}\), to calculate \textbf{confidence intervals} +and conduct \textbf{significance tests}. There are \textbf{two ways to +calculate} \(SE_{diff}\), and both based on different assumptions. + +\begin{itemize} +\tightlist +\item + \textbf{Fixed-effects (plural) model}: The fixed-effects-model for + subgroup comparisons is appropriate when \textbf{we are only + interested in the subgroups at hand} {[}@borenstein2013meta{]}. This + is the case when \textbf{the subgroups we chose to examine} were not + randomly ``chosen'', but represent fixed levels of a characteristic we + want to examine. Gender is such a characteristic, as its two subgroups + \textbf{female} and \textbf{male} were not randomly chosen, but are + the two subgroups that gender (in its classical conception) has. Same + does also apply, for example, if we were to examine if studies in + patients with \textbf{clinical depression} versus \textbf{subclinical + depression} yield different effects. Borenstein and Higgins + {[}@@borenstein2013meta{]} argue that the \textbf{fixed-effects + (plural) model} may be the \textbf{only plausible model} for most + analysis in \textbf{medical research, prevention, and other fields}. +\end{itemize} + +As this model assumes that \textbf{no further sampling error is +introduced at the subgroup level} (because subgroups were not randomly +sampled, but are fixed), \(SE_{diff}\) only depends on the +\emph{variance within the subgroups} \(A\) and \(B\), \(V_A\) and +\(V_B\). + +\[V_{Diff}=V_A + V_B\] + +The fixed-effects (plural) model can be used to test differences in the +pooled effects between subgroups, while the pooling \textbf{within the +subgroups is still conducted using a random-effects-model}. Such a +combination is sometimes called a \textbf{mixed-effects-model}. We'll +show you how to use this model in R in the +\protect\hyperlink{mixed}{next chapter}. + +\begin{itemize} +\tightlist +\item + \textbf{Random-effects-model}: The random-effects-model for + between-subgroup-effects is appropriate when the \textbf{subgroups we + use were randomly sampled from a population of subgroups}. Such an + example would be if we were interested if the effect of an + intervention \textbf{varies by region} by looking at studies from 5 + different countries (e.g., Netherlands, USA, Australia, China, + Argentina). These variable ``region'' has many different potential + subgroups (countries), from which we randomly selected five means that + this has introduced a \textbf{new sampling error}, for which we have + to control for using the \textbf{random-effects-model} for + between-subgroup-comparisons. +\end{itemize} + +The (simplified) formula for the estimation of \(V_{Diff}\) using this +model therefore looks like this: + +\[V_{Diff}=V_A + V_B + \frac{\hat T^2_G}{m} \] + +Where \(\hat T^2_G\) is the \textbf{estimated variance between the +subgroups}, and \(m\) is the \textbf{number of subgroups}. +\end{rmdinfo} + +\begin{rmdachtung} +Be aware that subgroup analyses should \textbf{always be based on an +informed, \emph{a priori} decision} which subgroup differences within +the study might be \textbf{practically relevant}, and would lead to +information gain on relevant \textbf{research questions} in your field +of research. It is also \textbf{good practice} to specify your subgroup +analyses \textbf{before you do the analysis}, and list them in +\textbf{the registration of your analysis}. + +It is also important to keep in mind that \textbf{the capabilites of +subgroup analyses to detect meaningful differences between studies is +often limited}. Subgroup analyses also need \textbf{sufficient power}, +so it makes no sense to compare two or more subgroups when your entire +number of studies in the meta-analysis is smaller than \(k=10\) +{[}@higgins2004controlling{]}. +\end{rmdachtung} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Subgroup Analyses using the Mixed-Effects-Model}\label{mixed} + +To conduct subgroup analyses using the \textbf{Mixed-Effects-Model} +(random-effects-model within subgroups, fixed-effects-model between +subgroups), you can use the \texttt{subgroup.analysis.mixed.effects} +function we prepared for you. To use the function, \texttt{meta} and +\texttt{metafor} need to be installed and loaded in your library. + +As the code for the function is pretty long, we \textbf{don't display it +here}. To access the function, use this +\href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/subgroup_analyses_mixed_effects_function.R}{link}. +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code from the website \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +For the \texttt{subgroup.analysis.mixed.effects} function, the following +parameters have to be set: + +\begin{tabular}{l|l} +\hline +Code & Description\\ +\hline +data & The output of you meta-analysis. In my case, this is 'm.hksj'\\ +\hline +sg.var & The variable in our dataset in which we coded which study belongs to which subgroup. Note that we also tell the function in which dataset this variable is stored. In my case, this was the 'madata' dataset i used to get the meta-analysis output 'm.hksj'. The dataset and the subgroup variable have to be connected with \$ (e.g. madata\$Control).\\ +\hline +n.sg & The number of subgroups we want to inlcude in our subgroup analysis (e.g. n.sg = 2)\\ +\hline +subgroup[x] & Here, we specify all the subgroups we want to include in the meta-analysis. Subgroup Analyses with up to 6 subgroups are possible with this function. The 'subgroup' parameters have to be numbered (e.g. subgroup1 = 'Name of your first subgroup', subgroup2 = 'Name of your second subgroup', ...)\\ +\hline +\end{tabular} + +In my \texttt{madata} dataset, which i used previously to generate my +meta-analysis output \texttt{m.hksj}, i stored the subgroup variable +\texttt{Control}. This variable specifies \textbf{which control group +type was employed in which study}. There are \textbf{three subgroups}: +\texttt{WLC} (waitlist control), \texttt{no\ intervention} and +\texttt{information\ only}. + +The function to do a subgroup analysis using the mixed-effects-model +with these paramters looks like this. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{subgroup.analysis.mixed.effects}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj,} + \DataTypeTok{sg.var=}\NormalTok{madata}\OperatorTok{$}\NormalTok{Control,} + \DataTypeTok{n.sg =} \DecValTok{3}\NormalTok{,} + \DataTypeTok{subgroup1 =} \StringTok{"WLC"}\NormalTok{,} + \DataTypeTok{subgroup2 =} \StringTok{"no intervention"}\NormalTok{,} + \DataTypeTok{subgroup3 =} \StringTok{"information only"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## 95%-CI %W(fixed) meta +## 1 0.7836 [0.3721; 1.1951] 8.9 3 +## 2 0.5108 [0.2767; 0.7449] 27.6 2 +## 3 0.4048 [0.2505; 0.5592] 63.5 1 +## +## Number of studies combined: k = 3 +## +## 95%-CI z p-value +## Fixed effect model 0.4679 [0.3449; 0.5909] 7.46 < 0.0001 +## +## Quantifying heterogeneity: +## tau^2 = 0.0079; H = 1.23 [1.00; 2.15]; I^2 = 34.0% [0.0%; 78.4%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 3.03 2 0.2196 +## +## Results for subgroups (fixed effect model): +## k 95%-CI Q tau^2 I^2 +## meta = information only 1 0.4048 [0.2505; 0.5592] 0.00 -- -- +## meta = no intervention 1 0.5108 [0.2767; 0.7449] 0.00 -- -- +## meta = WLC 1 0.7836 [0.3721; 1.1951] 0.00 -- -- +## +## Test for subgroup differences (fixed effect model): +## Q d.f. p-value +## Between groups 3.03 2 0.2196 +## Within groups 0.00 0 -- +## +## Details on meta-analytical method: +## - Inverse variance method +\end{verbatim} + +The results of the subgroup analysis are displayed under +\texttt{Results\ for\ subgroups\ (fixed\ effect\ model)}. We see that, +while the \textbf{pooled effects of the subgroups differ quite +substantially} (\emph{g} = 0.41-0.78), this difference is \textbf{not +statistically significant}. + +This can be seen under \texttt{Test\ for\ subgroup\ differences} in the +\texttt{Between\ groups} row. We can see that \(Q=3.03\) and +\(p=0.2196\). This information can be reported in our meta-analysis +paper. + +\begin{rmdachtung} +Please not that the values displayed under \texttt{k} in the +\texttt{Results\ for\ subgroups\ (fixed\ effects\ model)} section are +always 1, as the pooled effect of the subgroup is treated as a single +study. To determine the actual \(k\) of each subgroup, you can use the +\texttt{count} function from \texttt{dplyr} in R. +\end{rmdachtung} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(dplyr)} +\NormalTok{dplyr}\OperatorTok{::}\KeywordTok{count}\NormalTok{(madata, }\DataTypeTok{vars=}\NormalTok{madata}\OperatorTok{$}\NormalTok{Control)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## # A tibble: 3 x 2 +## vars n +## +## 1 information only 3 +## 2 no intervention 8 +## 3 WLC 7 +\end{verbatim} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Subgroup Analyses using the +Random-Effects-Model}\label{subgroup-analyses-using-the-random-effects-model} + +Now, let's assume i want to \textbf{know if intervention effects in my +meta-analysis differ by region}. I use a \textbf{random-effects-model} +and the selected coutries Argentina, Australia, China, and the +Netherlands. + +Again, i use the \texttt{m.hksj} meta-analysis output object. I can +perform a random-effects-model for between-subgroup-differences using +the \texttt{update.meta} function. For this function, we have to +\textbf{set two parameters}. + +\begin{tabular}{l|l} +\hline +Code & Description\\ +\hline +byvar & Here, we specify the variable in which the subgroup of each study is stored\\ +\hline +comb.random & Weather we want to use a random-effects-model for between-subgroup-differences. In this case, we have to set comb.random = TRUE\\ +\hline +\end{tabular} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{region.subgroup<-}\KeywordTok{update.meta}\NormalTok{(m.hksj, } + \DataTypeTok{byvar=}\NormalTok{region, } + \DataTypeTok{comb.random =} \OtherTok{TRUE}\NormalTok{, } + \DataTypeTok{comb.fixed =} \OtherTok{FALSE}\NormalTok{)} +\NormalTok{region.subgroup} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## 95%-CI %W(random) region +## Call et al. 0.7091 [ 0.1979; 1.2203] 5.2 Netherlands +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 6.1 Netherlands +## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 4.2 Netherlands +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 7.1 USA +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 6.8 USA +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 6.1 USA +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 5.7 USA +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 5.9 Argentina +## Hintz et al. 0.2840 [-0.0453; 0.6133] 6.5 Argentina +## Kang et al. 1.2751 [ 0.6142; 1.9360] 4.3 Argentina +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 6.1 Australia +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 5.6 Australia +## Phang et al. 0.5407 [ 0.0619; 1.0196] 5.4 Australia +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 5.3 China +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 4.1 China +## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 4.5 China +## SongLindquist 0.6126 [ 0.1683; 1.0569] 5.7 China +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 5.4 China +## +## Number of studies combined: k = 18 +## +## 95%-CI t p-value +## Random effects model 0.5935 [0.3891; 0.7979] 6.13 < 0.0001 +## +## Quantifying heterogeneity: +## tau^2 = 0.1337; H = 1.64 [1.27; 2.11]; I^2 = 62.6% [37.9%; 77.5%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 45.50 17 0.0002 +## +## Results for subgroups (random effects model): +## k 95%-CI Q tau^2 I^2 +## region = Netherlands 3 0.9142 [-0.9150; 2.7433] 13.06 0.4508 84.7% +## region = USA 4 0.4456 [ 0.0600; 0.8312] 6.87 0.0357 56.3% +## region = Argentina 3 0.6371 [-0.5837; 1.8580] 6.95 0.1826 71.2% +## region = Australia 3 0.3194 [-0.2427; 0.8815] 2.13 0.0204 6.1% +## region = China 5 0.7098 [ 0.2018; 1.2177] 7.81 0.1110 48.8% +## +## Test for subgroup differences (random effects model): +## Q d.f. p-value +## Between groups 4.52 4 0.3405 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +\end{verbatim} + +Here, we get the \textbf{pooled effect for each subgroup} (country). +Under +\texttt{Test\ for\ subgroup\ differences\ (random\ effects\ model)}, we +can see the \textbf{test for subgroup differences using the +random-effects-model}, which is \textbf{not significant} +(\(Q=4.52\),\(p=0.3405\)). This means that we did not find differences +in the overall effect between different regions, represented by the +country in which the study was conducted. + +\begin{rmdachtung} +\textbf{Using a fixed-effect-model for within-subgroup-pooling and a +fixed-effects-model for between-subgroup-differences} + +To use a fixed-effect-model in combination with a fixed-effects-model, +we can also use the \texttt{update.meta} function again. The procedure +is the same as the one we described before, but we have to set +\texttt{comb.random} as \texttt{FALSE} and \texttt{comb.fixed} as +\texttt{TRUE}. +\end{rmdachtung} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{Meta-Regression}\label{meta-regression} + +\begin{figure} +\centering +\includegraphics{regressionbild.jpg} +\caption{} +\end{figure} + +Conceptually, \textbf{Meta-Regression} does not differ much from a +\textbf{subgroup analysis}. In fact, subgroup analyses with more than +two groups are nothing more than a meta-regression with categorial +covariates. However, meta-regression does also allow us to use +\textbf{continuous data} as covariates and check weather values of this +variable are associated with effect size. + +\begin{rmdinfo} +\textbf{The idea behind meta-regression} + +You may have already performed regressions in regular data where +participants or patients are the \textbf{unit of analysis}. In typical +Meta-Analyses, we do not have the individual data for each participant +available, but only the \textbf{aggregated effects}, which is why we +have to perform meta-regressions with covariates at the \textbf{study +level}. This also means that while we conduct analyses on participant +samples much larger than usual in single studies, it is still very +likely that \textbf{we don't have enough studies for a meta-regression +to be sensible}. In \protect\hyperlink{subgroup}{Chapter 7}, we told you +that subgroup analyses make no sense when \emph{k}\textless{}10. For +\textbf{meta-regression}, Borenstein and colleages {[}@borenstein2011{]} +recommend that \textbf{each covariate should at least contain ten +studies}, although this should not be seen as clear rule. + +In a conventional regression, we want to estimate a parameter \(y\) +using a covariate \(x_i\) with \(n\) regression coefficients \(\beta\). +A standard regression equation therefore looks like this: + +\[y=\beta_0 + \beta_1x_1 + ...+\beta_nx_n\] + +In a meta-regression, we want to estimate the \textbf{effect size} +\(\theta\) for different values of the covariate(s), so our regression +looks like this: + +\[\hat \theta_k = \theta + \beta_1x_{1k} + ... + \beta_nx_{nk} + \epsilon_k + \zeta_k\] + +You might have seen that when estimating the effect size \(\theta_k\) of +a study \(k\) in our regression model, there are two \textbf{extra terms +in the equation}, \(\epsilon_k\) and \(\zeta_k\). The same terms can +also be found in the equation for the random-effects-model in +\protect\hyperlink{random}{Chapter 4.2}. The two terms signify two types +of \textbf{independent errors} which cause our regression prediction to +be \textbf{imperfect}. The first one, \(\epsilon_k\), is the sampling +error through which the effect size of the study deviates from its +``true'' effect. The second one, \(\zeta_k\), denotes that even the true +effect size of the study is only sampled from \textbf{an overarching +distribution of effect sizes} (see the Chapter on the +\protect\hyperlink{random}{Random-Effects-Model} for more details). In a +\textbf{fixed-effect-model}, we assume that all studies actually share +the \textbf{same true effect size} and that the \textbf{between-study +heterogeneity} \(\tau^2 = 0\). In this case, we do not consider +\(\zeta_k\) in our equation, but only \(\epsilon_k\). + +As the equation above has includes \textbf{fixed effects} (the \(\beta\) +coefficients) as well as \textbf{random effects} (\(\zeta_k\)), the +model used in meta-regression is often called \textbf{a +mixed-effects-model}. Mathematically, this model is identical to the +\textbf{mixed-effects-model} we described in +\protect\hyperlink{subgroup}{Chapter 7} where we explained how +\textbf{subgroup analyses} work. + +Indeed \textbf{subgroup analyses with more than two subgroups} are +nothing else than a \textbf{meta-regression} with a \textbf{categorical +predictor}. For meta-regression, these subgroups are then +\textbf{dummy-coded}, e.g. + +\[ D_k = \{\begin{array}{c}0:ACT \\1:CBT \end{array}\] + +\[\hat \theta_k = \theta + \beta x_{k} + D_k \gamma + \epsilon_k + \zeta_k\] + +In this case, we assume the same \textbf{regression line}, which is +simply ``shifted'' \textbf{up or down for the different subgroups} +\(D_k\). +\end{rmdinfo} + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-126-1} + +} + +\caption{Visualisation of a Meta-Regression with dummy-coded categorial predictors}\label{fig:unnamed-chunk-126} +\end{figure} + +\begin{rmdinfo} +\textbf{Assessing the fit of a regression model} + +To evaluate the \textbf{statistical significance of a predictor}, we a +\textbf{t-test} of its \(\beta\)-weight is performed. + +\[ t=\frac{\beta}{SE_{\beta}}\] + +Which provides a \(p\)-value telling us if a variable significantly +predicts effect size differences in our regression model. + +If we fit a regression model, our aim is to find a model \textbf{which +explains as much as possible of the current variability in effect sizes} +we find in our data. + +In conventional regression, \(R^2\) is commonly used to quantify the +\textbf{goodness of fit} of our model in percent (0-100\%). As this +measure is commonly used, and many researchers know how to to interpret +it, we can also calculate a \(R^2\) analog for meta-regression using +this formula: + +\[R_2=\frac{\hat\tau^2_{REM}-\hat\tau^2_{MEM}}{\hat\tau^2_{REM}}\] + +Where \(\hat\tau^2_{REM}\) is the estimated total heterogenetiy based on +the random-effects-model and \(\hat\tau^2_{REM}\) the total +heterogeneity of our mixed-effects regression model. +\end{rmdinfo} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Calculating meta-regressions in +R}\label{calculating-meta-regressions-in-r} + +Meta-regressions can be conducted in R using the \texttt{metareg} +function in \texttt{meta}. To show the similarity between +\texttt{subgroup} analysis and \texttt{meta-regression} with categorical +predictors, i'll first conduct a meta-regression with my variable +``Control'' as predictor again. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{metareg}\NormalTok{(m.hksj,Control)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Mixed-Effects Model (k = 18; tau^2 estimator: SJ) +## +## tau^2 (estimated amount of residual heterogeneity): 0.1343 (SE = 0.0536) +## tau (square root of estimated tau^2 value): 0.3665 +## I^2 (residual heterogeneity / unaccounted variability): 73.92% +## H^2 (unaccounted variability / sampling variability): 3.84 +## R^2 (amount of heterogeneity accounted for): 0.00% +## +## Test for Residual Heterogeneity: +## QE(df = 15) = 40.0161, p-val = 0.0005 +## +## Test of Moderators (coefficient(s) 2:3): +## F(df1 = 2, df2 = 15) = 0.9467, p-val = 0.4100 +## +## Model Results: +## +## estimate se tval pval ci.lb ci.ub +## intrcpt 0.4252 0.2250 1.8899 0.0782 -0.0543 0.9048 +## Controlno intervention 0.1003 0.2678 0.3744 0.7134 -0.4706 0.6711 +## ControlWLC 0.3380 0.2765 1.2224 0.2404 -0.2514 0.9274 +## +## intrcpt . +## Controlno intervention +## ControlWLC +## +## --- +## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 +\end{verbatim} + +We see in the output that the \texttt{metareg} function uses the values +of ``Control'' (i.e, the three different types of control groups) as a +\textbf{moderator}. It takes \textbf{``information only''} as a +dummy-coded \emph{reference group}, and \textbf{``no intervention''} and +\textbf{``WLC''} as dummy-coded \textbf{predictors}. Under +\texttt{Test\ of\ Moderators}, we can see that control groups are not +significantly associated with effect size differences +\(F_{2,15}=0.947\), \(p=0.41\). Our regression model does not explain +any of the variability in our effect size data (\(R^2=0\%\)). + +Below \texttt{Model\ Results}, we can also see the \(\beta\)-values +(\texttt{estimate}) of both predictors, and their significance level +\texttt{pval}. As we can see, both predictors were not significant. + +\textbf{Continuous variables} + +Let's assume i want to check if the \textbf{publication year} is +associated with effect size. I have stored the variable +\texttt{pub\_year}, containing the publication year of every study in my +dataset, and conducted the meta-analysis with it. I stored my +meta-analysis output in the \texttt{m.pubyear} output. + +\textbf{Now, i can use this predictor in a meta-regression.} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{output.metareg<-}\KeywordTok{metareg}\NormalTok{(m.pubyear,pub_year)} +\NormalTok{output.metareg} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Mixed-Effects Model (k = 18; tau^2 estimator: DL) +## +## tau^2 (estimated amount of residual heterogeneity): 0.0831 (SE = 0.0488) +## tau (square root of estimated tau^2 value): 0.2883 +## I^2 (residual heterogeneity / unaccounted variability): 64.69% +## H^2 (unaccounted variability / sampling variability): 2.83 +## R^2 (amount of heterogeneity accounted for): 0.00% +## +## Test for Residual Heterogeneity: +## QE(df = 16) = 45.3076, p-val = 0.0001 +## +## Test of Moderators (coefficient(s) 2): +## QM(df = 1) = 0.0054, p-val = 0.9412 +## +## Model Results: +## +## estimate se zval pval ci.lb ci.ub +## intrcpt -1.4580 27.6151 -0.0528 0.9579 -55.5825 52.6666 +## pub_year 0.0010 0.0137 0.0737 0.9412 -0.0259 0.0280 +## +## --- +## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 +\end{verbatim} + +As you can see from the output, \texttt{pub\_year} was now included as a +\textbf{predictor}, but it is not significantly associated with the +effect size (\(p=0.9412\)). + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Plotting regressions}\label{plotting-regressions} + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-131-1} + +} + +\caption{A finished bubble plot}\label{fig:unnamed-chunk-131} +\end{figure} + +To plot our meta-regression output, we can use the \texttt{bubble} +function in \texttt{meta}. Here a few parameters we can specify for this +function. + +\begin{tabular}{l|l} +\hline +Parameter & Function\\ +\hline +xlim & The x-axis limit of the plot. Must be specified as, e.g., 'xlim=c(0,1)'\\ +\hline +ylim & The y-axis limit of the plot. Must be specified as, e.g., 'ylim=c(0,1)'\\ +\hline +xlab & The label for the x axis\\ +\hline +ylab & The label for the y axis\\ +\hline +col & The color of the individual studies\\ +\hline +lwd & The line width of the regression line\\ +\hline +col.line & The color of the regression line\\ +\hline +studlab & If the labels for each study should be printed within the plot (TRUE/FALSE)\\ +\hline +\end{tabular} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{bubble}\NormalTok{(output.metareg,} + \DataTypeTok{xlab =} \StringTok{"Publication Year"}\NormalTok{,} + \DataTypeTok{col.line =} \StringTok{"blue"}\NormalTok{,} + \DataTypeTok{studlab =} \OtherTok{TRUE}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\chapter{Publication Bias}\label{publication-bias} + +\begin{figure} +\centering +\includegraphics{publicationbias.jpg} +\caption{} +\end{figure} + +In the last chapters, we have showed you how to \textbf{pool effects} in +meta-analysis, choose the \textbf{right pooling model}, assess the +\textbf{heterogeneity} of your effect estimate, and determine sources of +heterogeneity through \textbf{outlier, influence, and subgroup +analyses}. + +Nevertheless, even the most thoroughly conducted meta-analysis can only +work with the \textbf{study material at hand}. An issue commonly +discussed in research, however, is the \textbf{file-drawer} or +\textbf{publication bias} problem, which states that a study with high +effect sizes \textbf{is more likely to be published} than a study with a +low effect size \citep{rothstein2006publication}. + +Such \textbf{missing studies} with low effect sizes, it is assumed, thus +never get published and therefore cannot be integrated into our +meta-analysis. This leads to \textbf{publication bias}, as the pooled +effect we estimate in our meta-analysis might be higher than the +\textbf{true effect size} because we did not consider the missing +studies with lower effects due to the simple fact that they were never +published. + +Although this practice is gradually changing +\citep{nelson2018psychology}, weather a study is published or not +heavily depends on the \textbf{statistical significance} (\(p<0.05\)) of +its results \citep{dickersin2005publication}. For any sample size, a +result is more likely to become \textbf{statistically significant} if +its \textbf{effect size is high}. This is particularly true for +\textbf{small studies}, for which very large effect sizes are needed to +attain a statisitcally significant result. + +In the \protect\hyperlink{smallstudyeffects}{following chapter}, we will +describe the \textbf{idea behind statistical models for publication +bias} in further depth. We termed these concepts and methods +\textbf{small-study effect methods}, as it is small studies that they +mostly focus on. These methods assume that publication bias is primarily +driven by \textbf{effect size} and because researchers +\textbf{immediately put every study in the file drawer once the results +are not significant}. + +Recently, it has been argued that these \textbf{assumptions may not be +true}, and that publication bias is mostly caused by +\textbf{significance levels} and \textbf{p-hacking} +\citep{simonsohn2014p}. An alternative method called \textbf{P-Curve} +has therefore been suggested to examine \textbf{publication bias}. We +will present this method in the \protect\hyperlink{pcurve}{last chapter} +of this section. + +\begin{rmdinfo} +\textbf{Which method should i use for my meta-analysis?} + +While recent research suggests that the conventional \textbf{small-study +effect methods} may have substantial \textbf{limitations}, and that +\textbf{p-Curve} may be able to estimate the true effect with less bias +{[}@simonsohn2014p;@simonsohn2015better;@simonsohn2014pb{]}, please note +that both methods are based on different \textbf{theoretical +assumptions} about the origin of publication bias. As we cannot +ultimately decide which assumption is the \textbf{``true''} one in +specific research fields, and, in practice \textbf{the true effect is +unkown when doing meta-analysis}, we argue that you may use \textbf{both +methods} and compare results as \textbf{sensitivity analyses} +{[}@harrer2019internet{]}. + +\textbf{P-curve} was developed with \textbf{full-blown experimental +psychological research in mind}, in which researchers often have +\textbf{high degrees of ``researcher freedom''} {[}@simmons2011false{]} +in deleting outliers and performing statistical test on their data. + +We argue that this looks slightly different for \textbf{clinical +psychology} and the medical field, where researchers conduct +\textbf{randomized controlled trials} whith a clear \textbf{primary +outcome}: the difference between the control and the intervention group +after the treatment. While it is also true for \textbf{medicine and +clinical psychology that statistical significance plays an important +role}, the \textbf{effect size} of an intervention is often of greater +interest, as \textbf{treatments are often compared in terms of their +treatment effects} in this field. Furthermore, best practice for +randomized controlled trials is to perform \textbf{intention-to-treat} +analyses, in which all collected data in a trial has to be considered, +giving researchers less space to ``play around'' with their data and +perform p-hacking. While we certainly do not want to insinuate that +\textbf{outcome research in clinical psychology} is free from p-hacking +and bad data analysis practices, this should be seen as a +\textbf{caveat} that the assumptions of the small-study effects methods +may be more adequate for clinical psychology than other fields within +psychology, especially when **the risk of bias for each study is also +taken into account*. Facing this uncertainty, we think that conducting +both analyses and reporting them in our research paper may be the most +adequate approach until meta-scientific research gives us more certainty +about which \textbf{assumption actually best reflects the field of +clinical psychology}. +\end{rmdinfo} + +\hypertarget{smallstudyeffects}{\section{Small-study effect +methods}\label{smallstudyeffects}} + +The \textbf{small-study effect methods} we present here have been +conventional for many years. Thus various methods to assess and control +for publication bias have been developed, but we will only focus on the +most important ones here. + +\begin{rmdinfo} +\textbf{The model behind small-study effects methods} + +According to Borenstein et al. {[}@borenstein2011{]}. The model behind +the most common small-study effects methods has these core +\textbf{assumptions}: + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + Because they involve large commitment of ressources and time, + \textbf{large studies are likely to get published}, weather the + results are significant or not +\item + Moderately sized studies are at \textbf{greater risk of missing}, but + with a moderate sample size even moderately sized effects are likely + to become significant, which means that only some studies will be + missing +\item + Small studies are \textbf{at greatest risk} for being non-significant, + and thus being missing. Only small studies with a very large effect + size become significant, and will be found in the published + literature. +\end{enumerate} + +In accordance with these assumptions, the methods we present here +particularly focus \textbf{on small studies with small effect sizes, and +wheather they are missing}. +\end{rmdinfo} + +\subsection{Funnel plots}\label{funnel-plots} + +The best way to visualize weather \textbf{small studies with small +effect sizes are missing} is through \textbf{funnel plots}. + +We can generate a funnel plot for our \texttt{m.hksj} meta-analysis +output using the \texttt{funnel()} function in \texttt{meta}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{funnel}\NormalTok{(m.hksj,}\DataTypeTok{xlab =} \StringTok{"Hedges' g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-137-1.pdf} + +The \textbf{funnel plot} basically consists of a \textbf{funnel} and two +\textbf{axes}: the y-axis showing the \textbf{Standard Error} \(SE\) of +each study, with larger studies (which thus have a smaller \(SE\)) +plotted \textbf{on top of the y-axis}; and the x-axis showing the +\textbf{effect size} of each study. + +Given our assumptions, and in the case when there is \textbf{no +publication bias}, all studies would lie \textbf{symmetrically around +our pooled effect size (the striped line)} within the form of the +funnel. When \textbf{publication bias is present}, we would assume that +the funnel would look asymmetrical, because only the small studies with +a large effect size very published, \textbf{while small studies without +a significant, large effect would be missing}. + +We see from the plot that in the case of our meta-anlysis +\texttt{m.hksj}, the latter is probably true. We see that the plot is +highly asymmetrical, with exactly the small studies with low effect size +missing in the bottom-left corner of our plot. + +We can also display the name of each study using the \texttt{studlab} +parameter. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{funnel}\NormalTok{(m.hksj,}\DataTypeTok{xlab =} \StringTok{"g"}\NormalTok{,}\DataTypeTok{studlab =} \OtherTok{TRUE}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-138-1.pdf} + +Here, we see that asymmetry is primarily driven by \textbf{three studies +with high effects, but a small study sample} in the bottom right corner. +Interestingly, two of these studies are the ones we also detected in our +\protect\hyperlink{outliers}{outlier} and +\protect\hyperlink{influenceanalyses}{influence} analyses. + +An even better way to inspect the funnel plot is through +\textbf{contour-enhanced funnel plots}, which help to distinguish +publication bias from other forms of asymmetry +\citep{peters2008contour}. Contour-enhanced funnels include colors +signifying the significance level into which the effects size of each +study falls. We can plot such funnels using this code: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{funnel}\NormalTok{(m.hksj, }\DataTypeTok{xlab=}\StringTok{"Hedges' g"}\NormalTok{, } + \DataTypeTok{contour =} \KeywordTok{c}\NormalTok{(.}\DecValTok{95}\NormalTok{,.}\DecValTok{975}\NormalTok{,.}\DecValTok{99}\NormalTok{),} + \DataTypeTok{col.contour=}\KeywordTok{c}\NormalTok{(}\StringTok{"darkblue"}\NormalTok{,}\StringTok{"blue"}\NormalTok{,}\StringTok{"lightblue"}\NormalTok{))}\OperatorTok{+} +\KeywordTok{legend}\NormalTok{(}\FloatTok{1.4}\NormalTok{, }\DecValTok{0}\NormalTok{, }\KeywordTok{c}\NormalTok{(}\StringTok{"p < 0.05"}\NormalTok{, }\StringTok{"p<0.025"}\NormalTok{, }\StringTok{"< 0.01"}\NormalTok{),}\DataTypeTok{bty =} \StringTok{"n"}\NormalTok{,} + \DataTypeTok{fill=}\KeywordTok{c}\NormalTok{(}\StringTok{"darkblue"}\NormalTok{,}\StringTok{"blue"}\NormalTok{,}\StringTok{"lightblue"}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-140-1.pdf} + +We can see in the plot that while \textbf{some studies have +statistically significant effect sizes} (blue background), other do not +(white background). We also see a trend that while the moderately sized +studies are partly significant and non-significant, with slightly more +significant studies, the asymmetry is much larger for small studies. +This gives us a hint that publication bias might indeed be present in +our analysis. + +\subsection{Testing for funnel plot asymmetry using Egger's +test}\label{testing-for-funnel-plot-asymmetry-using-eggers-test} + +\textbf{Egger's test of the intercept} \citep{egger1997bias} quantifies +the funnel plot asymmetry and performs a statistical test. + +We have prepared a function called \texttt{eggers.test} for you, which +can be found below. The function is a wrapper for the \texttt{metabias} +function in \texttt{meta}. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{eggers.test<-}\ControlFlowTok{function}\NormalTok{(data)\{} + +\NormalTok{ data<-data} +\NormalTok{ eggers<-}\KeywordTok{metabias}\NormalTok{(data)} +\NormalTok{ intercept<-}\KeywordTok{as.numeric}\NormalTok{(eggers}\OperatorTok{$}\NormalTok{estimate[}\DecValTok{1}\NormalTok{])} +\NormalTok{ intercept<-}\KeywordTok{round}\NormalTok{(intercept,}\DataTypeTok{digits=}\DecValTok{3}\NormalTok{)} +\NormalTok{ se.intercept<-eggers}\OperatorTok{$}\NormalTok{estimate[}\DecValTok{2}\NormalTok{]} +\NormalTok{ lower.intercept<-}\KeywordTok{as.numeric}\NormalTok{(intercept}\OperatorTok{-}\FloatTok{1.96}\OperatorTok{*}\NormalTok{se.intercept)} +\NormalTok{ lower.intercept<-}\KeywordTok{round}\NormalTok{(lower.intercept,}\DataTypeTok{digits =} \DecValTok{2}\NormalTok{)} +\NormalTok{ higher.intercept<-}\KeywordTok{as.numeric}\NormalTok{(intercept}\OperatorTok{+}\FloatTok{1.96}\OperatorTok{*}\NormalTok{se.intercept)} +\NormalTok{ higher.intercept<-}\KeywordTok{round}\NormalTok{(higher.intercept,}\DataTypeTok{digits =} \DecValTok{2}\NormalTok{)} +\NormalTok{ ci.intercept<-}\KeywordTok{paste}\NormalTok{(lower.intercept,}\StringTok{"-"}\NormalTok{,higher.intercept)} +\NormalTok{ ci.intercept<-}\KeywordTok{gsub}\NormalTok{(}\StringTok{" "}\NormalTok{, }\StringTok{""}\NormalTok{, ci.intercept, }\DataTypeTok{fixed =} \OtherTok{TRUE}\NormalTok{)} +\NormalTok{ intercept.pval<-}\KeywordTok{as.numeric}\NormalTok{(eggers}\OperatorTok{$}\NormalTok{p.value)} +\NormalTok{ intercept.pval<-}\KeywordTok{round}\NormalTok{(intercept.pval,}\DataTypeTok{digits=}\DecValTok{5}\NormalTok{)} +\NormalTok{ eggers.output<-}\KeywordTok{data.frame}\NormalTok{(intercept,ci.intercept, intercept.pval)} + \KeywordTok{names}\NormalTok{(eggers.output)<-}\KeywordTok{c}\NormalTok{(}\StringTok{"intercept"}\NormalTok{,}\StringTok{"95%CI"}\NormalTok{,}\StringTok{"p-value"}\NormalTok{)} +\NormalTok{ title<-}\StringTok{"Results of Egger's test of the intercept"} + +\KeywordTok{print}\NormalTok{(title)} +\KeywordTok{print}\NormalTok{(eggers.output)} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +Now we can use the \texttt{eggers.test} function. We only have to +specify our meta-analysis output \texttt{m.hksj} as the \texttt{data} +the function should use. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{eggers.test}\NormalTok{(}\DataTypeTok{data=}\NormalTok{m.hksj)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] "Results of Egger's test of the intercept" +## intercept 95%CI p-value +## 1 4.111 2.39-5.83 0.00025 +\end{verbatim} + +The function returns the \texttt{intercept} along with its confidence +interval. We can see that the \texttt{p-value} of Egger's test is +\textbf{significant} (\(p<0.05\)), which means that there is substanital +asymmetry in the Funnel plot. This asymmetry could have been caused by +publication bias. + +\hypertarget{dant}{\subsection{Duval \& Tweedie's trim-and-fill +procedure}\label{dant}} + +\textbf{Duval \& Tweedie's trim-and-fill procedure} +\citep{duval2000trim} is also based the funnel plot and its +symmetry/asymmetry. When \textbf{Egger's test is significant}, we can +use this method to estimate what \textbf{the actaul effect size would be +had the ``missing'' small studies been published}. The procedure +\textbf{imputes} missing studies into the funnel plot until symmetry is +reached again. + +\begin{rmdinfo} +\textbf{The trim-and-fill procedure includes the following five steps} +{[}@schwarzer2015meta{]}: + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + Estimating the number of studies in the outlying (right) part of the + funnel plot +\item + Removing (trimming) these effect sizes and pooling the results with + the remaining effect sizes +\item + This pooled effect is then taken as the center of all effect sizes +\item + For each trimmed/removed study, a additional study is imputed, + mirroring the effect of the study on the left side of the funnel plot +\item + Pooling the results with the imputed studies and the trimmed studies + included +\end{enumerate} +\end{rmdinfo} + +The \textbf{trim-and-fill-procedure} can be performed using the +\texttt{trimfill} function in \texttt{meta}, and specifying our +meta-analysis output. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{trimfill}\NormalTok{(m.hksj)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## SMD 95%-CI %W(random) +## Call et al. 0.7091 [ 0.1979; 1.2203] 3.8 +## Cavanagh et al. 0.3549 [-0.0300; 0.7397] 4.1 +## DanitzOrsillo 1.7912 [ 1.1139; 2.4685] 3.3 +## de Vibe et al. 0.1825 [-0.0484; 0.4133] 4.4 +## Frazier et al. 0.4219 [ 0.1380; 0.7057] 4.3 +## Frogeli et al. 0.6300 [ 0.2458; 1.0142] 4.1 +## Gallego et al. 0.7249 [ 0.2846; 1.1652] 4.0 +## Hazlett-Stevens & Oren 0.5287 [ 0.1162; 0.9412] 4.0 +## Hintz et al. 0.2840 [-0.0453; 0.6133] 4.2 +## Kang et al. 1.2751 [ 0.6142; 1.9360] 3.4 +## Kuhlmann et al. 0.1036 [-0.2781; 0.4853] 4.1 +## Lever Taylor et al. 0.3884 [-0.0639; 0.8407] 3.9 +## Phang et al. 0.5407 [ 0.0619; 1.0196] 3.9 +## Rasanen et al. 0.4262 [-0.0794; 0.9317] 3.8 +## Ratanasiripong 0.5154 [-0.1731; 1.2039] 3.3 +## Shapiro et al. 1.4797 [ 0.8618; 2.0977] 3.5 +## SongLindquist 0.6126 [ 0.1683; 1.0569] 4.0 +## Warnecke et al. 0.6000 [ 0.1120; 1.0880] 3.9 +## Filled: Warnecke et al. 0.0520 [-0.4360; 0.5401] 3.9 +## Filled: SongLindquist 0.0395 [-0.4048; 0.4837] 4.0 +## Filled: Frogeli et al. 0.0220 [-0.3621; 0.4062] 4.1 +## Filled: Call et al. -0.0571 [-0.5683; 0.4541] 3.8 +## Filled: Gallego et al. -0.0729 [-0.5132; 0.3675] 4.0 +## Filled: Kang et al. -0.6230 [-1.2839; 0.0379] 3.4 +## Filled: Shapiro et al. -0.8277 [-1.4456; -0.2098] 3.5 +## Filled: DanitzOrsillo -1.1391 [-1.8164; -0.4618] 3.3 +## +## Number of studies combined: k = 26 (with 8 added studies) +## +## SMD 95%-CI t p-value +## Random effects model 0.3431 [ 0.0994; 0.5868] 2.90 0.0077 +## Prediction interval [-0.8463; 1.5326] +## +## Quantifying heterogeneity: +## tau^2 = 0.3181; H = 2.05 [1.70; 2.47]; I^2 = 76.2% [65.4%; 83.7%] +## +## Test of heterogeneity: +## Q d.f. p-value +## 105.15 25 < 0.0001 +## +## Details on meta-analytical method: +## - Inverse variance method +## - Sidik-Jonkman estimator for tau^2 +## - Hartung-Knapp adjustment for random effects model +## - Trim-and-fill method to adjust for funnel plot asymmetry +\end{verbatim} + +We see that the procedure identified and trimmed \textbf{eight studies} +\texttt{(with\ 8\ added\ studies)}). The overall effect estimated by the +procedure is \(g = 0.34\). + +Let's compare this to our initial results. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj}\OperatorTok{$}\NormalTok{TE.random} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 0.593535 +\end{verbatim} + +The initial pooled effect size was \(g = 0.59\), which is substantially +larger than the bias-corrected effect. In our case, if we assume that +\textbf{publication bias} was a problem in the analyses, the +\textbf{trim-and-fill procedure} lets us assume that our initial results +were \textbf{overestimated} due to publication bias, and the ``true'' +effect when controlling for selective publication might be \(g = 0.34\) +rather than \(g = 0.59\). + +If we store the results of the \texttt{trimfill} function in an +\textbf{object}, we can also create \textbf{funnel plots including the +imputed studies}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{m.hksj.trimfill<-}\KeywordTok{trimfill}\NormalTok{(m.hksj)} +\KeywordTok{funnel}\NormalTok{(m.hksj.trimfill,}\DataTypeTok{xlab =} \StringTok{"Hedges' g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-146-1.pdf} + +\hypertarget{pcurve}{\section{P-Curve}\label{pcurve}} + +In the last chapter, we showed you how you can apply \textbf{Egger's +test of the intercept}, \textbf{Duval \& Tweedie's trim and fill +procedure}, and inspect \textbf{Funnel plots} in R. + +As we have mentioned before, recent research has shown \textbf{that the +assumptions of the small-effect study methods may be inaccurate in many +cases}. The \textbf{Duval \& Tweedie trim-and-fill procedure} in +particular has been shown to be prone to providing \textbf{inaccurate +effect size estimates} \citep{simonsohn2014pb}. + +\textbf{P-curve Analysis} has been proposed as an alternative way to +assess publication bias and estimate the true effect behind our +collected data. + +P-Curve assumes that publication bias is not primarily generated because +researchers \textbf{do not publish non-significant results}, but because +the \textbf{``play'' around with their data (e.g., selectively removing +outliers, choosing different outcomes, controlling for different +variables) until a non-significant finding becomes significant}. This +(bad) practice is called \textbf{p-hacking}, and has been shown to be +extremely frequent among researchers \citep{head2015extent}. + +\begin{rmdinfo} +\textbf{The idea behind P-Curve} +\end{rmdinfo} + +\subsection{Preparing RStudio and our +data}\label{preparing-rstudio-and-our-data} + +To use \textbf{P-curve}, we need our data in the same format as the one +we used for the \texttt{metacont} function in +\protect\hyperlink{excel_preparation}{Chapter 3.1.1}. We need to specify +\texttt{Me}, \texttt{Se}, \texttt{Ne}, \texttt{Mc}, \texttt{Sc}, and +\texttt{Nc}. My \texttt{metacont} data already has this format. + +\begin{tabular}{l|l|l|l|r|l|l} +\hline +Author & Ne & Me & Se & Nc & Mc & Sc\\ +\hline +Cavanagh & 50 & 4.5 & 2.7 & 50 & 5.6 & 2.6\\ +\hline +Day & 64 & 18.3 & 6.4 & 65 & 20.2 & 7.6\\ +\hline +Frazier & 90 & 12.5 & 3.2 & 95 & 15.5 & 4.4\\ +\hline +Gaffney & 30 & 2.34 & 0.87 & 30 & 3.13 & 1.234\\ +\hline +Greer & 77 & 15.212 & 5.35 & 69 & 20.13 & 7.43\\ +\hline +Harrer & 60 & 3.153 & 1.256 & 60 & 3.4213 & 1.878\\ +\hline +\end{tabular} + +To use p-curve, we also have to install and load \textbf{four packages}. +The \texttt{esc} package \citep{esc}, the \texttt{compute.es} package +\citep{del2013compute}, the \texttt{stringr} package, and the +\texttt{poibin} package \citep{hong2011poibin}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(compute.es)} +\KeywordTok{library}\NormalTok{(esc)} +\KeywordTok{library}\NormalTok{(stringr)} +\KeywordTok{library}\NormalTok{(poibin)} +\end{Highlighting} +\end{Shaded} + +To \textbf{prepare our data} which we stored in the format described +above, we need to use the \texttt{pcurve\_dataprep} function, which we +prepared for you. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{pcurve_dataprep<-}\ControlFlowTok{function}\NormalTok{(data)\{} +\NormalTok{data<-data} +\NormalTok{Me<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Me)} +\NormalTok{Se<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Se)} +\NormalTok{Ne<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Ne)} +\NormalTok{Mc<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Mc)} +\NormalTok{Sc<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Sc)} +\NormalTok{Nc<-}\KeywordTok{as.numeric}\NormalTok{(data}\OperatorTok{$}\NormalTok{Nc)} + +\NormalTok{esc<-}\KeywordTok{esc_mean_sd}\NormalTok{(}\DataTypeTok{grp1m=}\NormalTok{Me, } + \DataTypeTok{grp1sd=}\NormalTok{Se, } + \DataTypeTok{grp1n=}\NormalTok{Ne, } + \DataTypeTok{grp2m=}\NormalTok{Mc, } + \DataTypeTok{grp2sd=}\NormalTok{Sc, } + \DataTypeTok{grp2n=}\NormalTok{Nc, } + \DataTypeTok{es.type =} \StringTok{"d"}\NormalTok{)} + +\NormalTok{output<-}\KeywordTok{des}\NormalTok{(}\DataTypeTok{d=}\NormalTok{esc}\OperatorTok{$}\NormalTok{es,}\DataTypeTok{n.1=}\NormalTok{Ne,}\DataTypeTok{n.2=}\NormalTok{Nc, }\DataTypeTok{verbose =} \OtherTok{FALSE}\NormalTok{)} +\NormalTok{output}\OperatorTok{$}\NormalTok{r<-}\KeywordTok{abs}\NormalTok{(output}\OperatorTok{$}\NormalTok{r)} +\NormalTok{tot<-}\KeywordTok{data.frame}\NormalTok{(}\KeywordTok{paste}\NormalTok{(}\StringTok{"r("}\NormalTok{,output}\OperatorTok{$}\NormalTok{N.total}\OperatorTok{-}\DecValTok{2}\NormalTok{,}\StringTok{")="}\NormalTok{,output}\OperatorTok{$}\NormalTok{r))} +\KeywordTok{colnames}\NormalTok{(tot)<-}\KeywordTok{c}\NormalTok{(}\StringTok{"output"}\NormalTok{)} +\NormalTok{tot}\OperatorTok{$}\NormalTok{output<-}\KeywordTok{gsub}\NormalTok{(}\StringTok{" "}\NormalTok{, }\StringTok{""}\NormalTok{, tot}\OperatorTok{$}\NormalTok{output, }\DataTypeTok{fixed =} \OtherTok{TRUE}\NormalTok{)} +\NormalTok{totoutput<-}\KeywordTok{as.character}\NormalTok{(tot}\OperatorTok{$}\NormalTok{output)} +\KeywordTok{print}\NormalTok{(tot, }\DataTypeTok{row.names =} \OtherTok{FALSE}\NormalTok{)} +\KeywordTok{write}\NormalTok{(totoutput,}\DataTypeTok{ncolumns=}\DecValTok{1}\NormalTok{, }\DataTypeTok{file=}\StringTok{"input.txt"}\NormalTok{)} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +To use the \texttt{pcurve\_dataprep} function, we simply have to specify +our dataset. In my case, this is \texttt{metacont}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{pcurve_dataprep}\NormalTok{(}\DataTypeTok{data=}\NormalTok{metacont)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## output +## r(98)=0.2 +## r(127)=0.13 +## r(183)=0.36 +## r(58)=0.35 +## r(144)=0.36 +## r(118)=0.08 +\end{verbatim} + +The function gives us \textbf{the correct format of our effect sizes +(t-values)} which we need to conduct the p-curve analysis. + +The function also \textbf{automatically stores a .txt-File (input.txt) +in your working directory folder on your computer}. + +If you forgot where your working directory is, use the \texttt{getwd} +function. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{getwd}\NormalTok{()} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] "/Users/Mathias2/Documents/R/WORKING_DIRECTORY/Windows/R/WORKING_DIRECTORY/Meta-Analyse Buch/bookdown-demo-master" +\end{verbatim} + +This tells you the path to your working directory, where the +\textbf{input.txt} file should be stored. + +\subsection{Creating P-Curves}\label{creating-p-curves} + +To create p-curves, we have to use the \texttt{pcurve\_app} function. +The code for this function is very long, so it is not displayed here. +The code for the function has been made publically available online by +\textbf{Uri Simonsohn, Leif Nelson, and Joseph Simmons} and can be found +\href{http://p-curve.com/app4/pcurve_app4.06.r}{here}. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +To use the \texttt{pcurve\_app} function, we have to specify that the +function should use the \textbf{input.txt} in which the data we created +using the \texttt{pcurve\_dataprep} function is stored. We also have to +provide the function with the \textbf{path to our folder/working +directory where the file is stored}. + +\begin{rmdachtung} +Please not that while the standard way that \textbf{Windows} provides +you with paths is using \textbf{backslashes}, we need to use +\textbf{normal slashes} (\textbf{/}) for our designated path. +\end{rmdachtung} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{pcurve_app}\NormalTok{(}\StringTok{"input.txt"}\NormalTok{,}\StringTok{"C://Users/Admin/Documents/R/WORKING_DIRECTORY/Meta-Analyse Buch/bookdown-demo-master"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +The function automatically \textbf{creates and stores several files and +figures into our folder/working directory}. Here the most important +ones: + +\textbf{Figure 1: ``input.png''} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-156-1.pdf} + +This figure shows you the \textbf{p-curve for your results (in blue)}. +On the bottom, you can also find the number of effect sizes with +\(p<0.05\) which were included in the analysis. There are two tests +displayed in the plot. + +\textbf{The test for right-skewness} + +If there is evidential value behind our data, the p-curve should be +\textbf{right-skewed}. Through eyeballing, we see that this is pretty +much the case here, and the \textbf{tests for the half and full p-curve} +are also both \textbf{significant} (\(p_{Full}<0.001, p_{Half}<0.001\)). +This means that the p-curve is heavily right-skewed, indicating that +\textbf{evidential value is present in our data} + +\textbf{The test for flatness} + +If there is evidential value behind our data, the p-curve should also +\textbf{not be flat}. Through eyeballing, we see that this is pretty +much the case here. The \textbf{tests for flatness} are both not +significant (\(p_{Full}=0.9887, p_{Binomial}=0.7459\)). + +\textbf{Figure 2: ``input\_fit.png''} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-157-1.pdf} + +This plot estimates the \textbf{power} behind our data, meaning if we +have sufficient studies with sufficient participants to find a true +effect if it exists. A conventional threshold for optimal power is +\textbf{80\%}, but P-curve can even assess evidential value if studies +are \textbf{underpowered}. In our case, the the power estimate is +\textbf{90\%}. + +\textbf{Figure 3: ``input\_cumulative.png''} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-158-1.pdf} + +This plot provides \textbf{sensitivity analyses} in which the +\textbf{highest and lowest p-values are dropped}. + +\subsection{\texorpdfstring{Estimating the ``true'' effect of our +data}{Estimating the true effect of our data}}\label{estimating-the-true-effect-of-our-data} + +To estimate the \textbf{true effect of our data} with p-curve (much like +the \protect\hyperlink{dant}{Duval \& Tweedie trim-and-fill procedure}), +we can use the \texttt{plotloss} function described and made openly +available by Simonsohn et al. \citep{simonsohn2014pb}. + +The code for this function is quite long, so it is not displayed here. +It can accessed using this +\href{https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R/blob/master/true_effect_estimation.R}{link}. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +For the \texttt{plotloss} function, we only have to provide the +\textbf{data} to be used to find the ``true'' effect size underlying the +data, and a \textbf{range of effect sizes in which the function should +search for the true effect}, delineated by \texttt{dmin} and +\texttt{dmax}. + +I will use my \texttt{metacont} data again here, and will search for the +true effect between \(d=0\) to \(d=1.0\). + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{plotloss}\NormalTok{(}\DataTypeTok{data=}\NormalTok{metacont,}\DataTypeTok{dmin=}\DecValTok{0}\NormalTok{,}\DataTypeTok{dmax=}\DecValTok{1}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-160-1.pdf} + +\begin{verbatim} +## NULL +\end{verbatim} + +The function provides an effect estimate of \(d=0.64\). + +\begin{rmdachtung} +It should be noted that this chapter should only be seen as \textbf{an +introduction into p-Curve}, which should not be seen as comprehensive. + +Simonsohn et al. {[}@simonsohn2015better{]} also stress that P-Curve +should only be used for \textbf{outcome data which was actually of +interest for the Authors of the specific article}, because those are the +one's likely to get p-hacked. They also ask meta-researchers to provide +a \textbf{detailed table in which the reported results of each outcome +data used in the p-Curve is documented} (a guide can be found +\href{http://p-curve.com/guide.pdf}{here}). + +It has also been shown that P-Curve's effect estimate are \textbf{not +robust when the heterogeneity of a meta-analyis is high} +(\emph{I}\textsuperscript{2} \textgreater{} 50\%). Van Aert et al. +{[}@van2016conducting{]} propose \textbf{not to determine the ``true'' +effect using P-Curve when heterogeneity is high} (defined as +I\textsuperscript{2} \textgreater{} 50\%). + +A poosible solution for this problem might be to \textbf{reduce the +overall heterogeneity using outlier removal}, or to p-Curve results in +\textbf{more homogeneous subgroups}. +\end{rmdachtung} + +\chapter{Creating a RevMan style Risk of Bias +summary}\label{creating-a-revman-style-risk-of-bias-summary} + +\begin{figure} +\centering +\includegraphics{robplot.png} +\caption{A finished Risk of Bias summary} +\end{figure} + +\section{Preparing your Risk of Bias +data}\label{preparing-your-risk-of-bias-data} + +In many Meta-Analyses, you will also want to have a look at the quality +of included studies using the +\href{https://handbook-5-1.cochrane.org/chapter_8/8_6_presentation_of_assessments_of_risk_of_bias.htm}{\textbf{RevMan +Risk of Bias assessment tool}}. However, many researchers don't use +\textbf{RevMan} to conduct Meta-Analyses, one has to put some extra +effort into inserting all study quality data by hand into RevMan, which +is extremely time-consuming. + +Furthermore, the quality of the Risk of Bias (RoB) summary figures in +RevMan are of suboptimal picture quality. Many journals will require +figures with better quality, or figures saved in another format (such as +\textbf{.svg} or \textbf{.pdf}). + +\begin{figure} +\centering +\includegraphics{robsummaryrevman.png} +\caption{This is the output created by RevMan} +\end{figure} + +To avoid all of this, you can easily plot the \textbf{RoB Summary in +RStudio yourself}. To do this, we again have to prepare an EXCEL sheet +in which we store our RoB data. In +\protect\hyperlink{excel_preparation}{Chapter 3.1.1}, we already +described how the preparation and import of EXCEL sheets into RStudio +works in general. For this data sheet, you need to follow a few +guidelines: + +\begin{itemize} +\tightlist +\item + Name the first column \textbf{Author}, and put all the study names in + this column (e.g., Frogeli et al.) +\item + Give the other \textbf{columns a name signifying the RoB criterion} + you assessed. Do this for all criteria you want to have included in + your plot. \textbf{Important}: when naming your column, do not use + spaces between word, but use underscores or points instead + (e.g.~allocation\_concealment) +\item + In these columns, you have to describe if the study received a + \textbf{High}, \textbf{Low}, or \textbf{Unclear} risk of bias rating. + Use exactly these codes for your data, including upper and lowercase + (R is case sensitive) +\item + Do \textbf{not} store any other information in your data + (e.g.~commentaries on your RoB decision) +\end{itemize} + +\section{Plotting the summary}\label{plotting-the-summary} + +To plot the summary, we have to import our dataset first. We describe +how to do this in \protect\hyperlink{import_excel}{Chapter 3.2}. I +simply called my dataset \texttt{rob}. + +Let's have a look at the structure of the data first: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{str}\NormalTok{(rob)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Classes 'tbl_df', 'tbl' and 'data.frame': 43 obs. of 13 variables: +## $ RoB_sequence : chr "Low" "Low" "Low" "Low" ... +## $ RoB_allocation_concealment_ : chr "Low" "Low" "Low" "Low" ... +## $ RoB_Blinding of participants and personnel : chr "High" "High" "High" "High" ... +## $ RoB_Blinding of outcome assessors : chr "Low" "Low" "Low" "Low" ... +## $ RoB_Incomplete outcome data for all outcomes: chr "Low" "Low" "Low" "Low" ... +## $ RoB_Selective outcome reporting : chr "Low" "Unclear" "Low" "Low" ... +## $ RoB_Co-interventions : chr "Unclear" "Low" "Low" "Low" ... +## $ RoB_Serious_flaw : chr "Low" "Low" "Low" "Low" ... +## $ RoB_ITT : chr "Low" "Low" "Low" "Low" ... +## $ RoB_SimilarGroups : chr "Unclear" "Low" "Low" "Low" ... +## $ RoB_compliance : chr "Low" "Unclear" "High" "High" ... +## $ RoB_identical_post_timing : chr "Low" "Low" "Low" "Low" ... +## $ Author : chr "BiesheuvelLeliefeld 2017" "Bockting 2005 & 2010 & 2015" "Bockting 2018" "Bockting 2018" ... +\end{verbatim} + +We can see that we have the data imported in RStudio now, with ratings +for every criterion in each column. If you named your columns +differently, or used less or more criteria, this is not that important +now. We will get to this later. + +We will plot the summary using the packages \texttt{ggplot2} and +\texttt{tidyr}. They should be installed as part of the +\texttt{tidyverse}, but be sure to have it on your computer, and then +load them from your \textbf{library}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(ggplot2)} +\KeywordTok{library}\NormalTok{(tidyr)} +\end{Highlighting} +\end{Shaded} + +We have prepared a function called \texttt{rob.summary} for you, which +**automatically plots the risk of bias summary for your prepared data +set. The code for this function can be seen below. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{rob.summary<-}\ControlFlowTok{function}\NormalTok{(data)\{} +\NormalTok{ rob.vars<-}\KeywordTok{data.frame}\NormalTok{(data)} +\NormalTok{ rob.vars}\OperatorTok{$}\NormalTok{Author<-}\OtherTok{NULL} +\NormalTok{ ncol.rob.vars<-}\KeywordTok{ncol}\NormalTok{(rob.vars)} +\NormalTok{ last<-}\KeywordTok{colnames}\NormalTok{(rob.vars[ncol.rob.vars])} +\NormalTok{ first<-}\KeywordTok{colnames}\NormalTok{(rob.vars[}\DecValTok{1}\NormalTok{])} +\NormalTok{ rob.long <-}\StringTok{ }\KeywordTok{gather}\NormalTok{(data,} +\NormalTok{ condition, measurement,} +\NormalTok{ first}\OperatorTok{:}\NormalTok{last,} + \DataTypeTok{factor_key=}\OtherTok{TRUE}\NormalTok{)} +\NormalTok{ rob.long}\OperatorTok{$}\NormalTok{measurement<-}\KeywordTok{as.factor}\NormalTok{(rob.long}\OperatorTok{$}\NormalTok{measurement)} +\NormalTok{ rob.long}\OperatorTok{$}\NormalTok{measurement<-}\KeywordTok{factor}\NormalTok{(rob.long}\OperatorTok{$}\NormalTok{measurement,} + \KeywordTok{levels}\NormalTok{(rob.long}\OperatorTok{$}\NormalTok{measurement)[}\KeywordTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{3}\NormalTok{,}\DecValTok{2}\NormalTok{)])} +\NormalTok{ rob.plot<-}\KeywordTok{ggplot}\NormalTok{(}\DataTypeTok{data=}\NormalTok{rob.long)}\OperatorTok{+} +\StringTok{ }\KeywordTok{geom_bar}\NormalTok{(}\DataTypeTok{mapping=}\KeywordTok{aes}\NormalTok{(}\DataTypeTok{x=}\NormalTok{condition,}\DataTypeTok{fill=}\NormalTok{measurement),} + \DataTypeTok{width=}\FloatTok{0.7}\NormalTok{,} + \DataTypeTok{position =} \StringTok{"fill"}\NormalTok{,} + \DataTypeTok{color=}\StringTok{"black"}\NormalTok{)}\OperatorTok{+} +\StringTok{ }\KeywordTok{coord_flip}\NormalTok{(}\DataTypeTok{ylim =} \KeywordTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{))}\OperatorTok{+} +\StringTok{ }\KeywordTok{guides}\NormalTok{(}\DataTypeTok{fill =} \KeywordTok{guide_legend}\NormalTok{(}\DataTypeTok{reverse =} \OtherTok{TRUE}\NormalTok{))}\OperatorTok{+} +\StringTok{ }\KeywordTok{scale_fill_manual}\NormalTok{(}\StringTok{"Risk of Bias"}\NormalTok{,} + \DataTypeTok{labels =} \KeywordTok{c}\NormalTok{(}\StringTok{" High risk of bias "}\NormalTok{,} + \StringTok{" Unclear risk of bias "}\NormalTok{,} + \StringTok{" Low risk of bias "}\NormalTok{),} + \DataTypeTok{values =} \KeywordTok{c}\NormalTok{(}\StringTok{"Unclear"}\NormalTok{ =}\StringTok{ "#E2DF07"}\NormalTok{,} + \StringTok{"High"}\NormalTok{ =}\StringTok{ "#BF0000"}\NormalTok{,} + \StringTok{"Low"}\NormalTok{ =}\StringTok{ "#02C100"}\NormalTok{))}\OperatorTok{+} +\StringTok{ }\KeywordTok{scale_y_continuous}\NormalTok{(}\DataTypeTok{labels =}\NormalTok{ scales}\OperatorTok{::}\NormalTok{percent)}\OperatorTok{+} +\StringTok{ }\KeywordTok{theme}\NormalTok{(}\DataTypeTok{axis.title.x=}\KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{axis.title.y=}\KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{axis.ticks.y=}\KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{axis.text.y =} \KeywordTok{element_text}\NormalTok{(}\DataTypeTok{size=}\DecValTok{18}\NormalTok{, }\DataTypeTok{color =} \StringTok{"black"}\NormalTok{),} + \DataTypeTok{axis.line.x =} \KeywordTok{element_line}\NormalTok{(}\DataTypeTok{colour =} \StringTok{"black"}\NormalTok{,} + \DataTypeTok{size =} \FloatTok{0.5}\NormalTok{, }\DataTypeTok{linetype =} \StringTok{"solid"}\NormalTok{),} + \DataTypeTok{legend.position =} \StringTok{"bottom"}\NormalTok{,} + \DataTypeTok{panel.grid.major =} \KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{panel.grid.minor =} \KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{panel.background =} \KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{legend.background =} \KeywordTok{element_rect}\NormalTok{(}\DataTypeTok{linetype=}\StringTok{"solid"}\NormalTok{,} + \DataTypeTok{colour =}\StringTok{"black"}\NormalTok{),} + \DataTypeTok{legend.title =} \KeywordTok{element_blank}\NormalTok{(),} + \DataTypeTok{legend.key.size =} \KeywordTok{unit}\NormalTok{(}\FloatTok{0.75}\NormalTok{,}\StringTok{"cm"}\NormalTok{),} + \DataTypeTok{legend.text=}\KeywordTok{element_text}\NormalTok{(}\DataTypeTok{size=}\DecValTok{14}\NormalTok{))} +\KeywordTok{return}\NormalTok{(rob.plot)} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +To use the \texttt{rob.summary}, we have to specify our RoB data set in +the function. In my case, this is \texttt{rob}. I will call my output +\texttt{plot}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{plot<-}\KeywordTok{rob.summary}\NormalTok{(}\DataTypeTok{data=}\NormalTok{rob)} +\NormalTok{plot} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-166-1.pdf} + +We see that the \texttt{rob.summary} function has already +\textbf{created a RoB summary plot}, but there's something missing. The +labels for each RoB criterion on the left are the \textbf{raw names} we +gave each column, so we should change them to their actual name. To do +this, we have to attach the correct labels for each criterion to our +\texttt{plot} object. + +To do this, we have to attach \texttt{+} a function which describes +which label should be replaced with which name. A generic version looks +like this: + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{new.plot<-plot}\OperatorTok{+} +\StringTok{ }\KeywordTok{scale_x_discrete}\NormalTok{(}\DataTypeTok{labels=}\KeywordTok{c}\NormalTok{(}\StringTok{"former_criterion_name_1"}\NormalTok{ =}\StringTok{ "New Criterion Name 1"}\NormalTok{, } + \StringTok{"former_criterion_name_2"}\NormalTok{ =}\StringTok{ "New Criterion Name 2"}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +Just define \textbf{all} the criterion labels this was way, and do not +forget to separate them with a comma within the bracket (except for the +last one). + +If i do this for my data, it looks like this: + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{new.plot<-plot}\OperatorTok{+}\KeywordTok{scale_x_discrete}\NormalTok{(}\DataTypeTok{labels=}\KeywordTok{c}\NormalTok{(}\StringTok{"RoB_sequence"}\NormalTok{ =}\StringTok{ "Sequence generation"}\NormalTok{,} + \StringTok{"RoB_allocation_concealment_"}\NormalTok{ =}\StringTok{ "Allocation Concealment"}\NormalTok{,} + \StringTok{"RoB_Blinding of participants and personnel"}\NormalTok{ =}\StringTok{ "Blinding of participants and personnel"}\NormalTok{,} + \StringTok{"RoB_Blinding of outcome assessors"}\NormalTok{ =}\StringTok{ "Blinding of outcome assessors"}\NormalTok{,} + \StringTok{"RoB_Incomplete outcome data for all outcomes"}\NormalTok{ =}\StringTok{ "Incomplete outcome data"}\NormalTok{,} + \StringTok{"RoB_Selective outcome reporting"}\NormalTok{ =}\StringTok{ "Selective outcome reporting"}\NormalTok{,} + \StringTok{"RoB_Co-interventions"}\NormalTok{ =}\StringTok{ "Co-Interventions"}\NormalTok{,} + \StringTok{"RoB_Serious_flaw"}\NormalTok{ =}\StringTok{ "Serious flaw"}\NormalTok{,} + \StringTok{"RoB_ITT"}\NormalTok{ =}\StringTok{ "ITT Analyses"}\NormalTok{,} + \StringTok{"RoB_SimilarGroups"}\NormalTok{ =}\StringTok{ "Similar Groups"}\NormalTok{,} + \StringTok{"RoB_compliance"}\NormalTok{ =}\StringTok{ "Compliance"}\NormalTok{,} + \StringTok{"RoB_identical_post_timing"}\NormalTok{ =}\StringTok{ "Identical Post Timing"}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +We have not updated the labels in the \texttt{plot} object, which is now +called \texttt{new.plot}. Let's have a peek how it looks now: + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{new.plot} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-170-1.pdf} + +Looks good so far. + +\hypertarget{saving}{\section{Saving the Summary Plot}\label{saving}} + +I want to save the plot as a \textbf{PDF} file in my working directory. +To do this, define the name of the file as \texttt{rob\_summary.pdf}, +and save it at the same time in the correct size and orientation using +this code: + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{pdf}\NormalTok{(}\DataTypeTok{file=}\StringTok{'rob_summary.pdf'}\NormalTok{, }\DataTypeTok{width =} \FloatTok{11.69}\NormalTok{, }\DataTypeTok{height =} \FloatTok{8.27}\NormalTok{) } +\NormalTok{new.plot;}\KeywordTok{dev.off}\NormalTok{() } +\end{Highlighting} +\end{Shaded} + +I can also save the plot as \textbf{PNG} file using this command. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{png}\NormalTok{(}\DataTypeTok{file=}\StringTok{'rob_summary.png'}\NormalTok{, }\DataTypeTok{width =} \DecValTok{842}\NormalTok{, }\DataTypeTok{height =} \DecValTok{595}\NormalTok{) } +\NormalTok{new.plot;}\KeywordTok{dev.off}\NormalTok{() } +\end{Highlighting} +\end{Shaded} + +Or as a \textbf{Scalable Vector Graphic} (.svg) with this command. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{svg}\NormalTok{(}\DataTypeTok{file=}\StringTok{'rob_summary.svg'}\NormalTok{, }\DataTypeTok{width =} \FloatTok{11.69}\NormalTok{, }\DataTypeTok{height =} \FloatTok{8.27}\NormalTok{) } +\NormalTok{new.plot;}\KeywordTok{dev.off}\NormalTok{() } +\end{Highlighting} +\end{Shaded} + +\chapter{Network Meta-Analysis}\label{network-meta-analysis} + +\begin{figure} +\centering +\includegraphics{networks.jpg} +\caption{} +\end{figure} + +Often when doing meta-analysis on the effectiveness of certain +interventions, we are less interested if \textbf{one particular +intervention is effective} (e.g., because it is quite well established +that the intervention can be efficacious), but weather \textbf{one +intervention is more or less effective than another type of intervention +for some condition}. Yet, once we are interested in +\textbf{head-to-head} comparisons between two treatments, we often face +the problem that \textbf{only very few, if any, randomized controlled +trials have compared the effects of two interventions directly}. This +makes it very hard, if not impossible for us to conduct conventional +meta-analyses to answer questions on the comparative effects of two or +more interventions for one indication or outcome (e.g., different types +of psychotherapy for major depression). + +Nevertheless, while direct comparisons between two or more interventions +may often not exist, it is often the case that the interventions were +evaluated in separate randomized controlled trials in which the +intervention effects were compared to similar \textbf{control groups} +(e.g., waitlist control groups, or placebos). This means that we do have +\textbf{indirect comparisons} of the effects of different interventions, +because they were compared to the same control condition. +\textbf{Multiple-treatments meta-analysis (MTM)} is an extension of +conventional meta-analysis which allows us to \textbf{incorporate +indirect comparisons}, and thus the simultaneous analysis of several +interventions. + +These meta-analysis methods are also referred to as \textbf{network +meta-analyses}, as such methods allow for \textbf{multiple interventions +comparisons to be integrated into our analysis}, which can be formalized +as a \textbf{``network''} of comparisons. + +\begin{rmdinfo} +\textbf{The idea behind network meta-analysis} + +Let's assume you have the results of two randomized controlled trials. +One trial evaluated the effect of cognitive behavioral therapy (CBT) for +depression to a control group. The second trial evaluted the effects of +short-term psychodynamic therapy (STPP) on depression compared to +control. We know the effect size \(\hat\theta\) of both interventions +which was found in the trial compared to control at post-test. These +studies produce \textbf{indirect evidence} for the comparative effect of +CBT versus STPP {[}@schwarzer2015meta{]}: + +\[\hat\theta_{CBT vs. STPP}^{indirect}=\hat\theta_{CBT vs. Control}^{direct} -\hat\theta_{STPP vs.Control}^{direct} \] + +On the other hand, it may also be the case that we found one study +\textbf{in which the effects of CBT were directly compared to the ones +of STPP}. We will denote this effect as +\(\hat\theta_{CBT vs. STPP}^{direct}\). In network-meta-analysis, we +want to integrate the \textbf{direct} as well as the \textbf{indirect} +evidence to get the most precise effect estimate of the comparative +effects. + +According to Schwarzer et al. {[}@schwarzer2015meta{]}, there are two +conditions which have to be met to conduct network meta-analyses: + +\begin{itemize} +\tightlist +\item + The studies are \textbf{independent} +\item + The effect sizes are \textbf{consistent}. This means that effect sizes + of interventions comparisons we attain through direct evidence should + be similar to the one we get from indirect evidence (e.g., + \(\theta_{CBT vs. STPP}^{direct}=\theta_{CBT vs. STPP}^{indirect}\)). + Inconsistency, on the other hand, is + \(\theta_{CBT vs. STPP}^{direct}-\theta_{CBT vs. STPP}^{indirect0} \not= 0\). + Assessing and dealing with inconsistency is highly important in + network meta-analysis. +\end{itemize} + +Below, you can see a simple first network of the comparisons between the +control condition and the two interventions. We could also think of a +network where \textbf{some comparisons are simply not available}, as is +the case in the second network. +\end{rmdinfo} + +\begin{verbatim} +## Warning: package 'network' was built under R version 3.4.4 +\end{verbatim} + +\begin{verbatim} +## Warning: package 'GGally' was built under R version 3.4.4 +\end{verbatim} + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-175-1} + +} + +\caption{A simple network}\label{fig:unnamed-chunk-175} +\end{figure} + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-176-1} + +} + +\caption{A simple network with one missing comparison}\label{fig:unnamed-chunk-176} +\end{figure} + +\begin{rmdinfo} +\textbf{Work in progress} + +A full version of this Chapter will be available soon. +\end{rmdinfo} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(netmeta)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Warning: package 'netmeta' was built under R version 3.4.4 +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(plyr)} +\end{Highlighting} +\end{Shaded} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{netmetdata} +\end{Highlighting} +\end{Shaded} + +\begin{tabular}{l|r|r|l|l|l} +\hline + & TE & seTE & treat1 & treat2 & Author\\ +\hline +1 & -1.90 & 0.1414 & MCT & Waitlist & DeFronzo1995\\ +\hline +2 & -0.82 & 0.0992 & MCT & Waitlist & Lewin2007\\ +\hline +4 & -1.34 & 0.1435 & CBT & Waitlist & Davidson2007\\ +\hline +5 & -1.10 & 0.1141 & CBT & Waitlist & Wolffenbuttel1999\\ +\hline +6 & -1.30 & 0.1268 & MBCT & Waitlist & Kipnes2001\\ +\hline +7 & -0.77 & 0.1078 & CBT & Waitlist & Kerenyi2004\\ +\hline +8 & 0.16 & 0.0849 & MBCT & MCT & Hanefeld2004\\ +\hline +9 & 0.10 & 0.1831 & MBCT & CBT & Derosa2004\\ +\hline +10 & -1.30 & 0.1014 & CBT & Waitlist & Baksi2004\\ +\hline +11 & -1.09 & 0.2263 & CBT & Waitlist & Rosenstock2008\\ +\hline +12 & -1.50 & 0.1624 & CBT & Waitlist & Zhu2003\\ +\hline +13 & -0.14 & 0.2239 & CBT & MCT & Yang2003\\ +\hline +14 & -1.20 & 0.1436 & CBT & Placebo & Vongthavaravat2002\\ +\hline +15 & -0.40 & 0.1549 & Psychodynamic & Placebo & Oyama2008\\ +\hline +16 & -0.80 & 0.1432 & Psychodynamic & Waitlist & Costa1997\\ +\hline +17 & -0.57 & 0.1291 & System & Waitlist & Hermansen2007\\ +\hline +18 & -0.70 & 0.1273 & Gestalt & Waitlist & Garber2008\\ +\hline +19 & -0.37 & 0.1184 & MCT & Placebo & Alex1998\\ +\hline +20 & -0.74 & 0.1839 & Psychoanalysis & Waitlist & Johnston1994\\ +\hline +21 & -1.41 & 0.2235 & Psychoanalysis & Waitlist & Johnston1998a\\ +\hline +22 & 0.00 & 0.2339 & CBT & MCT & Kim2007\\ +\hline +23 & -0.68 & 0.2828 & Psychoanalysis & Waitlist & Johnston1998b\\ +\hline +24 & -0.40 & 0.4356 & MCT & Waitlist & Gonzalez-Ortiz2004\\ +\hline +25 & -0.23 & 0.3467 & ACT & Waitlist & Stucci1996\\ +\hline +26 & -1.01 & 0.1366 & ACT & Waitlist & Moulin2006\\ +\hline +27 & -1.20 & 0.3758 & MCT & Waitlist & Ebert2018\\ +\hline +28 & -1.00 & 0.4669 & Psychodynamic & Waitlist & Ebert2018\\ +\hline +29 & -0.20 & 0.3579 & MCT & Psychodynamic & Ebert2018\\ +\hline +\end{tabular} + +Results + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{netmet <-}\StringTok{ }\KeywordTok{netmeta}\NormalTok{(TE, seTE, treat1, treat2, }\DataTypeTok{studlab=}\KeywordTok{paste}\NormalTok{(Author), }\DataTypeTok{data=}\NormalTok{netmetdata, }\DataTypeTok{sm=}\StringTok{"SMD"}\NormalTok{,}\DataTypeTok{reference.group =} \StringTok{"Waitlist"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Warning: Note, treatments within a comparison have been re-sorted in +## increasing order. +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{netmet} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Original data (with adjusted standard errors for multi-arm studies): +## +## treat1 treat2 TE seTE seTE.adj +## DeFronzo1995 MCT Waitlist -1.9000 0.1414 0.1414 +## Lewin2007 MCT Waitlist -0.8200 0.0992 0.0992 +## Davidson2007 CBT Waitlist -1.3400 0.1435 0.1435 +## Wolffenbuttel1999 CBT Waitlist -1.1000 0.1141 0.1141 +## Kipnes2001 MBCT Waitlist -1.3000 0.1268 0.1268 +## Kerenyi2004 CBT Waitlist -0.7700 0.1078 0.1078 +## Hanefeld2004 MBCT MCT 0.1600 0.0849 0.0849 +## Derosa2004 CBT MBCT -0.1000 0.1831 0.1831 +## Baksi2004 CBT Waitlist -1.3000 0.1014 0.1014 +## Rosenstock2008 CBT Waitlist -1.0900 0.2263 0.2263 +## Zhu2003 CBT Waitlist -1.5000 0.1624 0.1624 +## Yang2003 CBT MCT -0.1400 0.2239 0.2239 +## Vongthavaravat2002 CBT Placebo -1.2000 0.1436 0.1436 +## Oyama2008 Placebo Psychodynamic 0.4000 0.1549 0.1549 +## Costa1997 Psychodynamic Waitlist -0.8000 0.1432 0.1432 +## Hermansen2007 System Waitlist -0.5700 0.1291 0.1291 +## Garber2008 Gestalt Waitlist -0.7000 0.1273 0.1273 +## Alex1998 MCT Placebo -0.3700 0.1184 0.1184 +## Johnston1994 Psychoanalysis Waitlist -0.7400 0.1839 0.1839 +## Johnston1998a Psychoanalysis Waitlist -1.4100 0.2235 0.2235 +## Kim2007 CBT MCT 0.0000 0.2339 0.2339 +## Johnston1998b Psychoanalysis Waitlist -0.6800 0.2828 0.2828 +## Gonzalez-Ortiz2004 MCT Waitlist -0.4000 0.4356 0.4356 +## Stucci1996 ACT Waitlist -0.2300 0.3467 0.3467 +## Moulin2006 ACT Waitlist -1.0100 0.1366 0.1366 +## Ebert2018 MCT Waitlist -1.2000 0.3758 0.4125 +## Ebert2018 Psychodynamic Waitlist -1.0000 0.4669 0.8242 +## Ebert2018 MCT Psychodynamic -0.2000 0.3579 0.3884 +## narms multiarm +## DeFronzo1995 2 +## Lewin2007 2 +## Davidson2007 2 +## Wolffenbuttel1999 2 +## Kipnes2001 2 +## Kerenyi2004 2 +## Hanefeld2004 2 +## Derosa2004 2 +## Baksi2004 2 +## Rosenstock2008 2 +## Zhu2003 2 +## Yang2003 2 +## Vongthavaravat2002 2 +## Oyama2008 2 +## Costa1997 2 +## Hermansen2007 2 +## Garber2008 2 +## Alex1998 2 +## Johnston1994 2 +## Johnston1998a 2 +## Kim2007 2 +## Johnston1998b 2 +## Gonzalez-Ortiz2004 2 +## Stucci1996 2 +## Moulin2006 2 +## Ebert2018 3 * +## Ebert2018 3 * +## Ebert2018 3 * +## +## Number of treatment arms (by study): +## narms +## Alex1998 2 +## Baksi2004 2 +## Costa1997 2 +## Davidson2007 2 +## DeFronzo1995 2 +## Derosa2004 2 +## Ebert2018 3 +## Garber2008 2 +## Gonzalez-Ortiz2004 2 +## Hanefeld2004 2 +## Hermansen2007 2 +## Johnston1994 2 +## Johnston1998a 2 +## Johnston1998b 2 +## Kerenyi2004 2 +## Kim2007 2 +## Kipnes2001 2 +## Lewin2007 2 +## Moulin2006 2 +## Oyama2008 2 +## Rosenstock2008 2 +## Stucci1996 2 +## Vongthavaravat2002 2 +## Wolffenbuttel1999 2 +## Yang2003 2 +## Zhu2003 2 +## +## Results (fixed effect model): +## +## treat1 treat2 SMD 95%-CI +## DeFronzo1995 MCT Waitlist -1.1141 [-1.2309; -0.9973] +## Lewin2007 MCT Waitlist -1.1141 [-1.2309; -0.9973] +## Davidson2007 CBT Waitlist -1.2018 [-1.2953; -1.1084] +## Wolffenbuttel1999 CBT Waitlist -1.2018 [-1.2953; -1.1084] +## Kipnes2001 MBCT Waitlist -1.0664 [-1.2151; -0.9178] +## Kerenyi2004 CBT Waitlist -1.2018 [-1.2953; -1.1084] +## Hanefeld2004 MBCT MCT 0.0477 [-0.0891; 0.1845] +## Derosa2004 CBT MBCT -0.1354 [-0.2957; 0.0249] +## Baksi2004 CBT Waitlist -1.2018 [-1.2953; -1.1084] +## Rosenstock2008 CBT Waitlist -1.2018 [-1.2953; -1.1084] +## Zhu2003 CBT Waitlist -1.2018 [-1.2953; -1.1084] +## Yang2003 CBT MCT -0.0877 [-0.2203; 0.0449] +## Vongthavaravat2002 CBT Placebo -0.7623 [-0.9427; -0.5820] +## Oyama2008 Placebo Psychodynamic 0.3879 [ 0.1662; 0.6095] +## Costa1997 Psychodynamic Waitlist -0.8274 [-1.0401; -0.6147] +## Hermansen2007 System Waitlist -0.5700 [-0.8230; -0.3170] +## Garber2008 Gestalt Waitlist -0.7000 [-0.9495; -0.4505] +## Alex1998 MCT Placebo -0.6746 [-0.8482; -0.5011] +## Johnston1994 Psychoanalysis Waitlist -0.9439 [-1.1927; -0.6952] +## Johnston1998a Psychoanalysis Waitlist -0.9439 [-1.1927; -0.6952] +## Kim2007 CBT MCT -0.0877 [-0.2203; 0.0449] +## Johnston1998b Psychoanalysis Waitlist -0.9439 [-1.1927; -0.6952] +## Gonzalez-Ortiz2004 MCT Waitlist -1.1141 [-1.2309; -0.9973] +## Stucci1996 ACT Waitlist -0.9052 [-1.1543; -0.6561] +## Moulin2006 ACT Waitlist -0.9052 [-1.1543; -0.6561] +## Ebert2018 MCT Waitlist -1.1141 [-1.2309; -0.9973] +## Ebert2018 Psychodynamic Waitlist -0.8274 [-1.0401; -0.6147] +## Ebert2018 MCT Psychodynamic -0.2867 [-0.5113; -0.0622] +## Q leverage +## DeFronzo1995 30.89 0.18 +## Lewin2007 8.79 0.36 +## Davidson2007 0.93 0.11 +## Wolffenbuttel1999 0.80 0.17 +## Kipnes2001 3.39 0.36 +## Kerenyi2004 16.05 0.20 +## Hanefeld2004 1.75 0.68 +## Derosa2004 0.04 0.20 +## Baksi2004 0.94 0.22 +## Rosenstock2008 0.24 0.04 +## Zhu2003 3.37 0.09 +## Yang2003 0.05 0.09 +## Vongthavaravat2002 9.29 0.41 +## Oyama2008 0.01 0.53 +## Costa1997 0.04 0.57 +## Hermansen2007 0.00 1.00 +## Garber2008 0.00 1.00 +## Alex1998 6.62 0.56 +## Johnston1994 1.23 0.48 +## Johnston1998a 4.35 0.32 +## Kim2007 0.14 0.08 +## Johnston1998b 0.87 0.20 +## Gonzalez-Ortiz2004 2.69 0.02 +## Stucci1996 3.79 0.13 +## Moulin2006 0.59 0.87 +## Ebert2018 0.04 0.02 +## Ebert2018 0.04 0.02 +## Ebert2018 0.05 0.09 +## +## Results (random effects model): +## +## treat1 treat2 SMD 95%-CI +## DeFronzo1995 MCT Waitlist -1.1268 [-1.4291; -0.8244] +## Lewin2007 MCT Waitlist -1.1268 [-1.4291; -0.8244] +## Davidson2007 CBT Waitlist -1.2335 [-1.4839; -0.9830] +## Wolffenbuttel1999 CBT Waitlist -1.2335 [-1.4839; -0.9830] +## Kipnes2001 MBCT Waitlist -1.1291 [-1.5596; -0.6986] +## Kerenyi2004 CBT Waitlist -1.2335 [-1.4839; -0.9830] +## Hanefeld2004 MBCT MCT -0.0023 [-0.4444; 0.4398] +## Derosa2004 CBT MBCT -0.1044 [-0.5435; 0.3347] +## Baksi2004 CBT Waitlist -1.2335 [-1.4839; -0.9830] +## Rosenstock2008 CBT Waitlist -1.2335 [-1.4839; -0.9830] +## Zhu2003 CBT Waitlist -1.2335 [-1.4839; -0.9830] +## Yang2003 CBT MCT -0.1067 [-0.4304; 0.2170] +## Vongthavaravat2002 CBT Placebo -0.8169 [-1.2817; -0.3521] +## Oyama2008 Placebo Psychodynamic 0.4252 [-0.0951; 0.9456] +## Costa1997 Psychodynamic Waitlist -0.8418 [-1.3236; -0.3600] +## Hermansen2007 System Waitlist -0.5700 [-1.2640; 0.1240] +## Garber2008 Gestalt Waitlist -0.7000 [-1.3927; -0.0073] +## Alex1998 MCT Placebo -0.7102 [-1.1713; -0.2491] +## Johnston1994 Psychoanalysis Waitlist -0.9497 [-1.4040; -0.4955] +## Johnston1998a Psychoanalysis Waitlist -0.9497 [-1.4040; -0.4955] +## Kim2007 CBT MCT -0.1067 [-0.4304; 0.2170] +## Johnston1998b Psychoanalysis Waitlist -0.9497 [-1.4040; -0.4955] +## Gonzalez-Ortiz2004 MCT Waitlist -1.1268 [-1.4291; -0.8244] +## Stucci1996 ACT Waitlist -0.7311 [-1.2918; -0.1705] +## Moulin2006 ACT Waitlist -0.7311 [-1.2918; -0.1705] +## Ebert2018 MCT Waitlist -1.1268 [-1.4291; -0.8244] +## Ebert2018 Psychodynamic Waitlist -0.8418 [-1.3236; -0.3600] +## Ebert2018 MCT Psychodynamic -0.2850 [-0.7908; 0.2208] +## +## Number of studies: k = 26 +## Number of treatments: n = 10 +## Number of pairwise comparisons: m = 28 +## Number of designs: d = 15 +## +## Fixed effect model +## +## Treatment estimate (sm = 'SMD', comparison: other treatments vs 'Waitlist'): +## SMD 95%-CI +## ACT -0.9052 [-1.1543; -0.6561] +## CBT -1.2018 [-1.2953; -1.1084] +## Gestalt -0.7000 [-0.9495; -0.4505] +## MBCT -1.0664 [-1.2151; -0.9178] +## MCT -1.1141 [-1.2309; -0.9973] +## Placebo -0.4395 [-0.6188; -0.2602] +## Psychoanalysis -0.9439 [-1.1927; -0.6952] +## Psychodynamic -0.8274 [-1.0401; -0.6147] +## System -0.5700 [-0.8230; -0.3170] +## Waitlist . . +## +## Random effects model +## +## Treatment estimate (sm = 'SMD', comparison: other treatments vs 'Waitlist'): +## SMD 95%-CI +## ACT -0.7311 [-1.2918; -0.1705] +## CBT -1.2335 [-1.4839; -0.9830] +## Gestalt -0.7000 [-1.3927; -0.0073] +## MBCT -1.1291 [-1.5596; -0.6986] +## MCT -1.1268 [-1.4291; -0.8244] +## Placebo -0.4166 [-0.8887; 0.0556] +## Psychoanalysis -0.9497 [-1.4040; -0.4955] +## Psychodynamic -0.8418 [-1.3236; -0.3600] +## System -0.5700 [-1.2640; 0.1240] +## Waitlist . . +## +## Quantifying heterogeneity / inconsistency: +## tau^2 = 0.1087; I^2 = 81.4% +## +## Tests of heterogeneity (within designs) and inconsistency (between designs): +## Q d.f. p-value +## Total 96.99 18 < 0.0001 +## Within designs 74.46 11 < 0.0001 +## Between designs 22.53 7 0.0021 +\end{verbatim} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{netgraph}\NormalTok{(netmet,}\DataTypeTok{seq =} \KeywordTok{c}\NormalTok{(}\StringTok{"Waitlist"}\NormalTok{,}\StringTok{"Gestalt"}\NormalTok{,}\StringTok{"Psychodynamic"}\NormalTok{,}\StringTok{"MCT"}\NormalTok{,}\StringTok{"MBCT"}\NormalTok{,}\StringTok{"Placebo"}\NormalTok{,}\StringTok{"CBT"}\NormalTok{,}\StringTok{"System"}\NormalTok{,}\StringTok{"Psychoanalysis"}\NormalTok{,}\StringTok{"ACT"}\NormalTok{))} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-183-1.pdf} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{netgraph}\NormalTok{(netmet, }\DataTypeTok{start=}\StringTok{"circle"}\NormalTok{, }\DataTypeTok{iterate=}\OtherTok{TRUE}\NormalTok{, }\DataTypeTok{col=}\StringTok{"darkgray"}\NormalTok{, }\DataTypeTok{cex=}\FloatTok{1.5}\NormalTok{, }\DataTypeTok{multiarm=}\OtherTok{TRUE}\NormalTok{, }\DataTypeTok{points=}\OtherTok{TRUE}\NormalTok{, }\DataTypeTok{col.points=}\StringTok{"blue"}\NormalTok{, }\DataTypeTok{cex.points=}\DecValTok{3}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-183-2.pdf} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{forest}\NormalTok{(netmet, }\DataTypeTok{xlim=}\KeywordTok{c}\NormalTok{(}\OperatorTok{-}\FloatTok{1.5}\NormalTok{, }\FloatTok{0.5}\NormalTok{), }\DataTypeTok{ref=}\StringTok{"Waitlist"}\NormalTok{, }\DataTypeTok{leftlabs=}\StringTok{"Contrast to Waitlist"}\NormalTok{, }\DataTypeTok{xlab=}\StringTok{"Effect on Depression (SMD)"}\NormalTok{,}\DataTypeTok{sortvar =}\NormalTok{ TE, }\DataTypeTok{smlab =} \StringTok{""}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-184-1.pdf} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{forest}\NormalTok{(netmet, }\DataTypeTok{xlim=}\KeywordTok{c}\NormalTok{(}\OperatorTok{-}\FloatTok{1.5}\NormalTok{, }\FloatTok{0.5}\NormalTok{), }\DataTypeTok{ref=}\StringTok{"Placebo"}\NormalTok{, }\DataTypeTok{leftlabs=}\StringTok{"Contrast to Placebo"}\NormalTok{, }\DataTypeTok{xlab=}\StringTok{"Effect on Depression (SMD)"}\NormalTok{,}\DataTypeTok{sortvar =}\NormalTok{ TE, }\DataTypeTok{smlab =} \StringTok{""}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-184-2.pdf} + +\chapter{Effect size calculators}\label{effect-size-calculators} + +\begin{figure} +\centering +\includegraphics{effect.jpg} +\caption{} +\end{figure} + +Although the \texttt{meta} package can calculate all \textbf{individual +effect sizes for every study} if we use the \texttt{metabin} or +\texttt{metacont} function, a frequent scenario is that \textbf{some +papers do not report the effect size data in the right format}. +Especially older articles may often only report results of +\(t\)\textbf{-tests}, \textbf{ANOVAs}, or \(\chi^2\)\textbf{-tests}. If +enough data is reported, we can also use \textbf{such outcome formats to +calculate effect sizes}. This way, we can calculate the \textbf{effect +size (e.g., Hedges' g)} and the \textbf{Standard Error (SE)}, which we +can then use in a meta-analysis with \textbf{pre-calculated effect +sizes} using the \texttt{metagen} function (see +\protect\hyperlink{pre.calc}{Chapter 4.1.1}). + +\begin{rmdinfo} +\textbf{Hedges' g} + +When dealing with \textbf{continuous outcome data}, it is conventional +to calculate the \textbf{Standardized Mean Difference} (SMD) as an +outcome for each study, and as your \textbf{summary measure} +{[}@borenstein2011{]}. + +A common format to to calculate the SMD in single trials is +\textbf{Cohen's d} {[}@cohen1988statistical{]}. Yet, this summary +measure has been \textbf{shown to have a slight bias in small studies, +for which it overestimates the effect} {[}@hedges1981distribution{]}. + +\textbf{Hedges \emph{g} } is a similar summary measure, but it +\textbf{controls for this bias}. It uses a slightly different formula to +calculate the pooled variance \(s_{pooled}\), \(s*_{pooled}\). The +transformation from \emph{d} to \emph{g} is often performed using the +formula by Hedges and Olkin {[}@hedges1985statistical{]}. + +\[g \simeq d\times(1-\frac{3}{4(n_1+n_2)-9}) \] +\end{rmdinfo} + +\begin{rmdachtung} +Hedges' g is \textbf{commonly used in meta-analysis}, and it's the +standard output format in \textbf{RevMan}. Therefore, we highly +recommend that you also use this measure in you meta-analysis. + +In \texttt{meta}`s \texttt{metabin} and \texttt{metacont} function, +Hedges' g is automatically calculated for each study if we set +\texttt{sm="SMD"}. If you use the \texttt{metgen} function, however, you +should calculate Hedges' g for each study yourself first. +\end{rmdachtung} + +To calculate the effect sizes, we will use Daniel Lüdecke's extremely +helpful \texttt{esc} package \citep{esc}. So, please \textbf{install +this package first} using the \texttt{install.packages("esc")} command, +and then load it in you library. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{library}\NormalTok{(esc)} +\end{Highlighting} +\end{Shaded} + +\textbf{Here's an overview of all calculators covered in this guide} + +\begin{enumerate} +\def\labelenumi{\arabic{enumi}.} +\tightlist +\item + \protect\hyperlink{a}{Calculating Hedges' \emph{g} from the Mean and + SD} +\item + \protect\hyperlink{b}{Calculating Hedges' \emph{g} from a regression + coefficient} +\item + \protect\hyperlink{c}{Calculating an Odd's Ratio from + \emph{Chi-square}} +\item + \protect\hyperlink{d}{Calculating Hedges' \emph{g} from a one-way + ANOVA} +\item + \protect\hyperlink{e}{Calculating Hedges' \emph{g} from the Mean and + SE} +\item + \protect\hyperlink{f}{Calculating Hedges' \emph{g} from a correlation} +\item + \protect\hyperlink{g}{Calculating Hedges' \emph{g} from an independent + t-test} +\item + \protect\hyperlink{h}{Calculating Hedges' \emph{g} from Cohen's + \emph{d}} +\end{enumerate} + +\hypertarget{a}{\section{Calculating Hedges' g from the Mean and +SD}\label{a}} + +To calculate Hedges' \emph{g} from the \emph{Mean}, \emph{Standard +Deviation}, and \(n_{group}\) of both trial arms, we can use the +\texttt{esc\_mean\_sd} function with the following parameters. + +\begin{itemize} +\tightlist +\item + \texttt{grp1m}: The \textbf{mean} of the \textbf{first group} (e.g., + the intervention). +\item + \texttt{grp1sd}: The \textbf{standard deviation} of the \textbf{first + group}. +\item + \texttt{grp1n}: The \textbf{sample size} of the \textbf{first group}. +\item + \texttt{grp2m}: The \textbf{mean} of the \textbf{second group}. +\item + \texttt{grp2sd}: The \textbf{standard deviation} of the \textbf{second + group}. +\item + \texttt{grp2n}: The \textbf{sample size} of the \textbf{second group}. +\item + \texttt{totalsd}: The \textbf{full sample standard deviation}, if the + standard deviation for each trial arm is not reported +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_mean_sd}\NormalTok{(}\DataTypeTok{grp1m =} \FloatTok{10.3}\NormalTok{, }\DataTypeTok{grp1sd =} \FloatTok{2.5}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{60}\NormalTok{,} +\DataTypeTok{grp2m =} \FloatTok{12.3}\NormalTok{, }\DataTypeTok{grp2sd =} \FloatTok{3.1}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{56}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: mean and sd to effect size Hedges' g +## Effect Size: -0.7082 +## Standard Error: 0.1916 +## Variance: 0.0367 +## Lower CI: -1.0837 +## Upper CI: -0.3326 +## Weight: 27.2374 +\end{verbatim} + +\hypertarget{b}{\section{\texorpdfstring{Calculating Hedges' \emph{g} +from a regression +coefficient}{Calculating Hedges' g from a regression coefficient}}\label{b}} + +\subsection{Unstandardized regression +coefficients}\label{unstandardized-regression-coefficients} + +It is also possible to calculate \textbf{Hedges' \emph{g} } from an +unstandardized or standardized regression coeffiecent +\citep{lipsey2001practical}. + +For \textbf{unstardardized coefficients}, we can use the \texttt{esc\_B} +function with the following parameters: + +\begin{itemize} +\tightlist +\item + \texttt{b}: unstandardized coefficient \(b\) (the ``treatment'' + predictor). +\item + \texttt{sdy}: the standard deviation of the dependent variable \(y\) + (i.e., the outcome). +\item + \texttt{grp1n}: the number of participants in the first group. +\item + \texttt{grp2n}: the number of participants in the second group. +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_B}\NormalTok{(}\DataTypeTok{b=}\FloatTok{3.3}\NormalTok{,}\DataTypeTok{sdy=}\DecValTok{5}\NormalTok{,}\DataTypeTok{grp1n =} \DecValTok{100}\NormalTok{,}\DataTypeTok{grp2n =} \DecValTok{150}\NormalTok{,}\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: unstandardized regression coefficient to effect size Hedges' g +## Effect Size: 0.6941 +## Standard Error: 0.1328 +## Variance: 0.0176 +## Lower CI: 0.4338 +## Upper CI: 0.9544 +## Weight: 56.7018 +\end{verbatim} + +\subsection{Standardized regression +coefficents}\label{standardized-regression-coefficents} + +Here, we can use the \texttt{esc\_beta} function with the follwing +parameters: + +\begin{itemize} +\tightlist +\item + \texttt{beta}: standardized coefficient \(\beta\) (the ``treatment'' + predictor). +\item + \texttt{sdy}: the standard deviation of the dependent variable \(y\) + (i.e., the outcome). +\item + \texttt{grp1n}: the number of participants in the first group. +\item + \texttt{grp2n}: the number of participants in the second group. +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_beta}\NormalTok{(}\DataTypeTok{beta=}\FloatTok{0.7}\NormalTok{, }\DataTypeTok{sdy=}\DecValTok{3}\NormalTok{, }\DataTypeTok{grp1n=}\DecValTok{100}\NormalTok{, }\DataTypeTok{grp2n=}\DecValTok{150}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: standardized regression coefficient to effect size Hedges' g +## Effect Size: 1.9868 +## Standard Error: 0.1569 +## Variance: 0.0246 +## Lower CI: 1.6793 +## Upper CI: 2.2942 +## Weight: 40.6353 +\end{verbatim} + +\hypertarget{c}{\section{\texorpdfstring{Calculating an Odd's Ratio from +\emph{Chi-square}}{Calculating an Odd's Ratio from Chi-square}}\label{c}} + +To calculate the \textbf{Odd's Ratio} (or any other kind of effect size +measure) from \(\chi^2\) using the \texttt{esc\_chisq} function with the +following paramters: + +\begin{itemize} +\tightlist +\item + \texttt{chisq}: The value of Chi-squared (or only \texttt{p}) +\item + \texttt{p}: the chi squared p or phi value (or only \texttt{chisq}) +\item + \texttt{totaln}: total sample size +\item + \texttt{es.type}: the summary measure (in our case, \texttt{"cox.or"}) +\end{itemize} + + \textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_chisq}\NormalTok{(}\DataTypeTok{chisq=}\FloatTok{9.9}\NormalTok{,}\DataTypeTok{totaln=}\DecValTok{100}\NormalTok{,}\DataTypeTok{es.type=}\StringTok{"cox.or"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: chi-squared-value to effect size Cox odds ratios +## Effect Size: 2.9858 +## Standard Error: 0.3478 +## Variance: 0.1210 +## Lower CI: 1.5101 +## Upper CI: 5.9036 +## Weight: 8.2667 +\end{verbatim} + +\hypertarget{d}{\section{\texorpdfstring{Calculating Hedges' \emph{g} +from a one-way +ANOVA}{Calculating Hedges' g from a one-way ANOVA}}\label{d}} + +We can also derive the SMD from the \(F\)-value of a \textbf{one-way +ANOVA with two groups}. Such ANOVAs can be detected if you look for the +\textbf{degrees of freedom} (\(df\)) underneath of \(F\). In a one-way +ANOVA with two groups, the degrees of freedom should always start with +\(1\) (e.g. \(F_{1,147}=5.31\)). The formula for this transformation +looks like this +\citep{cohen1992power, rosnow1996computing, rosnow2000contrasts}: + +\[d = \sqrt{ F(\frac{n_t+n_c}{n_t n_c})(\frac{n_t+n_c}{n_t+n_c-2})}\] + +To calculate \textbf{Hedges' g} from \(F\)-values, we can use the +\texttt{esc\_f} function with the following parameters: + +\begin{itemize} +\tightlist +\item + \texttt{f}: \emph{F}-value of the ANOVA +\item + \texttt{grp1n}: Number of participants in group 1 +\item + \texttt{grp2n}: Number of participants in group 2 +\item + \texttt{totaln}: The total number of participants (if the \emph{n} for + each group is not reported) +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_f}\NormalTok{(}\DataTypeTok{f=}\FloatTok{5.04}\NormalTok{,}\DataTypeTok{grp1n =} \DecValTok{519}\NormalTok{,}\DataTypeTok{grp2n =} \DecValTok{528}\NormalTok{,}\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: F-value (one-way-Anova) to effect size Hedges' g +## Effect Size: 0.1387 +## Standard Error: 0.0619 +## Variance: 0.0038 +## Lower CI: 0.0174 +## Upper CI: 0.2600 +## Weight: 261.1022 +\end{verbatim} + +\hypertarget{e}{\section{\texorpdfstring{Calculating Hedges' \emph{g} +from the Mean and +SE}{Calculating Hedges' g from the Mean and SE}}\label{e}} + +When calculating \textbf{Hedges' g} from the \textbf{Mean} and +\textbf{Standard Error}, we simply make use of the fact that the +Standard error is not much more than the \textbf{Standard Deviation} +when the sample size is taken into account +\citep{thalheimer2002calculate}: + +\[SD = SE\sqrt{n_c}\] + +We can calculate \textbf{Hedges' g} using the \texttt{esc\_mean} +function with the following parameters: + +\begin{itemize} +\tightlist +\item + \texttt{grp1m}: The mean of the first group. +\item + \texttt{grp1se}: The standard error of the first group. +\item + \texttt{grp1n}: The sample size of the first group. +\item + \texttt{grp2m}: The mean of the second group. +\item + \texttt{grp2se}: The standard error of the second group. +\item + \texttt{grp2n}: The sample size of the second group. +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_mean_se}\NormalTok{(}\DataTypeTok{grp1m =} \FloatTok{8.5}\NormalTok{, }\DataTypeTok{grp1se =} \FloatTok{1.5}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{50}\NormalTok{,} + \DataTypeTok{grp2m =} \DecValTok{11}\NormalTok{, }\DataTypeTok{grp2se =} \FloatTok{1.8}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{60}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: mean and se to effect size Hedges' g +## Effect Size: -0.1998 +## Standard Error: 0.1920 +## Variance: 0.0369 +## Lower CI: -0.5760 +## Upper CI: 0.1765 +## Weight: 27.1366 +\end{verbatim} + +\hypertarget{f}{\section{\texorpdfstring{Calculating Hedges' \emph{g} +from a correlation}{Calculating Hedges' g from a correlation}}\label{f}} + +For \textbf{equally sized groups} (\(n_1=n_2\)), we can use the +following formula to derive the SMD from the pointbiserial +\textbf{correlation} \citep{rosenthal1984meta}. + +\[r_{pb} = \frac{d}{\sqrt{d^2+4}}\] And this formula for +\textbf{unequally sized groups} \citep{aaron1998equating}: + +\[r_{pb} = \frac{d}{\sqrt{d^2+ \frac{(N^2-2 \times N)}{n_1 n_2} }}\] To +convert \(r_{pb}\) to \textbf{Hedges' g}, we can use the +\texttt{esc\_rpb} function with the following parameters: + +\begin{itemize} +\tightlist +\item + \texttt{r}: The \emph{r}-value. Either \emph{r} or its \emph{p}-value + must be given. +\item + \texttt{p}: The \emph{p}-value of the correlation. Either \emph{r} or + its \emph{p}-value must be given. +\item + \texttt{grp1n}: The sample size of group 1. +\item + \texttt{grp2n}: The sample size of group 2. +\item + \texttt{totaln}: Total sample size, if \texttt{grp1n} and + \texttt{grp2n} are not given. +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_rpb}\NormalTok{(}\DataTypeTok{r =} \FloatTok{0.25}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{99}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{120}\NormalTok{, }\DataTypeTok{es.type =} \StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: point-biserial r to effect size Hedges' g +## Effect Size: 0.5170 +## Standard Error: 0.1380 +## Variance: 0.0190 +## Lower CI: 0.2465 +## Upper CI: 0.7875 +## Weight: 52.4967 +\end{verbatim} + +\hypertarget{g}{\section{\texorpdfstring{Calculating Hedges' \emph{g} +from an independent +t-test}{Calculating Hedges' g from an independent t-test}}\label{g}} + +The SMD can also be derived from an \textbf{independent t-test value} +with the following formula \citep{thalheimer2002calculate}: + +\[d = \frac {t(n_1+n_2)}{\sqrt{(n_1+n_2-2)(n_1n_2)}}\] + +We can calculate \textbf{Hedges' g} from a \textbf{t-test} using the +\texttt{esc\_t} function with the following paramters: + +\begin{itemize} +\tightlist +\item + \texttt{t}: The t-value of the t-test. Either \emph{t} or its + \emph{p}-value must be given. +\item + \texttt{p}: The \emph{p}-value of the t-test. Either \emph{t} or its + \emph{p}-value must be given. +\item + \texttt{grp1n}: The sample size of group 1. +\item + \texttt{grp2n}: The sample size of group 2. +\item + \texttt{totaln}: Total sample size, if \texttt{grp1n} and + \texttt{grp2n} are not given. +\item + \texttt{es.type}: the \textbf{effect measure} we want to calculate. In + our case this is \texttt{"g"}. But we could also calculate Cohen's + \emph{d} using \texttt{"d"}. +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{esc_t}\NormalTok{(}\DataTypeTok{t =} \FloatTok{3.3}\NormalTok{, }\DataTypeTok{grp1n =} \DecValTok{100}\NormalTok{, }\DataTypeTok{grp2n =} \DecValTok{150}\NormalTok{,}\DataTypeTok{es.type=}\StringTok{"g"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## +## Effect Size Calculation for Meta Analysis +## +## Conversion: t-value to effect size Hedges' g +## Effect Size: 0.4247 +## Standard Error: 0.1305 +## Variance: 0.0170 +## Lower CI: 0.1690 +## Upper CI: 0.6805 +## Weight: 58.7211 +\end{verbatim} + +\hypertarget{h}{\section{\texorpdfstring{Calculating Hedges' \emph{g} +from Cohen's \emph{d}}{Calculating Hedges' g from Cohen's d}}\label{h}} + +We can also directly correct \textbf{Cohen's \emph{d} } and thus +generate \textbf{Hedges' g} using the formula by Hedges and Olkin +\citep{hedges1985statistical}: + +\[g \simeq d\times(1-\frac{3}{4(n_1+n_2)-9}) \] This can be done in R +using the \texttt{hedges\_g} function with the following parameters: + +\begin{itemize} +\tightlist +\item + \texttt{d}: The value of \textbf{Cohen's d} +\item + \texttt{totaln}: the total \emph{N} in the study +\end{itemize} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{hedges_g}\NormalTok{(}\DataTypeTok{d =} \FloatTok{0.75}\NormalTok{, }\DataTypeTok{totaln =} \DecValTok{50}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## [1] 0.7382199 +\end{verbatim} + +\section{Calculating effect sizes for studies with multiple +outcomes}\label{i} + +\begin{figure} +\centering +\includegraphics{stack.jpg} +\caption{} +\end{figure} + +Many randomized-controlled trials do not only include a single +\textbf{intervention} and \textbf{control group}, but compare the effect +of \textbf{two or more interventions} to a control group. It might be +tempting in such a scenario to \textbf{simply include all the +comparisons between the intervention groups and control within a study +into one meta-analysis}. Yet, researchers should abstain from this +practice, as this would mean that the control group is used twice for +the meta-analysis, thus \textbf{``double-counting''} the participants in +the control group. This results in a \textbf{unit-of-analysis} error, as +the effect size are correlated, and thus not independent, but are +treated as if they would stem from independent samples. + +\textbf{There are two ways to deal with this:} + +\begin{itemize} +\tightlist +\item + Splitting the N of the control group: One method to control for the + unit-of-analysis error to some extent would be to \textbf{split} the + number of participants in the control group between the two + intervention groups. So, if your control group has \(N=50\) + participants, you could divide the control group into two control + groups with he same mean and standard deviation, and \(N=25\) + participants each. After this preparation step, you could calculate + the effect sizes for each intervention arm. As this procedure only + partially removes the unit of analysis error, it is not generally + recommended. A big plus of this procedure, however, is that it makes + \protect\hyperlink{heterogeneity}{\textbf{investigations of + hetereogeneity}} between study arms possible. +\item + Another option would be to \textbf{synthesize the results of the + intervention arms} to obtain one single comparison to the control + group. Despite its practical limitations (sometimes, this would mean + synthesizing the results from extremely different types of + interventions), this procedure does get rid of the unit-of-analysis + error problem, and is thus recommended from a statistical standpoint. + The following calculations will deal with this option. +\end{itemize} + +To synthesize the \textbf{pooled effect size data} (pooled Mean, +Standard Deviation and N), we have to use the following formula: + +\[N_{pooled}=N_1+N_2\] + +\[M_{pooled}=\frac{N_1M_1+N_2M_2}{N_1+N_2}\] + +\[SD_{pooled} = \sqrt{\frac{(N_1-1)SD^{2}_{1}+ (N_2-1)SD^{2}_{2}+\frac{N_1N_2}{N_1+N_2}(M^{2}_1+M^{2}_2-2M_1M_2)} {N_1+N_2-1}}\] + +As these formulae are quite lengthy, we prepared the function +\texttt{pool.groups} for you, which does the pooling for you +automatically. Again, R doesn't know this function yet, so we have to +let R learn it by \textbf{copying and pasting} the code underneath +\textbf{in its entirety} into the \textbf{console} on the bottom left +pane of RStudio, and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{pool.groups<-}\ControlFlowTok{function}\NormalTok{(n1,n2,m1,m2,sd1,sd2)\{} + +\NormalTok{n1 <-}\StringTok{ }\NormalTok{n1} +\NormalTok{n2 <-}\StringTok{ }\NormalTok{n2} +\NormalTok{m1 <-}\StringTok{ }\NormalTok{m1} +\NormalTok{m2 <-}\StringTok{ }\NormalTok{m2} +\NormalTok{sd1 <-}\StringTok{ }\NormalTok{sd1} +\NormalTok{sd2 <-}\StringTok{ }\NormalTok{sd2} + +\NormalTok{Npooled <-}\StringTok{ }\NormalTok{n1}\OperatorTok{+}\NormalTok{n2} +\NormalTok{Mpooled <-}\StringTok{ }\NormalTok{(n1}\OperatorTok{*}\NormalTok{m1}\OperatorTok{+}\NormalTok{n2}\OperatorTok{*}\NormalTok{m2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)} +\NormalTok{SDpooled <-}\StringTok{ }\KeywordTok{sqrt}\NormalTok{(((n1}\OperatorTok{-}\DecValTok{1}\NormalTok{)}\OperatorTok{*}\NormalTok{sd1}\OperatorTok{^}\DecValTok{2}\OperatorTok{+}\NormalTok{(n2}\OperatorTok{-}\DecValTok{1}\NormalTok{)}\OperatorTok{*}\NormalTok{sd2}\OperatorTok{^}\DecValTok{2}\OperatorTok{+}\NormalTok{(((n1}\OperatorTok{*}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2))}\OperatorTok{*}\NormalTok{(m1}\OperatorTok{^}\DecValTok{2}\OperatorTok{+}\NormalTok{m2}\OperatorTok{^}\DecValTok{2}\OperatorTok{-}\DecValTok{2}\OperatorTok{*}\NormalTok{m1}\OperatorTok{*}\NormalTok{m2)) )}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2}\OperatorTok{-}\DecValTok{1}\NormalTok{))} + +\KeywordTok{return}\NormalTok{(}\KeywordTok{data.frame}\NormalTok{(Mpooled,SDpooled,Npooled))} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +\textbf{To use this function, we have to specifiy the following +parameters:} + +\begin{itemize} +\tightlist +\item + \texttt{n1}: The N in the first group +\item + \texttt{n2}: The N in the second group +\item + \texttt{m1}: The Mean of the first group +\item + \texttt{m2}: The Mean of the second group +\item + \texttt{sd1}: The Standard Deviation of the first group +\item + \texttt{sd2}: The Standard Deviation of the second grop +\end{itemize} + +\textbf{Here's an example} + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{pool.groups}\NormalTok{(}\DataTypeTok{n1=}\DecValTok{50}\NormalTok{,} + \DataTypeTok{n2=}\DecValTok{50}\NormalTok{,} + \DataTypeTok{m1=}\FloatTok{3.5}\NormalTok{,} + \DataTypeTok{m2=}\DecValTok{4}\NormalTok{,} + \DataTypeTok{sd1=}\DecValTok{3}\NormalTok{,} + \DataTypeTok{sd2=}\FloatTok{3.8}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Mpooled SDpooled Npooled +## 1 3.75 3.415369 100 +\end{verbatim} + +\begin{rmdinfo} +\textbf{What should i do when an study has more than two intervention +groups} + +If a study has more than one two intervention groups you want to +synthesize (e.g.~four arms, with three distinct intervention arms), you +can \textbf{pool the effect size data for the first two interventions}, +and then \textbf{synthesize the pooled data you calculated with the data +from the third group}. + +This is fairly straightforward if you save the output from +\texttt{pool.groups} as an object, and then use the \texttt{\$} +operator: +\end{rmdinfo} + +First, pool the \textbf{first} and \textbf{second intervention group}. I +will save the output as \texttt{res}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{res<-}\KeywordTok{pool.groups}\NormalTok{(}\DataTypeTok{n1 =} \DecValTok{50}\NormalTok{,} + \DataTypeTok{n2 =} \DecValTok{50}\NormalTok{,} + \DataTypeTok{m1 =} \FloatTok{3.5}\NormalTok{,} + \DataTypeTok{m2 =} \DecValTok{4}\NormalTok{,} + \DataTypeTok{sd1 =} \DecValTok{3}\NormalTok{,} + \DataTypeTok{sd2 =} \FloatTok{3.8}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +Then, use the pooled data saved in \texttt{res} and \textbf{pool it with +the data from the third group}, using the \texttt{\$} operator to access +the different values saved in \texttt{res}. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{pool.groups}\NormalTok{(}\DataTypeTok{n1 =}\NormalTok{ res}\OperatorTok{$}\NormalTok{Npooled,} + \DataTypeTok{n2 =} \DecValTok{60}\NormalTok{,} + \DataTypeTok{m1 =}\NormalTok{ res}\OperatorTok{$}\NormalTok{Mpooled,} + \DataTypeTok{m2 =} \FloatTok{4.1}\NormalTok{,} + \DataTypeTok{sd1=}\NormalTok{res}\OperatorTok{$}\NormalTok{SDpooled,} + \DataTypeTok{sd2 =} \FloatTok{3.8}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +\begin{verbatim} +## Mpooled SDpooled Npooled +## 1 3.88125 3.556696 160 +\end{verbatim} + +\chapter{Power Analysis}\label{power-analysis} + +\begin{figure} +\centering +\includegraphics{poweranalysis.jpg} +\caption{} +\end{figure} + +A big asset (and probably one of the reasons why meta-analysis can be +helpful in practical research) of meta-analyses is that \textbf{they +allow for large data to be combined} to attain a more precise pooled +effect. \textbf{Lack of statistical power}, however, may still play an +important role, even in meta-analysis. This is particularly true for the +clinical field, where it is often the case that only \textbf{few studies +are available for synthesis}. The median number of included studies in +the \emph{Cochrane Database for Systematic Reviews}, for example, is six +\citep{borenstein2011}. This is even more grave once we consider that +(1) many meta-analysts will also want to perform \textbf{subgroup +analyses and meta-regression}, for which even more power is required, +and (2) many meta-analyses have \textbf{high heterogeneity}, which +reduces our precision, and thus our power. + +Power is directly related to the \textbf{Type II error level} +(\(\beta\)) we defined: \(Power = 1- \beta\). It is common practice to +set our \textbf{Type I error level} (\(\alpha\)) to \(\alpha=0.05\), and +thus to assume that the \textbf{Type I error is four times as grave as +the Type II error} (i.e., falsely finding an effect while there is no +effect in reality is four times as bad as not finding an effect while +there is one in reality). The \textbf{Type II error} is therefore set at +\(\beta=0.20\), and the power should thus be \(1-\beta=1-0.20=80\%\). + +\begin{rmdinfo} +\textbf{What assumptions should i make for my meta-analysis?} + +While researchers conducting primary studies can \textbf{plan the size +of their sample based on the effect size they want to find}, the +situation is a little different in meta-analysis, where we can only work +with the published material. However, we have some \textbf{control over +the number of studies we want to include in our meta-analysis} (e.g., +through more leniently or strictly defined inclusion criteria). +Therefore, we can change our power to some extent by including more or +less studies into the meta-analysis. There are \textbf{four things we +have to make assumptions about when assessing the power of our +meta-analysis a priori}. + +\begin{itemize} +\tightlist +\item + The \textbf{number of included or includable studies} \(k\) +\item + The \textbf{overall size of the studies we want to include} (are the + studies in the field rather small or large?) +\item + The \textbf{effect size we want to determine}. This is particularly + important, as we have to make assumptions about how \textbf{big an + effect size has to be to still be clinically meaningful}. One study + calculated that for interventions against depression, even effects as + small as \(SMD=0.24\) may still be meaningful for patients + {[}@cuijpers2014threshold{]}. If we want to study \textbf{negative + effects of an intervention} (e.g., death or symptom deterioration), + even \textbf{very small effect sizes are extremely important and + should be detected}. +\item + The \textbf{heterogeneity} of our studies' effect sizes, as this also + affects the precision of our meta-analysis, and thus its potential to + find significant effects. +\end{itemize} + +Besides these parameters, it is also important to think about other +analyses, such as the \textbf{subgroup analyses} we want to conduct. How +many studies are there for each subgroup, and what effects do we want to +find in the subgroups? This is particularly important if we +\textbf{hypothesize that an intervention is not effective in a subgroup +of patients}, because we do not want to falsely find a treatment to be +ineffective simply because the power was insufficient. +\end{rmdinfo} + +\begin{rmdachtung} +\textbf{Post-hoc power tests: the abuse of power} + +Please note that power analyses should always be conducted \textbf{a +priori}, meaning \emph{before} you perform the meta-analysis. + +Power analyses conducted \emph{after} an analysis (``post hoc'') are +fundamentally flawed {[}@hoenig2001abuse{]}, as they suffer from the +so-called \textbf{``power approach paradox''}, in which an analysis +yielding no significant effect is thought to show more evidence that the +null hypothesis is true when the p-value is smaller, since then, the +power to detect a true effect would be higher. +\end{rmdachtung} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\hypertarget{fixed.power}{\section{Fixed-Effect +Model}\label{fixed.power}} + +To determine the \textbf{power} of a meta-analysis \textbf{under the +fixed-effect model}, we have to assume the \textbf{true value of a +distribution when the alternative hypothesis is correct} (i.e., when +there is an effect). For power analysis in a conventional study, this +distribution is \(Z\). Follwing Borenstein et al. +\citep{borenstein2011}, we will call the true value \(\lambda\) here to +make clear that we are dealing with a meta-analysis, and not a primary +study. \(\lambda\) is defined as: + +\[\lambda=\frac{\delta}{\sqrt{V_{\delta}}}\] + +Where \(\delta\) is the \textbf{true effect size} and \(V_{\delta}\) its +variance. + +\(V_{\delta}\) can be calculated for meta-analysis using the +fixed-effect model with this formula: + +\[V_{\delta}=\frac{\frac{n_1+n_2}{n_1xn_2}+\frac{d^2}{2(n_1+n_2)}}{k}\] + +Where \(k\) are all the included studies, and \(n_1\) and \(n_2\) are +the \textbf{average sample sizes in each trial arm we assume across our +studies}. + +Assuming a normal distribution and using \(\lambda\), we can calculate +the Power: + +\[Power = 1- \beta\] +\[Power = 1- \Phi(c_{\alpha}-\lambda)+\Phi(-c_{\alpha}-\lambda) \] + +Where \(c_{\alpha}\) is the critical value of a \(Z\)-distribution. +\(\Phi\) is the \textbf{standard normal density function}, which we we +need to calcuate the power using this equation: +\[\Phi(Z)=\frac{1}{\sqrt {2\pi}}e^{-\frac{Z^2}{2}}\] + +Luckily, you don't have too think about these statistical details too +much, as we have prepared a \textbf{function} for you with which you can +easily conduct a \textbf{power analysis} using the fixed-effect model +yourself. The function is called \texttt{power.analysis.function} and +its code can be found below. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{power.analysis.fixed<-}\ControlFlowTok{function}\NormalTok{(d,k,n1,n2,p)\{} + +\NormalTok{ n1<-n1} +\NormalTok{ n2<-n2} +\NormalTok{ d<-d} +\NormalTok{ k<-k} +\NormalTok{ p<-p} +\NormalTok{ title<-}\KeywordTok{c}\NormalTok{(}\StringTok{"Power for a fixed-effect meta-analysis:"}\NormalTok{)} + +\NormalTok{ v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))} +\NormalTok{ v.m<-v.d}\OperatorTok{/}\NormalTok{k} +\NormalTok{ lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))} +\NormalTok{ plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)} +\NormalTok{ zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)} +\NormalTok{ power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))} + \KeywordTok{return}\NormalTok{(power)} +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\textbf{For this function, we have to specify the following parameters:} + +\begin{tabular}{l|l} +\hline +Parameter & Description\\ +\hline +d & The effect size we want to be able to detect\\ +\hline +k & The number of studies we will likely be able to include into our analysis\\ +\hline +n1 & The average number of participats we assume in the intervention arms of our included studies\\ +\hline +n2 & The average number of participats we assume in the control arms of our included studies\\ +\hline +p & The Type I error rate (p-level). It is common to use 'p=0.05'\\ +\hline +\end{tabular} + +Now, let's give an example. I assume that an effect of \(d=0.30\) is +likely and meaningful for the field of my meta-analysis. I also assume +that on average, the studies in my analysis will be rather small, with +25 participants in each trial arm, and that there will be 10 studies in +my analysis. I will set the \(\alpha\)-level to 0.05, as is convention. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{power.analysis.fixed}\NormalTok{(}\DataTypeTok{d=}\FloatTok{0.30}\NormalTok{,}\DataTypeTok{k=}\DecValTok{10}\NormalTok{,}\DataTypeTok{n1=}\DecValTok{25}\NormalTok{,}\DataTypeTok{n2=}\DecValTok{25}\NormalTok{,}\DataTypeTok{p=}\FloatTok{0.05}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +The output of the function is: + +\begin{verbatim} +## [1] 0.9155008 +\end{verbatim} + +Meaning that my power is 91\%. This is more than the desired 80\%, so +given that my assumptions are remotely true, my meta-analysis will have +\textbf{sufficient power using the fixed-effect model to detect a +clinically relevant effect if it exists}. + +So, if i assume an effect of \(d = 0.30\) in this example, i am lucky. +If we play around with the effect size a little, however, while holding +the other paramters constant, this can look very different. + +\begin{verbatim} +## Warning: package 'reshape' was built under R version 3.4.4 +\end{verbatim} + +\begin{center}\includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-208-1} \end{center} + +As you can see from this plot, sufficient power (see the \textbf{dashed +line}) is soon reached for \(d=0.30\), even if only few studies are +included. If i assume a smaller effect size of \(d=0.10\), however, +\textbf{even 50 studies will not be sufficient to find a true effect}. + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Random-Effects Model}\label{random-effects-model} + +For power analyses under the \textbf{random-effects model}, the formula +to calculate the variance of my true mean effect looks slightly +different: + +\[V_{\delta}^*=\frac{V_Y+\tau^2}{k}\] + +We see that again, \(tau^2\) has to be included to take the +\textbf{between-study heterogeneity into account} (see +\protect\hyperlink{random}{Chapter 4.2} for more details). However, i +\textbf{do not know the between-study heterogeneity of my analysis} +before i perform it, so what value should i assume? + +According to Hedges and Pigott \citep{hedges2004power}, the follwing +formulae may be used to calculate the power in the random-effect model +assuming \textbf{small, moderate or high heterogeneity}: + +\textbf{Small heterogeneity:} + +\[V_{\delta}^*=1.33\times\frac{V_Y}{k}\] + +\textbf{Moderate heterogeneity:} + +\[V_{\delta}^*=1.67\times\frac{V_Y}{k}\] + +\textbf{Large heterogeneity:} + +\[V_{\delta}^*=2\times\frac{V_Y}{k}\] + +Again, you don't have to worry about the statistical details here. We +have put the entire calculations into the \texttt{power.analysis.random} +function, which can be found below. + +Again, R doesn't know this function yet, so we have to let R learn it by +\textbf{copying and pasting} the code underneath \textbf{in its +entirety} into the \textbf{console} on the bottom left pane of RStudio, +and then hit \textbf{Enter ⏎}. + +\begin{Shaded} +\begin{Highlighting}[] +\NormalTok{power.analysis.random<-}\ControlFlowTok{function}\NormalTok{(d,k,n1,n2,p,heterogeneity)\{} + +\NormalTok{ n1<-n1} +\NormalTok{ n2<-n2} +\NormalTok{ d<-d} +\NormalTok{ k<-k} +\NormalTok{ p<-p} +\NormalTok{ heterogeneity<-heterogeneity} + + \ControlFlowTok{if}\NormalTok{(heterogeneity}\OperatorTok{==}\StringTok{"low"}\NormalTok{)\{} + +\NormalTok{ v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))} +\NormalTok{ v.m<-v.d}\OperatorTok{/}\NormalTok{k} +\NormalTok{ v.m<-}\FloatTok{1.33}\OperatorTok{*}\NormalTok{v.m} +\NormalTok{ lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))} +\NormalTok{ plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)} +\NormalTok{ zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)} +\NormalTok{ power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))} + \KeywordTok{return}\NormalTok{(power)} +\NormalTok{ \}} + + \ControlFlowTok{if}\NormalTok{(heterogeneity}\OperatorTok{==}\StringTok{"moderate"}\NormalTok{)\{} + +\NormalTok{ v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))} +\NormalTok{ v.m<-v.d}\OperatorTok{/}\NormalTok{k} +\NormalTok{ v.m<-}\FloatTok{1.67}\OperatorTok{*}\NormalTok{v.m} +\NormalTok{ lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))} +\NormalTok{ plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)} +\NormalTok{ zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)} +\NormalTok{ power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))} + \KeywordTok{return}\NormalTok{(power)} +\NormalTok{ \}} + + \ControlFlowTok{if}\NormalTok{(heterogeneity}\OperatorTok{==}\StringTok{"high"}\NormalTok{)\{} + +\NormalTok{ v.d<-((n1}\OperatorTok{+}\NormalTok{n2)}\OperatorTok{/}\NormalTok{(n1}\OperatorTok{*}\NormalTok{n2))}\OperatorTok{+}\NormalTok{((d}\OperatorTok{*}\NormalTok{d)}\OperatorTok{/}\NormalTok{(}\DecValTok{2}\OperatorTok{*}\NormalTok{(n1}\OperatorTok{+}\NormalTok{n2)))} +\NormalTok{ v.m<-v.d}\OperatorTok{/}\NormalTok{k} +\NormalTok{ v.m<-}\DecValTok{2}\OperatorTok{*}\NormalTok{v.m} +\NormalTok{ lambda<-(d}\OperatorTok{/}\KeywordTok{sqrt}\NormalTok{(v.m))} +\NormalTok{ plevel<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(p}\OperatorTok{/}\DecValTok{2}\NormalTok{)} +\NormalTok{ zval<-}\KeywordTok{qnorm}\NormalTok{(}\DataTypeTok{p=}\NormalTok{plevel, }\DecValTok{0}\NormalTok{,}\DecValTok{1}\NormalTok{)} +\NormalTok{ power<-}\DecValTok{1}\OperatorTok{-}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(zval}\OperatorTok{-}\NormalTok{lambda))}\OperatorTok{+}\NormalTok{(}\KeywordTok{pnorm}\NormalTok{(}\OperatorTok{-}\NormalTok{zval}\OperatorTok{-}\NormalTok{lambda))} + \KeywordTok{return}\NormalTok{(power)} +\NormalTok{ \}} + +\NormalTok{\}} +\end{Highlighting} +\end{Shaded} + +Now you are set and ready to use the function. I will assume the same +parameters from \protect\hyperlink{fixed.power}{before}, but this time i +will also have to specify the \texttt{heterogeneity} in the function, +which can take the values \texttt{"low"}, \texttt{"moderate"} and +\texttt{"high"}. I will choose \texttt{"moderate"} for this example. + +\begin{Shaded} +\begin{Highlighting}[] +\KeywordTok{power.analysis.random}\NormalTok{(}\DataTypeTok{d=}\FloatTok{0.30}\NormalTok{,}\DataTypeTok{k=}\DecValTok{10}\NormalTok{,}\DataTypeTok{n1=}\DecValTok{25}\NormalTok{,}\DataTypeTok{n2=}\DecValTok{25}\NormalTok{,}\DataTypeTok{p=}\FloatTok{0.05}\NormalTok{,} + \DataTypeTok{heterogeneity =} \StringTok{"moderate"}\NormalTok{)} +\end{Highlighting} +\end{Shaded} + +The output i get is: + +\begin{verbatim} +## [1] 0.7327163 +\end{verbatim} + +Interestingly, we see that this value is 73\%, which is smaller than the +value of 91\% which was calculated using the \textbf{fixed-effect +model}. The value is also below 80\%, meaning that i would not have +optimal power to find the desired effect of \(d=0.30\) to be +statistically significant if it exists. + +This has to do with the \textbf{larger heterogeneity} i assume in this +simulation, which decreases the precision of my effect size estimate, +and thus increases my need for statistical power. + +\textbf{The graph below visualizes this relationship:} + +\begin{figure} + +{\centering \includegraphics{Doing_Meta_Analysis_in_R_files/figure-latex/unnamed-chunk-212-1} + +} + +\caption{Power in the random-effects-model. Darker colors indicate higher heterogeneity}\label{fig:unnamed-chunk-212} +\end{figure} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\section{Power Calculator Tool}\label{power-calculator-tool} + +If you're feeling lazy, or if you want to quickly \textbf{check for the +power of your meta-analysis under varying assumptions}, you might need a +tool which makes it easier for you to calculate the power without having +to run the R functions we described before each time. + +We therefore built a online \textbf{Power Calculator Tool}, which you +can find below. The calculations are based on the formulae and functions +we described in the previous chapters. + +\href{https://mathiasharrer.shinyapps.io/power_calculator_meta_analysis/}{View +in full page mode} + +\begin{center}\rule{0.5\linewidth}{\linethickness}\end{center} + +\bibliography{book.bib,packages.bib} + + +\end{document} diff --git a/_pcurve/.DS_Store b/_pcurve/.DS_Store new file mode 100644 index 0000000..5008ddf Binary files /dev/null and b/_pcurve/.DS_Store differ diff --git a/book.bib b/book.bib index 08b3654..be38b63 100644 --- a/book.bib +++ b/book.bib @@ -609,3 +609,58 @@ @article{labbe year={1987}, publisher={Am Coll Physicians} } +<<<<<<< HEAD +======= + +@article{pastor2018multilevel, + title={On the multilevel nature of meta-analysis: a tutorial, comparison of software programs, and discussion of analytic choices}, + author={Pastor, Dena A and Lazowski, Rory A}, + journal={Multivariate behavioral research}, + volume={53}, + number={1}, + pages={74--89}, + year={2018}, + publisher={Taylor \& Francis} +} + +@article{cheung2014modeling, + title={Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach.}, + author={Cheung, Mike W-L}, + journal={Psychological Methods}, + volume={19}, + number={2}, + pages={211}, + year={2014}, + publisher={American Psychological Association} +} + +@article{assink2016fitting, + title={Fitting three-level meta-analytic models in R: A step-by-step tutorial}, + author={Assink, Mark and Wibbelink, Carlijn JM}, + journal={The Quantitative Methods for Psychology}, + volume={12}, + number={3}, + pages={154--174}, + year={2016} +} + +@article{hedges2009statistical, + title={Statistical considerations}, + author={Hedges, Larry V}, + journal={The handbook of research synthesis and meta-analysis}, + pages={38--47}, + year={2009}, + publisher={Russell Sage Foundation New York} +} + + @article{lme4, + title = {Fitting Linear Mixed-Effects Models Using {lme4}}, + author = {Douglas Bates and Martin M{\"a}chler and Ben Bolker and Steve Walker}, + journal = {Journal of Statistical Software}, + year = {2015}, + volume = {67}, + number = {1}, + pages = {1--48}, + doi = {10.18637/jss.v067.i01}, + } +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/dats.xlsx b/dats.xlsx new file mode 100644 index 0000000..23c2645 Binary files /dev/null and b/dats.xlsx differ diff --git a/favicon.ico b/favicon.ico new file mode 100644 index 0000000..f203752 Binary files /dev/null and b/favicon.ico differ diff --git a/index.Rmd b/index.Rmd index 6a16693..6b4c54b 100644 --- a/index.Rmd +++ b/index.Rmd @@ -1,11 +1,16 @@ --- title: "Doing Meta Analysis in R" +<<<<<<< HEAD subtitle: "A Hands-On Guide" +======= +subtitle: "A Hands-on Guide" +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 author: "Mathias Harrer, B.Sc. & Dr. habil. David Ebert" date: "Behavioral Health Promotion & Technology Lab" github-repo: "MathiasHarrer/Doing-Meta-Analysis-in-R" site: bookdown::bookdown_site output: +<<<<<<< HEAD bookdown::pdf_book: toc: true fig_caption: false @@ -14,15 +19,33 @@ output: geometry: margin=2cm fontsize: 12pt mainfont: Rosario +======= + bookdown::gitbook: + config: + toc: + collapse: section + search: yes + fontsettings: + size: 1 + split_by: section + includes: + after_body: banner.html + df_print: paged +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 documentclass: book bibliography: [book.bib, packages.bib] biblio-style: apalike link-citations: yes description: "This is a guide on how to conduct Meta-Analysis in R." +favicon: "favicon.ico" --- # About this Guide +<<<<<<< HEAD +======= +![](coverbild.jpg) +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 ```{block} This guide shows you how to conduct Meta-Analyses in R from scratch. The focus of this guide is primarily on clinical outcome research in psychology. It is designed for staff and collaborators of the [**PROTECT Lab**](https://www.protectlab.org), which is headed by **Dr. David D. Ebert**. @@ -30,7 +53,18 @@ This guide shows you how to conduct Meta-Analyses in R from scratch. The focus o +<<<<<<< HEAD $~$ +======= +```{r, echo=FALSE, fig.width=3,fig.height=2} +library(png) +library(grid) +img <- readPNG("protectlogo.PNG") +grid.raster(img) +``` + +--- +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 **The guide will show you how to:** @@ -64,11 +98,15 @@ $~$ **How to read this book online** +<<<<<<< HEAD We highly recommend to also have a look at the online version of this guide, which can be found here: https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/. The online guide is more interactive and updated regularly with new material. It contains Web Apps and further guides on other topics around doing Meta-Analyses in R which may be of interest to you. $~$ **How to get the R code for this guide** +======= +![](githublogo.png) +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 All code behind this book is available online on **GitHub**. We have created a website containing a **download link** for all codes, and a **quick guide** on how to get the code running on your computer. The site can be found [here](https://mathiasharrer.github.io/Doing-Meta-Analysis-in-R/). diff --git a/mlm. b/mlm. new file mode 100644 index 0000000..ddb710b Binary files /dev/null and b/mlm. differ diff --git a/mlm.data.RData b/mlm.data.RData new file mode 100644 index 0000000..1c276fb Binary files /dev/null and b/mlm.data.RData differ diff --git a/multilevel-model.png b/multilevel-model.png new file mode 100644 index 0000000..a5f9db2 Binary files /dev/null and b/multilevel-model.png differ diff --git a/multilevel-model2.png b/multilevel-model2.png new file mode 100644 index 0000000..50891c8 Binary files /dev/null and b/multilevel-model2.png differ diff --git a/multilevel.jpg b/multilevel.jpg new file mode 100644 index 0000000..fa4a476 Binary files /dev/null and b/multilevel.jpg differ diff --git a/packages.bib b/packages.bib index 636ea7e..6d988aa 100644 --- a/packages.bib +++ b/packages.bib @@ -6,6 +6,16 @@ @Manual{R-base year = {2017}, url = {https://www.R-project.org/}, } +<<<<<<< HEAD +======= +@Manual{R-bindrcpp, + title = {bindrcpp: An 'Rcpp' Interface to Active Bindings}, + author = {Kirill Müller}, + year = {2018}, + note = {R package version 0.2.2}, + url = {https://CRAN.R-project.org/package=bindrcpp}, +} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 @Manual{R-bookdown, title = {bookdown: Authoring Books and Technical Documents with R Markdown}, author = {Yihui Xie}, @@ -13,6 +23,37 @@ @Manual{R-bookdown note = {R package version 0.7}, url = {https://CRAN.R-project.org/package=bookdown}, } +<<<<<<< HEAD +======= +@Manual{R-dplyr, + title = {dplyr: A Grammar of Data Manipulation}, + author = {Hadley Wickham and Romain François and Lionel Henry and Kirill Müller}, + year = {2018}, + note = {R package version 0.7.6}, + url = {https://CRAN.R-project.org/package=dplyr}, +} +@Manual{R-forcats, + title = {forcats: Tools for Working with Categorical Variables (Factors)}, + author = {Hadley Wickham}, + year = {2017}, + note = {R package version 0.2.0}, + url = {https://CRAN.R-project.org/package=forcats}, +} +@Manual{R-ggplot2, + title = {ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics}, + author = {Hadley Wickham and Winston Chang}, + year = {2016}, + note = {R package version 2.2.1}, + url = {https://CRAN.R-project.org/package=ggplot2}, +} +@Manual{R-kableExtra, + title = {kableExtra: Construct Complex Table with 'kable' and Pipe Syntax}, + author = {Hao Zhu}, + year = {2018}, + note = {R package version 0.9.0}, + url = {https://CRAN.R-project.org/package=kableExtra}, +} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 @Manual{R-knitr, title = {knitr: A General-Purpose Package for Dynamic Report Generation in R}, author = {Yihui Xie}, @@ -20,6 +61,72 @@ @Manual{R-knitr note = {R package version 1.20}, url = {https://CRAN.R-project.org/package=knitr}, } +<<<<<<< HEAD +======= +@Manual{R-lme4, + title = {lme4: Linear Mixed-Effects Models using 'Eigen' and S4}, + author = {Douglas Bates and Martin Maechler and Ben Bolker and Steven Walker}, + year = {2017}, + note = {R package version 1.1-13}, + url = {https://CRAN.R-project.org/package=lme4}, +} +@Manual{R-Matrix, + title = {Matrix: Sparse and Dense Matrix Classes and Methods}, + author = {Douglas Bates and Martin Maechler}, + year = {2017}, + note = {R package version 1.2-10}, + url = {https://CRAN.R-project.org/package=Matrix}, +} +@Manual{R-meta, + title = {meta: General Package for Meta-Analysis}, + author = {Guido Schwarzer}, + year = {2018}, + note = {R package version 4.9-2}, + url = {https://CRAN.R-project.org/package=meta}, +} +@Manual{R-metafor, + title = {metafor: Meta-Analysis Package for R}, + author = {Wolfgang Viechtbauer}, + year = {2017}, + note = {R package version 2.0-0}, + url = {https://CRAN.R-project.org/package=metafor}, +} +@Manual{R-png, + title = {png: Read and write PNG images}, + author = {Simon Urbanek}, + year = {2013}, + note = {R package version 0.1-7}, + url = {https://CRAN.R-project.org/package=png}, +} +@Manual{R-purrr, + title = {purrr: Functional Programming Tools}, + author = {Lionel Henry and Hadley Wickham}, + year = {2018}, + note = {R package version 0.2.5}, + url = {https://CRAN.R-project.org/package=purrr}, +} +@Manual{R-RColorBrewer, + title = {RColorBrewer: ColorBrewer Palettes}, + author = {Erich Neuwirth}, + year = {2014}, + note = {R package version 1.1-2}, + url = {https://CRAN.R-project.org/package=RColorBrewer}, +} +@Manual{R-readr, + title = {readr: Read Rectangular Text Data}, + author = {Hadley Wickham and Jim Hester and Romain Francois}, + year = {2017}, + note = {R package version 1.1.1}, + url = {https://CRAN.R-project.org/package=readr}, +} +@Manual{R-readxl, + title = {readxl: Read Excel Files}, + author = {Hadley Wickham and Jennifer Bryan}, + year = {2017}, + note = {R package version 1.0.0}, + url = {https://CRAN.R-project.org/package=readxl}, +} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 @Manual{R-rmarkdown, title = {rmarkdown: Dynamic Documents for R}, author = {JJ Allaire and Yihui Xie and Jonathan McPherson and Javier Luraschi and Kevin Ushey and Aron Atkins and Hadley Wickham and Joe Cheng and Winston Chang and Richard Iannone}, @@ -27,3 +134,34 @@ @Manual{R-rmarkdown note = {https://rmarkdown.rstudio.com, https://github.com/rstudio/rmarkdown}, } +<<<<<<< HEAD +======= +@Manual{R-stringr, + title = {stringr: Simple, Consistent Wrappers for Common String Operations}, + author = {Hadley Wickham}, + year = {2018}, + note = {R package version 1.3.1}, + url = {https://CRAN.R-project.org/package=stringr}, +} +@Manual{R-tibble, + title = {tibble: Simple Data Frames}, + author = {Kirill Müller and Hadley Wickham}, + year = {2018}, + note = {R package version 1.4.2}, + url = {https://CRAN.R-project.org/package=tibble}, +} +@Manual{R-tidyr, + title = {tidyr: Easily Tidy Data with 'spread()' and 'gather()' Functions}, + author = {Hadley Wickham and Lionel Henry}, + year = {2018}, + note = {R package version 0.8.1}, + url = {https://CRAN.R-project.org/package=tidyr}, +} +@Manual{R-tidyverse, + title = {tidyverse: Easily Install and Load the 'Tidyverse'}, + author = {Hadley Wickham}, + year = {2017}, + note = {R package version 1.2.1}, + url = {https://CRAN.R-project.org/package=tidyverse}, +} +>>>>>>> f3259eafbebf95ffc6044d4af3f61e06d59c7876 diff --git a/~$dats.xlsx b/~$dats.xlsx new file mode 100644 index 0000000..171d11b Binary files /dev/null and b/~$dats.xlsx differ