Okay, in this video, we're going to conduct our first analysis of a multi-factor general linear model. When I say multi-factor, I just mean a general linear model that has more than one factor. So our simple case that we're going to consider just involves two different factors. And then we're going to use this analysis as a way to discuss when we should model interactions and a general linear model with multiple factors. I've created the data by simulation for this introductory example. And I've done that because I wanted to make sure the data meet all the assumptions of our analysis. So I've simulated the data so that we can be very certain if the data will meets the assumption of equal variance and of normality. And as well, I've created the data in such a way that we don't have to worry about the assumption of random sampling or whether or not the data are independent. The experimental design that looks something like this, where we have two factors, one being sex. And so we have either females or males. And then we have another fact which I've called treatment, which involves either the administration of a drug or administration of a control treatment. And the, the variable that we've measured I haven't even named, I think I've just called it Y. And that doesn't really matter for this pretend example. So we can just imagine we have an experiment where we're either giving females or males a drug or a control treatment. And then we're measuring something about those individuals. We have a total sample size of 80 in this experiment. And specifically to the sample size of 20 in each of our different combinations. Because we have the same number of data points in each of these four combinations, we call a data set like this balanced. If we had different sample sizes among our various combinations of our factors, then we would call that kind of situation an unbalanced dataset. And, and a later video we're going to talk about the differences between balanced and unbalanced datasets and how we analyze them. But this dataset is, has a balanced design. Okay, so that's our dataset. Let's start analyzing it. So we'll open are. The first thing we want to do is just to import the data set. And I've just called the dataset. It's an a CSV file called Example dot CSB. We'll just call our DataFrame, which holds these data what's called EEG sort, for example. And we use the read table command. Whoops. This should all be familiar to you. If you're not, then you can go back, go back and watch a previous video on how to import data. So there we go. Let's have a look at this DataFrame. So we'll just type the name of the DataFrame and say run. But DataFrame has 1234 columns. You can ignore this second column. I created it for another purpose that we're not gonna deal with in this video. But you can see that we have our measurement of each individual, which is denoted y. And then we have a column called treatment, which notes whether or not an individual is experienced a control versus a drug. And then we have another column called sex, which denotes whether or not the individuals are females or males. I really want you to take quick note or tick, good note of how these data are organized, okay? Because I want you to realize that each line or each row within this DataFrame provides all the information about a particular measurement. So we know for this measurement which had a value of 24.709, we know that this measurement corresponds to an individual that receives the control treatment. And this individual was female. Whereas at some other point, we have this measurement of 11.87. It corresponds to an individual that received a control treatment and this individual happens to be male. So the data are organized so that each row contains all the information from this experiments that relates to a particular measurement. Okay? Now, those are our data. That's how they're organized. They could get a quick summary of those data just by saying summary. Let's do that. And you can see here that for sex we have 40 females and 40 males. For a treatment we have 40 control 40 in the drug treatment, which is what we expected base my description of the experiment earlier. I'm, I'll just note that my impression is that different versions of R will provide different output for the summary file. This is something I seem to have gotten from talking with some of my students who have installed more up-to-date versions of R, then I actually have. So just, just to note that if your summary output looks different from mine, that's probably why. Okay, as always, you want to start. Logging the data. When we plot these data, you're going to see this is the first time that we're going to encounter a real difference between a two factor analysis and a one factor analysis. Let's start with creating a box plot. So just like we've done before, we're going to list our in, our dependent variable first or hypothesis in this experiment is that sex and treatment will likely influence our y variable. And so y is going to be our dependent variable. So just like we've done before, we just placed the name of our column, which is our dependent variable, which in this case I've conveniently named y. And we have tilda. And now we have to list our independent variables. But in this case, remember we don't just have one independent variable. We have to, we have the column sex and we have the column treatment. How do we specify a box plot? When we have both of these, when we now have two independent variables. Well, we can do that simply by listing both of them. So sex and treatment. But what we're gonna do is we're going to put an asterix between them. Because when we do that, when we include this asterix, what we're telling our to do or what we're telling the box plot function to do is to create a box plot. But using all combinations of the levels of these three different treatments, which we'll see in a moment. Okay, so now we need to specify our dataset. So say data equals e g. And now I'll say Run. And here is our box plot. You can see that we don't really have enough space for it to list all the different X axis labels will just make this bigger. And now you can see our nice box plot, okay? And you can see that what are what the boxplot function is done. It has taken all combinations of the sex and treatment factors and plotted the data from each of those combinations separately. So first we have these data from the combination of females in the control treatment. Then we have males and the control treatment. And then we have the data for females and the drug treatment and males from the drug treatment. Okay? And let's start just by seeing what we can learn from this plot. Let's start just by comparing these first two combinations. Both of these combinations involve looking at measurements that were made from individuals that experienced the control treatment. And what we can see here is that the females that experienced the control treatment, we have good reason to think that they have a higher y-value. Than the males that experienced the control treatment. So this would suggest that for whatever biological reason, females tend to have a higher measurement for this particular variable than males do. You can see that the distributions based on these box plots hardly overlap at all. And so if we were to do an analysis just on these data alone, so let's say imagine that we took these data, pulled them out of our larger DataFrame, and just did a t-test on these data, we would expect there to be evidence for a difference between the females and the males. Now let's compare the females and males for the drug treatment. Can see in this case, we can see that again, there looks like there's a difference between the females and the males for individuals that bow, for individuals that experienced the drug. But you can see that this time, the difference between females and males seems to be much larger when the drug is present than when the drug is absent. So this suggests that the size of the difference between females and males may depend on whether or not the individuals experienced the control treatment or the drug treatment. That's something we should really keep in mind. So our comparisons here also suggests that we expects there to be a difference between females and males. What about a difference between the drug and control treatment? While to do that, we can do that roughly in our minds by pooling over our different sexes. So if we were to take these data here and pool over the females and males, just pool all the control data. Then we would expect that they're kind of average would probably be somewhere around there. Whereas if we pool over the females and males for the drug treatment, their average would be somewhere around there. So that would suggest then that we might expect, very roughly speaking, because we're doing this pooling in our head. That would lead us to suspect that the average measurement for individuals that experienced the drug would be higher than the average measurement for individuals that experience to control treatment. So that would lead us to potentially expect a difference between the drug and the control treatments. But we need to bear in mind, we've, what we've already learned, which is that we expect the size of the sex treatment to differ between the control and the drug treatment. Okay. Just complete this tour, these data, I'll just point out that we can still, we can still add on the original raw data points if you want. Uses trip chart command. When we do that, we specify the data in exactly the same way as we did in boxplots. So we'll keep that Asterix. See if I can remember all the commands off top of my head. We want to add this. So say add equals true, vertical equals true. We want to jitter the points. So method equals f equals jitter. Let's make the points be circles. Whoops. And we'll say color equals VR una, okay. There we go. Adding these individual data points, I don't think actually does a whole lot for us. But it's still I wanted to show you you can still do that even when we have a two factor analysis. Ok, let's start now by analyzing our data. So we're going to use the function lm. And we're going to create, will store output in a DataFrame or into an object will call EEG dot lm. And so we'll say lm. And we're going to create our LM function in the same way as we did with one, with a one factor general linear model. So we're going to list our dependent variable first, which is y. Then we have our tilda. And now we want to list our independent variables. But in this case we have to not just one. We had one independent variable when we're doing a one factor general linear model. In this case we have two. So how do we adds to, well, in this case we can just say Sx plus treatment. It's if we done one fact, the general linear model than we'd just say sex. But we want to, in this case, we want to look at sex and treatment. So we just list them both with a plus in-between. And now we will say data equals E G. Okay, so the basic process of creating this general linear model is basically identical to what we learned with a one factor general linear model. But we just have to list more independent variables. That's it. So now let's run this analysis. Here we go. And now as always, the next thing we want to do is you want to test our assumptions. And I told you earlier that I created these data so that we expect them to meet the assumptions of equal variance and normality. So let see, we get, okay, we test the assumptions and exactly the same way as we did for a one factor generator model. Okay, so this first plot, which gives our fitted values versus the residuals. We would use this plot in order to test our assumption of equal variance. And we're getting something here that is pretty weird. We're getting something we've not seen before. Normally we would see all of our residuals basically centered around this dotted line at 0. But now we're seeing that are for treatment combinations 1234. There are no where near centered. This is really kind of odd. And so really, one of the main points from this video is to try to figure out what's going on here. Okay? This first plot worries us. For normality. Not great, but I've seen worse. And then we're going to ignore or last few plots. Okay? So why would we be getting that strange, that strange plot that allows us to test equivariance. Well, let's go back a little bit. Let's go back in our plots and look at our box plot. And compare this box plots to our linear model. When we created our linear model like this, what we're essentially doing is we're creating a model that allows us to look the effect of sex and to look ineffective treatment. But this specification of the model assumes that the effect of sex will be consistent among our two levels of treatment and vice versa. It's suggest any effective treatment will be consistent for over our two levels of sex. But that's not what we really concluded when we looked at our original Boxplot. When we originally looked at this box plot, we concluded that this really kind of two effects of sex. There's one effective sex in the control treatment. Because in the control treatment it looks like there's a difference between the females and males, but it's relatively small. But in the drug treatment, we see again that we expect there to be a difference between females and males, but in this case, the effect is really large. So the way we specified this model doesn't really match what we see here. Again, just to repeat what I said earlier, when we specify a model like this, we're assuming that the effect of sex be consistent over our levels of treatment and vice versa. And original data suggests that that's really not true. So how can we fix that? How can we plot the data or model the data more appropriately? What we can do is we can add something called an interaction. Okay? And I'm going to change this to be EEG dot lm to. And we model this by simply saying sex, no colon. Treatment. So this is how we model an interaction between two different factors. We'll list both of the factors. But then we put a colon in between them. And that colon says that we want to consider an interaction between these two variables. So now our linear model considers three different terms. And effective sex and effective treatment. And an interaction where what this interaction allows us, allows the model to do is it allows the effect of sex to depend on the level of treatments that we're looking at, and vice versa. It allows effective treatment to be different between our different levels of sex. Okay, so now let's run this model. And we'll say plot EEG dot lm dot two. Now let see what these data look like. Now that looks nicer. So again, this first plot allows us to assess the assumption of equal variance. And you can see that we have our four columns of residuals which correspond to R for treatment combinations. And you can see that in general, the residuals are pretty consistent among our 40 different treatment combinations. So we really don't have much reason to worry about the assumption of equal variance. And this plot which tests normality. This is very nice now. Okay? Okay. So now we've produced an analysis where we're generally satisfied with our assumptions. How do we find our p-values? Well, when our data are balanced in the way I described earlier. So in other words, when your experimental design is such that you have an equal number of data points in each of your treatment combinations than you can use the ANOVA function just like we did in our one factor general linear models in order to obtain our p-values. So we can say anova EEG dot lm, taught too. And here we go. From this output, we can see that we have a statistically significant effect of an interaction. Okay? You can also, from this plot, or sorry, from this output, you can get the f values. It, you'd want to report an F value for the sex factor, an F value for the treatment factor, and an F value for our interaction. We also get the degrees of freedom for each of these terms. A degrees of freedom for the sex factor, degrees freedom for the treatment factor, and a degrees of freedom for. The interaction and we get degrees of freedom for our residuals. These are all things that we want to report when we were reporting their results. That's something I want to highlight though in this output, which is how to think about these results when we have a significant interaction, the way we do here will have a significant interaction. What that means is that the effect of one of our factors. So the effective Sx, for example, depends on the level that we're considering for our other factor being treatments. And this matches what we saw when we looked at our box plot originally. When we looked at our box plot originally, we said, well, it looks like the size of the effect of sex will depend on which treatment we're looking at. So dependent whether or not we're looking at the control group or the treatment group. This significant interaction term tells us that our original intuition is supported by formal statistics. It says, yes. This tells us that the effective sex depends or the size the effective sex depends on which treatment were looking like looking at. It also tells us that the reverse is also true. It also says the effective treatment. So the size, the effect of drug versus control will depend on which sex we're looking at. I'm not gonna highlight that in this video, but that reverse way of looking at these data will also be true. Okay? So that's what the significant effect to treatment means. Because we have a significant interaction that tells us. Q other things. First of all, it tells us that the effective sex itself does influence our data. Okay? It also tells us the effect of treatment influences our data. In other words, if you have a significant interaction that implies that each of the factors that are involved in this interaction also have an influence. Or we have evidence that each of the factors that are involved in this interaction will influence our data. So the fact that we have a significant sex by treatment interaction, this tells us that SEX influences our data and treatment influences our data or to speak more generally, it says that we have reason to believe sex and treatment each influence our data. There's a but however, the but is that although we've learned that sex and treatment, both these factors influence our data. Or the but is that because we know that the effect of sex depends on the effective treatment and vice versa. The affected treatment depends on our level of sex. That means that we cannot. Interpret these, these effects here, which I call the main effects of sex and treatment. We cannot interpret them on their own. So regardless of what their P-values tell us here, we cannot interpret the effective sex on its own or the effect of treatment on its own. Because this significant interaction tells us that we cannot interpret the effect of sex on its own in any simple way. Because the effect of sex will depend on the level of treatment that we're considering, and vice versa. Effective treatment will depend on which sex we're considering. Okay. So I spent a lot of words to tell us that if we have a significant interaction, then that means that we cannot interpret the p-values for each of our main effects for the factors that are involved in this interaction. So if we have a significant sex by treatment interaction, that means that we should not try to interpret the p-values for each of these other terms, which I call the main effects. The main effect of sex and the main effect of treatment. Okay? So that's our general interpretation of our p-values. A general interpretation is focused on this interaction. This p-value tells us that yes, we have reason to believe that SEX influences our data. And yes, we reasonably the treatment influences our data. And moreover, this interaction tells us that we have reason to believe that the size, the effective sex depends on the level of treatment we're looking at, and vice versa. The size effect to treatments defend depends on which sex we're looking at. We now want to understand that interaction in a bit more detail. I'm just going to introduce a few other ways of doing that. The first is I want to show you the function interaction plot. To use the function interaction plot, we need to do what's called attach our DataFrame. This is not something that I generally recommend. But when we attach the dataframe, that means that we can simply list the variables or list the column headings in that DataFrame without having to tell a function which DataFrame we're talking about. In some ways that can make things easier. But in other cases that can make things a bit dangerous. Because if we have multiple DataFrames, if we're working with multiple DataFrames in a single analysis and we've attached one of them, then we could end up accidentally referring to columns in a DataFrame. You don't actually want to. The point here is, I'm going to attach this DataFrame for this particular analysis. But then I'm going to undo that immediately because attaching a DataFrame can cause confusion that we want to avoid. So we're gonna use a function interaction plot. Add. When we use this function, what we do first is we specify the name of the variable that we want to have on our x-axis. So I'm just going to say, let's say we'll have sex on our X axis. And we'll have treatment separated within the main plot that will not make a whole lot of sense until I show you what the plot looks like. And then I'm going to say type equals b. And that's just to say that i, we can say type equals p for points, or L for lines, or B for both. And so I want to create a plot that gives both points and lines. And I'm going to say legend equals true. Okay? And then you say detach. Eeg. Okay, now let's just submit all of this. Whoops, oops, I forgot something else, which is the name of the variable. That's a third thing you have to give. Pardon me? It's good to make these kinds of mistakes. Sometimes. There we go. So what it's plotting is the mean of y, which is the third thing we have to list in the interaction plot. I just forgot to do that the first time. Along the x-axis we're indicating whether or not we have females or males. And within the, within the main plot, we've separated the means for the drug treatment versus the control treatment. So this is really just another way of plotting the data that we saw in our initial boxplot. And what we can see here is that for females, we have a large difference between the drug and the control treatment. Where as for males, we have a relatively small difference between the drug and the control treatment. Okay? I'm, I said earlier that we can, that this perspective is also true from the opposite perspective. So let's actually create this plot again, but let's swap what goes on the x axis. So less just input the 0s. Okay? So here's another way of looking at this interaction. I said before wasn't consider this alternate point of view, but I changed my mind. So along the x axis you can see we're either looking at the control treatment or the drug treatment. But within this plot, we've separated the data for females versus males. And what we can see here is that in the control treatment. Females tend to be larger than males. But in the drug treatment we can see that's also true, but we can see the difference between females and males is much greater in the drug treatment than we see in the control treatment. So I was talking earlier about this interaction being due to the fact that the difference between females and males is much greater for the control treatment than, sorry, is much greater for the drug treatment than the control treatment. That's, it's what's borne out in this plot. Went back to our first interaction plot. This is an alternate way of looking at the data. We can see that the size of the effect of the drug also depends on which sex we're looking at. Okay? So the size, the effective drug is much greater in females than it is in males. Okay? So going back here to our interaction, discussion of interaction, these interaction plots really clarify the fact that the effective sex depends on the effective treatment. But likewise, effective treatment depends on which level of sex but looking at, okay, so that basically explains the reason for our interaction. We can see that the effective sex depends on which. We can see that the effect of sex depends on which chicken we're looking at and vice versa. When we're reporting results like this, we also want to be able to report effect sizes. And to do that, just like, just like with our one factor generally or model, we're going to use e, the EM means library. So we'll say library. Em means, say run. Okay, there we go. And I want to show you what, what happens. If we run the EM means function in a way that we normally would have just with a single factor general linear model. Let's say example dot EM. Em means. So. So when we did a one factor, generally tomato, we just gave the name of our output from our LM function, which was EEG dot lm dots. To remember we want the model that includes the interaction, which was that model. And then we would just name the particular treatment that we're interested in. Okay. So in this case, let's just say sex. Look at this warning that we're getting. It says Note, results be built, maybe misleading due to involvement in interactions. Let's see it we've done here. So what we've done by a coding this in the way that we, in the way we would have done with a single fact channeling your model is we got, I'm an EM mean. So what's called an estimated marginal mean for females and for males with standard error associated with each of them and with confidence intervals. Remember though that this experiment had two factors that had sex and also treatment. And so what the output is telling us here is that the EM means function has produced these means for females and males by averaging over the level of treatment. Okay. That's what the EM that refers to were taking. We're averaging over other factors in our model in a way that is appropriate. But let's stop and think about this for a minute. Doesn't actually make sense to just take the average effect of females and males. An experiment like this? Well, probably not, because we know that the effective females and males really depends on which particular drug treatment we're looking at. So this way of looking at the data is probably not appropriate. And that's probably why we're getting this warning from the EM means function. So we're going to do something different. Okay? Well we can do, is, we can say EM, We're going to do this, but we're going to ask for EM means to give us the means separately for each level of treatment. And we can do that by saying By equals. By equals. Treatment will say dot by T. Just to remind us that we're doing again these Ye means by treatment. Now we'll run that. And now let's see we get, okay. Now you can see that we're getting different results for females and males, but listed separately for when the control, for when the treatment is to control versus when the treatment is the drug. Okay? Now, let's imagine that we wanted to know how females and males differ from one another. And want to make that comparison separately for the control treatment versus the drug treatment. Okay, we can do that simply by saying. Pairs. Now of that output from EM means. And we'll say E G dot pairs and by t. We can kind of just to remind us what this means. And I will say run. Now we can look at these results. Run that. And now you can see that we can get our contrast just like we did with our one fact Italia models. But we're getting separate results for each level of treatment. So we're comparing females versus males and the control treatment. Here we get our estimate and our standard error for that estimate. So these are, these our, these are our estimates of our effect size of sex in the control treatment. And so we can report this estimate as the effect size of sex in this context, when the context is in the control treatment. And we can report the standard error for this, which is useful if someone conducted a meta-analysis with our data. Include the standard error for this effect size that can be useful for anyone that might want to conduct a meta-analysis with our data. Likewise, we get a standard error and an estimate for the difference between females and males when they're in the drug treatment. Okay, so this is our estimate of our effect size. And we can see from this that the effect size is drastically different between the control treatment and the drug treatment. You can see that the difference between females and males is three times larger in the drug treatment compared to the control treatment. Let's imagine you want to get confidence intervals for this. Then we can produce them in exactly the same ways we did earlier. So I really, I mean with a one factor generally model. So we can just say confint and then use the output from our Paris function. And here we go. Now we're getting confidence interval for our effect sizes for comparing females versus males, either in the control treatment or in the drug treatment. Now, I'll just end by pointing out that we could have looked at these data. Also from the perspective of saying we want to look at the effect of treatment, but do it by sex. In which case we'd say by sex and these output. Ok. So in this case. We're going to get similar output, but by sex. So now we have mean values for the control and the drug, but separated by sex. So we have these the means for when the sex is female. And these the means when the sex is male. And then we could go on and you, our pairwise comparisons generate confidence intervals just like we did previously. Okay? We're going to stop our analysis there except to say that this demonstration of how to use EM means in this experiment is relatively straightforward because we only have two func, we only have two factors. I want to point out that we need to be more careful when using EM means to create these kinds of contrasts. Especially if we have more than two factors in our analysis. And so I really, I'm not gonna go into that at this point, but I want to point you to an appropriate website to learn about that. So if we go to our seek, so just our Seek.org. And then we just search for means, interaction or sorry, EM means and interaction than this first output here gives us this webpage. And this webpage talks in great detail and very clearly I think, about how we can use EM means in order to understand interactions. And the example they use involves a three factor general linear model. Okay? So please, if you're using a more complicated design, please go and read some more about this. Em means provides excellent documentation to help us. Okay, so that is our analysis. Let's wrap up our discussion just with some general comments about what interactions do and a multi-factor model. What interactions allow us to do is they allow the effects of one factor to be different among the levels of another factor. Okay, that's essentially what the interactions allow our LM function to do. What that means is that if a model leaves out into an interaction as we did when we initially analyzed our data. Then that means that we're left to assume that the effects of one factor do not depend on the effects of another factor. And we will look at some examples where that's true in some future videos. Ok, so that's what an interaction does when we include it in a model. We now want to ask, when can we actually model and interaction? We cannot model interactions for all experimental designs. In order for us to model an interaction, an experimental design has to have an appropriate level of replication. What I mean by that, let's consider this experimental design on the left. In this case, we can imagine an experiments where we have females and males. And now instead of having a drug with a, sorry, a treatment with either drug or control, now we're imagining an experiment. We have a treatment which we call either low or high. If we wanted to model an interaction between sex and treatment, then we need to have at least two data points for each combination of for each combination of our two factors are factors sex and our factor treatment. And you can see that we have that here we have at least two measurements are two data points for the factor for the combination of high and female, high and male, low and female, a low and male. So for this data set, It's a pretty pathetic dataset since it has so few data points. But technically, we would be able to model and interaction. And we can also model the main effects of sex and treatment. That's not true in this experiment which you have on the right here. Where in this case, we have data for all combinations of our treatment levels. So we have data for all combinations of high and low and females and males. But we do not have replication for all combinations. In fact, we don't have replication for any of the combinations, okay? And so because we only have one data point in each of these combinations, we can not effectively model the interaction. In order to effectively model the interaction we want have replication for all of our different treatment combinations. So for this experimental design on the right, we can not model the interaction in an experiment like this, but we can model the main effects of sex and treatments. In other words, we could include a model where we said LM and then Sx plus treatment, but we could not include plus interaction between treatment and sex. Okay? So there are some data sets where we can model in interaction and others with the data sets will not allow us to model the interaction. So when you're deciding whether or not to model an interaction of a particular data set, you need to look at the replication. So in that last slide we've dealt with when can we model and interaction. In this last slide, we're going to deal with when should we model interaction? There are few cases where I would argue you should model and interaction. And in the first case. That is when your research question specifically addresses in interaction. So for example, if our research question was, does one sex respond more to a change in drug concentration? So that would corresponds to this experimental design where we either have a high or low dosage of a particular drug. If we want to know whether or not the effect of one factor depends on the effective another. That is a research question that focuses on an interaction. So of course, we should include interaction in our model. A second reason for modeling interaction is simply to be able to model the data. Well. We saw that in our toy dataset where excluding an interaction meant that we could not model the data well, our residuals were pretty terrible. And we found that when we modeled the data well, we could, sorry, when we include an interaction, we can model the data much more appropriately. Some people argue that if an experimental design includes the opportunity for looking at interaction than you should include it. Not everyone feels this way. My general feeling is yes, I agree with this perspective. If you design an experiment where the experimental design includes the potential furniture action that I would say yes, you should include an interaction in your in your model. Not everyone feels that way, however, feel I should mention that and mentioned that not everyone has the same opinion on this. There's also the question of what we should do with an interaction. If we model interaction, then find that it's not significant or not statistically significant. Some people would argue that you should then remove the interaction. This is, and then run your model again. Sorry, I didn't finish my sentence. So some people would argue that if you have an interaction being nonsignificant, than they might argue that you should remove that nonsignificant interaction if your model, and then look at your are and then run, run your model again. This is debatable. And I'm going to produce a video that looks at what's called model selection. So basically deciding which model among a series of models would be most appropriate for your data. So I'm not going to say much more on this now. I'm just going to have to say, be patient and watch out for that video in the future. Ok. So this video now has walked through how to model interaction. How to try to understand the biology that interaction by looking for the effect sizes that are involved in an interaction. We've considered when you can model interaction based on the experimental design. And then finally, you talked about when should you model an interaction? That's a lot about interactions. I'm going to stop the video there. We've gone on for a long time and I'll say thank you very much for your patience. And thank you very much.