Assignment(statistics8).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af7b42aa-f6d7-4875-9430-5e641f46c00e",
   "metadata": {},
   "source": [
    "# Q1. Explain the assumptions required to use ANOVA and provide examples of violations that could impact the validity of the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bb86b72-1bc2-487c-924d-ceaa95cb8bcf",
   "metadata": {},
   "source": [
    "ANOVA (Analysis of Variance) is a statistical test used to determine whether there are any significant differences between means of two or more groups. The basic assumptions of ANOVA are as follows:\n",
    "\n",
    "Normality: The data should be normally distributed within each group.\n",
    "\n",
    "Homogeneity of variance: The variance of the data in each group should be approximately equal.\n",
    "\n",
    "Independence: The observations in each group should be independent of each other.\n",
    "\n",
    "If any of these assumptions are violated, the results of the ANOVA may not be valid. Here are some examples of violations that could impact the validity of the results:\n",
    "\n",
    "Non-normality: If the data in any of the groups is not normally distributed, this can affect the validity of the ANOVA results. For example, if the data in one group is skewed, the mean of that group may not be a good representation of the data.\n",
    "\n",
    "Heterogeneity of variance: If the variance of the data in each group is significantly different, this can affect the validity of the ANOVA results. For example, if one group has a much larger variance than the other groups, this can make it more difficult to detect differences between means.\n",
    "\n",
    "Dependence: If the observations in any of the groups are not independent, this can affect the validity of the ANOVA results. For example, if the observations in one group are paired (e.g., before and after measurements), this violates the independence assumption and can lead to spurious results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7561e734-b7b6-4db1-9036-fbb8942d8224",
   "metadata": {},
   "source": [
    "# Q2. What are the three types of ANOVA, and in what situations would each be used?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e061e4-62ce-4038-a55c-9d9c12f5a732",
   "metadata": {},
   "source": [
    "There are three types of ANOVA:\n",
    "\n",
    "1. One-way ANOVA: This type of ANOVA is used when there is only one independent variable, which has three or more levels (or groups). One-way ANOVA is used to determine if there are any significant differences between the means of the different levels/groups. For example, one-way ANOVA can be used to compare the average weight loss among three different diet groups.\n",
    "\n",
    "2. Two-way ANOVA: This type of ANOVA is used when there are two independent variables, both of which have two or more levels. Two-way ANOVA is used to determine if there are any significant main effects (i.e., the effect of each independent variable on the dependent variable) and/or interaction effects (i.e., the combined effect of the two independent variables on the dependent variable). For example, two-way ANOVA can be used to compare the average sales of two different products (product A and product B) in two different regions (North and South).\n",
    "\n",
    "3. MANOVA (Multivariate Analysis of Variance): This type of ANOVA is used when there are two or more dependent variables (i.e., outcome variables) and one or more independent variables. MANOVA is used to determine if there are any significant differences between the means of the dependent variables across the different levels of the independent variable. For example, MANOVA can be used to compare the average scores on multiple personality traits (e.g., extroversion, agreeableness, neuroticism) between different age groups (young, middle-aged, and old)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83baaa3-2828-47e0-962a-b94d8f050678",
   "metadata": {},
   "source": [
    "# Q3. What is the partitioning of variance in ANOVA, and why is it important to understand this concept?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f09552e-cbfb-460c-8d5a-07af45a18f47",
   "metadata": {},
   "source": [
    "Partitioning of variance in ANOVA (Analysis of Variance) refers to the process of decomposing the total variation in the data into different sources of variation. In other words, ANOVA breaks down the total variation in the dependent variable into the variation explained by the independent variable(s) and the variation that is not explained by the independent variable(s). This is important because it helps us understand how much of the variation in the dependent variable can be attributed to the independent variable(s).\n",
    "\n",
    "The partitioning of variance in ANOVA is typically represented by the following equation:\n",
    "\n",
    "Total variation = Variation explained by independent variable(s) + Variation not explained by independent variable(s)\n",
    "\n",
    "The variation explained by the independent variable(s) is referred to as the \"between-group\" variation, while the variation not explained by the independent variable(s) is referred to as the \"within-group\" variation.\n",
    "\n",
    "Understanding the partitioning of variance is important in ANOVA because it helps us determine if there is a significant difference between the means of the different groups being compared. If the variation explained by the independent variable(s) (i.e., the between-group variation) is much larger than the variation not explained by the independent variable(s) (i.e., the within-group variation), then it suggests that there is a significant difference between the means of the groups being compared."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9502060f-e525-4525-b74f-c6b7a4354ef0",
   "metadata": {},
   "source": [
    "# Q4. How would you calculate the total sum of squares (SST), explained sum of squares (SSE), and residual sum of squares (SSR) in a one-way ANOVA using Python?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1e64e59f-3be1-4cce-9cba-99ad1f72d3d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SST = 223.33333333333337\n",
      "SSE = 223.33333333333337\n",
      "SSR = 0.0\n"
     ]
    }
   ],
   "source": [
    "import scipy.stats as stats\n",
    "\n",
    "# create three sample groups\n",
    "group1 = [10, 12, 14, 16, 18]\n",
    "group2 = [8, 11, 14, 17, 20]\n",
    "group3 = [9, 12, 15, 18, 21]\n",
    "\n",
    "# concatenate the groups\n",
    "data = group1 + group2 + group3\n",
    "\n",
    "# calculate the mean of the data\n",
    "mean = sum(data) / len(data)\n",
    "\n",
    "# calculate the total sum of squares (SST)\n",
    "SST = sum([(x - mean)**2 for x in data])\n",
    "\n",
    "# calculate the sum of squares between (SSB)\n",
    "SSB = len(group1) * (sum([(x - mean)**2 for x in group1]) / len(group1))\n",
    "SSB += len(group2) * (sum([(x - mean)**2 for x in group2]) / len(group2))\n",
    "SSB += len(group3) * (sum([(x - mean)**2 for x in group3]) / len(group3))\n",
    "\n",
    "# calculate the explained sum of squares (SSE)\n",
    "SSE = SSB\n",
    "\n",
    "# calculate the residual sum of squares (SSR)\n",
    "SSR = SST - SSE\n",
    "\n",
    "print(\"SST =\", SST)\n",
    "print(\"SSE =\", SSE)\n",
    "print(\"SSR =\", SSR)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "419c1808-13c2-4904-a2ea-798e38f11c75",
   "metadata": {},
   "source": [
    "# Q5. In a two-way ANOVA, how would you calculate the main effects and interaction effects using Python?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be628434-01bf-4a38-8b41-915af4b641bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "# Load data into a pandas dataframe\n",
    "data = pd.read_csv('data.csv')\n",
    "\n",
    "# Fit the two-way ANOVA model\n",
    "model = ols('dependent_variable ~ independent_variable_1 + independent_variable_2 + independent_variable_1 * independent_variable_2', data=data).fit()\n",
    "\n",
    "# Calculate the main effects\n",
    "main_effects = sm.stats.anova_lm(model, typ=1)\n",
    "print(main_effects)\n",
    "\n",
    "# Calculate the interaction effect\n",
    "interaction_effect = sm.stats.anova_lm(model, typ=2)\n",
    "print(interaction_effect)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b6f4343-dd33-4138-85c1-9964f6e93f5c",
   "metadata": {},
   "source": [
    "# Q6. Suppose you conducted a one-way ANOVA and obtained an F-statistic of 5.23 and a p-value of 0.02. What can you conclude about the differences between the groups, and how would you interpret these results?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69482d1e-9baa-4190-823f-7192f2cc5955",
   "metadata": {},
   "source": [
    "If we conducted a one-way ANOVA and obtained an F-statistic of 5.23 and a p-value of 0.02, we can conclude that there is at least one significant difference between the means of the groups being compared. The F-statistic tells us the ratio of the variance between the groups to the variance within the groups. A larger F-statistic indicates a greater difference between the group means compared to the variation within the groups. The p-value tells us the probability of obtaining a result as extreme as the observed result if there were no true differences between the groups.\n",
    "\n",
    "With a p-value of 0.02, we can interpret this result as follows: if there were no true differences between the groups, we would only expect to obtain a result as extreme as an F-statistic of 5.23 or higher by chance 2% of the time. Therefore, we reject the null hypothesis (that there are no true differences between the groups) and conclude that there is sufficient evidence to suggest that at least one of the group means is different from the others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "746e43de-20a4-4ecf-8e8b-cafeb6157a45",
   "metadata": {},
   "source": [
    "# Q7. In a repeated measures ANOVA, how would you handle missing data, and what are the potential consequences of using different methods to handle missing data?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff342101-2683-400c-89a3-8063db1e077c",
   "metadata": {},
   "source": [
    "In a repeated measures ANOVA, missing data can be handled in several ways:\n",
    "\n",
    "1. Listwise deletion: This method involves excluding any cases with missing data from the analysis. This can be done using the dropna() function in pandas. While this approach is simple, it may result in a loss of statistical power if a large amount of data is missing.\n",
    "\n",
    "2. Mean imputation: This method involves replacing missing values with the mean of the non-missing values. This can be done using the fillna() function in pandas. While this approach is simple and easy to implement, it may underestimate the variability of the data and result in biased estimates.\n",
    "\n",
    "3. Last observation carried forward (LOCF): This method involves imputing missing values with the last observed value. This can be done using the fillna(method='ffill') function in pandas. While this approach is useful for data with a temporal order, it may not be appropriate for all situations and may result in biased estimates.\n",
    "\n",
    "4. Multiple imputation: This method involves imputing missing values multiple times using a statistical model, and then combining the results to obtain estimates and standard errors. This can be done using the fancyimpute library in Python. While this approach is more sophisticated and can produce more accurate estimates than mean imputation or LOCF, it is computationally intensive and requires careful consideration of the underlying assumptions.\n",
    "\n",
    "The potential consequences of using different methods to handle missing data in a repeated measures ANOVA include bias in the estimated means, standard errors, and effect sizes, as well as a loss of statistical power. It's important to carefully consider the underlying assumptions and potential limitations of each method and choose the approach that is most appropriate for the specific dataset and research question. Additionally, it may be beneficial to conduct sensitivity analyses to assess the robustness of the results to different methods of handling missing data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4c87991-e2e0-44b6-a06f-41d11070e6e8",
   "metadata": {},
   "source": [
    "# Q8. What are some common post-hoc tests used after ANOVA, and when would you use each one? Provide an example of a situation where a post-hoc test might be necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4919ea03-f55d-4948-891f-f66377f3362d",
   "metadata": {},
   "source": [
    "Post-hoc tests are used after ANOVA to make pairwise comparisons between groups when the overall ANOVA result is statistically significant. The purpose of post-hoc tests is to determine which specific groups differ from each other and to control for the familywise error rate, which is the probability of making at least one Type I error (false positive) across all the pairwise comparisons. Here are some common post-hoc tests used after ANOVA, along with an example of a situation where each one might be necessary:\n",
    "\n",
    "1. Tukey's HSD (honestly significant difference) test: This test is a conservative post-hoc test that is commonly used when the sample sizes are equal across groups. It controls for the familywise error rate by adjusting the significance level for each pairwise comparison. For example, if we have four groups (A, B, C, D), and the overall ANOVA result is significant, we might use Tukey's HSD test to determine which specific groups differ from each other. If the test shows that group A is significantly different from group B and group C, but not group D, we can conclude that group A is significantly different from groups B and C, but not group D.\n",
    "\n",
    "2. Bonferroni correction: This test is a simple and commonly used method to adjust the significance level for each pairwise comparison. It divides the significance level (usually 0.05) by the number of comparisons being made. For example, if we have four groups (A, B, C, D), and the overall ANOVA result is significant, we might use the Bonferroni correction to determine which specific groups differ from each other. If the test shows that group A is significantly different from group B, group C, and group D, we can conclude that group A is significantly different from all the other groups.\n",
    "\n",
    "3. Dunnett's test: This test is used when we have one control group and several treatment groups. It compares each treatment group to the control group, while controlling for the overall familywise error rate. For example, if we have one control group and three treatment groups (A, B, C), and the overall ANOVA result is significant, we might use Dunnett's test to determine which specific treatment groups differ from the control group. If the test shows that group A is significantly different from the control group, but groups B and C are not significantly different from the control group, we can conclude that group A is significantly different from the control group, but groups B and C are not.\n",
    "\n",
    "4. Scheffe's test: This test is a conservative post-hoc test that is used when the sample sizes are unequal across groups. It controls for the familywise error rate by adjusting the significance level for each pairwise comparison. For example, if we have four groups (A, B, C, D), and the overall ANOVA result is significant, we might use Scheffe's test to determine which specific groups differ from each other. If the test shows that group A is significantly different from group B and group C, but not group D, we can conclude that group A is significantly different from groups B and C, but not group D."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44b4f4d4-abf1-43d1-a5a0-0f042d0af627",
   "metadata": {},
   "source": [
    "# Q9. A researcher wants to compare the mean weight loss of three diets: A, B, and C. They collect data from 50 participants who were randomly assigned to one of the diets. Conduct a one-way ANOVA using Python to determine if there are any significant differences between the mean weight loss of the three diets. Report the F-statistic and p-value, and interpret the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "53a44246-a92c-4598-9c78-88d318da20ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F-statistic: 176.89689491604346\n",
      "p-value: 1.63227843737611e-28\n"
     ]
    }
   ],
   "source": [
    "import scipy.stats as stats\n",
    "\n",
    "# Sample data\n",
    "diet_A = [4.2, 5.1, 3.7, 6.2, 4.9, 2.8, 5.4, 4.7, 3.9, 4.1,\n",
    "          3.3, 5.5, 4.8, 5.3, 3.9, 6.1, 4.3, 3.5, 5.2, 5.0,\n",
    "          5.6, 3.8, 5.8, 4.0, 5.7]\n",
    "diet_B = [2.9, 1.9, 2.5, 2.3, 3.2, 2.7, 1.8, 3.1, 2.6, 3.0,\n",
    "          2.4, 2.0, 2.8, 2.2, 2.1, 2.6, 2.7, 1.6, 2.4, 2.9,\n",
    "          3.0, 1.8, 2.1, 2.3, 2.6]\n",
    "diet_C = [1.3, 0.9, 1.1, 1.6, 1.4, 1.8, 1.2, 1.5, 2.0, 1.4,\n",
    "          1.7, 1.3, 1.9, 1.0, 1.6, 1.2, 1.8, 1.4, 1.1, 1.7,\n",
    "          1.5, 1.6, 1.9, 1.2, 1.4]\n",
    "\n",
    "# One-way ANOVA\n",
    "f_statistic, p_value = stats.f_oneway(diet_A, diet_B, diet_C)\n",
    "\n",
    "# Print results\n",
    "print(\"F-statistic:\", f_statistic)\n",
    "print(\"p-value:\", p_value)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e8f11c3-a67b-432f-9e6f-a79b296326aa",
   "metadata": {},
   "source": [
    "# Q10. A company wants to know if there are any significant differences in the average time it takes to complete a task using three different software programs: Program A, Program B, and Program C. They randomly assign 30 employees to one of the programs and record the time it takes each employee to complete the task. Conduct a two-way ANOVA using Python to determine if there are any main effects or interaction effects between the software programs and employee experience level (novice vs. experienced). Report the F-statistics and p-values, and interpret the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aa4bc26-2f2c-4d28-b979-ada1577226fe",
   "metadata": {},
   "source": [
    "To conduct a two-way ANOVA in Python, we first need to import the necessary packages and read in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a898fc87-03c6-4c96-84e6-029be6287471",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "data = pd.read_csv(\"task_completion_times.csv\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc37fbd-4b8b-4f92-bb75-adc944a93639",
   "metadata": {},
   "source": [
    "Next, we can use the ols function from the statsmodels.formula.api module to create a model formula for the two-way ANOVA:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffb7b4a7-c40f-49c1-a896-914392016a8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ols('Time ~ C(Program) + C(Experience) + C(Program):C(Experience)', data=data).fit()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce805da1-fb93-4a61-8f9a-53b2081ac97b",
   "metadata": {},
   "source": [
    "Here, we're using the C function to indicate that Program and Experience are categorical variables, and : to indicate the interaction term between Program and Experience.\n",
    "\n",
    "We can then use the anova_lm function from the statsmodels.api module to obtain the ANOVA table and calculate the F-statistics and p-values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "865fa1af-23aa-42a8-834f-7c6a5c0e5e35",
   "metadata": {},
   "outputs": [],
   "source": [
    "anova_table = sm.stats.anova_lm(model, typ=2)\n",
    "print(anova_table)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b334100b-fe9c-4919-9177-7fe8b2fcf97b",
   "metadata": {},
   "source": [
    "# Q11. An educational researcher is interested in whether a new teaching method improves student test scores. They randomly assign 100 students to either the control group (traditional teaching method) or the experimental group (new teaching method) and administer a test at the end of the semester. Conduct a two-sample t-test using Python to determine if there are any significant differences in test scores between the two groups. If the results are significant, follow up with a post-hoc test to determine which group(s) differ significantly from each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6902472d-3a99-4305-9887-f9cd9638a435",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To conduct a two-sample t-test in Python, we first need to import the necessary packages and read in the data:\n",
    "import pandas as pd\n",
    "from scipy.stats import ttest_ind\n",
    "\n",
    "data = pd.read_csv(\"test_scores.csv\")\n",
    "\n",
    "#The data should be in a CSV file with two columns: \"Group\" (control or experimental) and \"Score\" (the test score).\n",
    "#We can then use the ttest_ind function from the scipy.stats module to conduct a two-sample t-test:\n",
    "\n",
    "\n",
    "control_scores = data[data[\"Group\"] == \"control\"][\"Score\"]\n",
    "experimental_scores = data[data[\"Group\"] == \"experimental\"][\"Score\"]\n",
    "t, p = ttest_ind(control_scores, experimental_scores)\n",
    "print(\"t = {:.2f}, p = {:.4f}\".format(t, p))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94666faf-bb45-4588-9730-f2e0165ea8ff",
   "metadata": {},
   "source": [
    "# Q12. A researcher wants to know if there are any significant differences in the average daily sales of three retail stores: Store A, Store B, and Store C. They randomly select 30 days and record the sales for each store n those days. Conduct a repeated measures ANOVA using Python to determine if there are any significant differences in sales between the three stores. If the results are significant, follow up with a post- h c test to determine which store(s) differ significantly from each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc633068-9211-458b-928e-c127f3792b65",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "# Create a dataframe with sales data\n",
    "sales_data = {'Store': ['A']*30 + ['B']*30 + ['C']*30, \n",
    "              'Day': list(range(1, 31))*3, \n",
    "              'Sales': [100, 120, 130, 110, 130, 140, 90, 110, 120, 80,\n",
    "                        100, 130, 140, 120, 140, 150, 110, 130, 140, 100,\n",
    "                        90, 110, 120, 100, 120, 130, 80, 100, 110, 70,\n",
    "                        80, 100, 110, 90, 110, 120, 70, 90, 100, 60,\n",
    "                        70, 90, 100, 80, 100, 110, 60, 80, 90, 50,\n",
    "                        60, 80, 90, 70, 90, 100, 50, 70, 80, 40,\n",
    "                        50, 70, 80, 60, 80, 90, 40, 60, 70, 30,\n",
    "                        40, 60, 70, 50, 70, 80, 30, 50, 60, 20,\n",
    "                        30, 50, 60, 40, 60, 70, 20, 40, 50, 10,\n",
    "                        20, 40, 50, 30, 50, 60, 10, 30, 40, 0]}\n",
    "\n",
    "sales_df = pd.DataFrame(sales_data)\n",
    "\n",
    "# Conduct one-way repeated measures ANOVA\n",
    "rm_anova = sm.stats.anova.AnovaRM(data=sales_df, depvar='Sales', subject='Day', \n",
    "                                  within=['Store'], aggregate_func='mean')\n",
    "result = rm_anova.fit()\n",
    "\n",
    "# Print ANOVA table\n",
    "print(result.summary())\n",
    "\n",
    "# Conduct post-hoc tests using Tukey's HSD test\n",
    "from statsmodels.stats.multicomp import MultiComparison\n",
    "\n",
    "comp = MultiComparison(sales_df['Sales'], sales_df['Store'])\n",
    "tukey_result = comp.tukeyhsd()\n",
    "\n",
    "# Print post-hoc results\n",
    "print(tukey_result.summary())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c83ca9c-55a0-4bca-9832-8a07e54317b6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}