test_llama_index.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex\n",
    "from llama_index import download_loader\n",
    "#os.environ['OPENAI_API_KEY'] = 'OPEN AI API KEY HERE'\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading in ISLR as an txt file\n",
    "- https://gpt-index.readthedocs.io/en/latest/guides/primer/usage_pattern.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_text = SimpleDirectoryReader('data/').load_data()\n",
    "index_txt = GPTVectorStoreIndex.from_documents(document_text)\n",
    "query_engine = index_txt.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Response(response='\\nLinear regression is a statistical technique used to model the relationship between a dependent variable (the response) and one or more independent variables (the predictors). It is used to predict the value of the response based on the values of the predictors. The model is linear in the sense that it assumes a linear relationship between the response and the predictors. The model is fit by minimizing the sum of the squared residuals, which is the difference between the observed response values and the predicted response values. The coefficients of the model are estimated using the least squares criterion.', source_nodes=[NodeWithScore(node=Node(text='. . . ,n. In other words, we want to find an intercept ˆ β0 and a slope ˆ β1 such\\nthat the resulting line is as close as possible to the n = 200 data points.\\nThere are a number of ways of measuring closeness. However, by far the\\nmost common approach involves minimizing the least squares criterion,\\nleast squares\\nand we take that approach in this chapter. Alternative approaches will be\\nconsidered in Chapter 6.\\nLet ˆyi = ˆ β0 + ˆ β1xi be the prediction for Y based on the ith value of X.\\nThen ei = yi−ˆyi represents the ith residual—this is the difference between\\nresidual\\n62 3. Linear Regression\\n0 50 100 150 200 250 300\\n5 10 15 20 25\\nTV\\nSales\\nFIGURE 3.1. For the Advertising data, the least squares fit for the regression\\nof sales onto TV is shown. The fit is found by minimizing the residual sum of\\nsquares. Each grey line segment represents a residual. In this case a linear fit\\ncaptures the essence of the relationship, although it overestimates the trend in the\\nleft of the plot.\\nthe ith observed response value and the ith response value that is predicted\\nby our linear model. We define the residual sum of squares (RSS) as\\nresidual sum\\nof squares RSS = e21\\n+ e22\\n+ · · · + e2\\nn,\\nor equivalently as\\nRSS = (y1− ˆ β0− ˆ β1x1)2+(y2− ˆ β0− ˆ β1x2)2+· · ·+(yn− ˆ β0− ˆ β1xn)2. (3.3)\\nThe least squares approach chooses ˆ β0 and ˆ β1 to minimize the RSS. Using\\nsome calculus, one can show that the minimizers are\\nˆ β1 =\\n)n\\ni=1)(xi − x¯)(yi − y¯) n\\ni=1(xi − ¯x)2 ,\\nˆ β0 = ¯y − ˆ β1¯x,\\n(3.4)\\nwhere ¯y ≡ 1n\\n)n\\ni=1 yi and ¯x ≡ 1n\\n)n\\ni=1 xi are the sample means. In other\\nwords, (3.4) defines the least squares coefficient estimates for simple linear\\nregression.\\nFigure 3.1 displays the simple linear regression fit to the Advertising\\ndata, where ˆ β0 = 7.03 and ˆ β1 = 0.0475. In other words, according to\\nthis approximation, an additional $1,000 spent on TV advertising is associated\\nwith selling approximately 47.5 additional units of the product. In\\n3.1 Simple Linear Regression 63\\nβ0\\nβ1\\n2.11\\n2.15\\n2.2\\n2.3\\n2.5\\n2.5\\n3\\n3\\n5 6 7 8 9\\n0.03 0.04 0.05 0.06\\n●\\nRSS\\nβ1 β0\\nFIGURE 3.2. Contour and three-dimensional plots of the RSS on the\\nAdvertising data, using sales as the response and TV as the predictor. The\\nred dots correspond to the least squares estimates ˆβ0 and ˆβ1, given by (3.4).\\nFigure 3.2, we have computed RSS for a number of values of β0 and β1,\\nusing the advertising data with sales as the response and TV as the predictor.\\nIn each plot, the red dot represents the pair of least squares estimates\\n( ˆ β0, ˆ β1) given by (3.4). These values clearly minimize the RSS.\\n3.1.2 Assessing the Accuracy of the Coefficient Estimates\\nRecall from (2.1) that we assume that the true relationship between X and\\nY takes the form Y = f(X) + ϵ for some unknown function f, where ϵ\\nis a mean-zero random error term. If f is to be approximated by a linear\\nfunction, then we can write this relationship as\\nY = β0 + β1X + ϵ.', doc_id='f1ffc41d-678e-42b1-aa62-3f6d32228432', embedding=None, doc_hash='9144d43a170e3498c7b394a04a369bd769edf7342fbcde6806bc2522d6a228e5', extra_info=None, node_info={'start': 155478, 'end': 158388, '_node_type': <NodeType.TEXT: '1'>}, relationships={<DocumentRelationship.SOURCE: '1'>: 'a2db4d93-478d-41f9-854e-ac00ecdc385e', <DocumentRelationship.PREVIOUS: '2'>: '1a990ac3-9da6-4789-a63a-ed581e847763', <DocumentRelationship.NEXT: '3'>: 'ccda122c-aa3d-4d08-9b8f-246967a2c446'}), score=0.8562624975110771), NodeWithScore(node=Node(text='the data in (a) until you come up with a scenario in\\nwhich the test set MSE is minimized for an intermediate model\\nsize.\\n(f) How does the model at which the test set MSE is minimized\\ncompare to the true model used to generate the data? Comment\\non the coefficient values.\\n(g) Create a plot displaying\\nG)p\\nj=1(βj − ˆ βr\\nj )2 for a range of values\\nof r, where ˆ βr\\nj is the jth coefficient estimate for the best model\\ncontaining r coefficients. Comment on what you observe. How\\ndoes this compare to the test MSE plot from (d)?\\n11. We will now try to predict per capita crime rate in the Boston data\\nset.\\n(a) Try out some of the regression methods explored in this chapter,\\nsuch as best subset selection, the lasso, ridge regression, and\\nPCR. Present and discuss results for the approaches that you\\nconsider.\\n288 6. Linear Model Selection and Regularization\\n(b) Propose a model (or set of models) that seem to perform well on\\nthis data set, and justify your answer. Make sure that you are\\nevaluating model performance using validation set error, crossvalidation,\\nor some other reasonable alternative, as opposed to\\nusing training error.\\n(c) Does your chosen model involve all of the features in the data\\nset? Why or why not?\\n7\\nMoving Beyond Linearity\\nSo far in this book, we have mostly focused on linear models. Linear models\\nare relatively simple to describe and implement, and have advantages over\\nother approaches in terms of interpretation and inference. However, standard\\nlinear regression can have significant limitations in terms of predictive\\npower. This is because the linearity assumption is almost always an\\napproximation, and sometimes a poor one. In Chapter 6 we see that we can\\nimprove upon least squares using ridge regression, the lasso, principal components\\nregression, and other techniques. In that setting, the improvement\\nis obtained by reducing the complexity of the linear model, and hence the\\nvariance of the estimates. But we are still using a linear model, which can\\nonly be improved so far! In this chapter we relax the linearity assumption\\nwhile still attempting to maintain as much interpretability as possible. We\\ndo this by examining very simple extensions of linear models like polynomial\\nregression and step functions, as well as more sophisticated approaches\\nsuch as splines, local regression, and generalized additive models.\\n• Polynomial regression extends the linear model by adding extra predictors,\\nobtained by raising each of the original predictors to a power.\\nFor example, a cubic regression uses three variables, X, X2, and X3,\\nas predictors. This approach provides a simple way to provide a nonlinear\\nfit to data.\\n• Step functions cut the range of a variable into K distinct regions in\\norder to produce a qualitative variable. This has the effect of fitting\\na piecewise constant function.\\n© Springer Science+Business Media, LLC, part of Springer Nature 2021\\nG. James et al., An Introduction to Statistical Learning, Springer Texts in Statistics,\\nhttps://doi.org/10.1007/978-1-0716-1418-1_7\\n289\\n290 7. Moving Beyond Linearity\\n• Regression splines are more flexible than polynomials and step functions,\\nand in fact are an extension of the two. They involve dividing\\nthe range of X into K distinct regions. Within each region, a polynomial\\nfunction is fit to the data. However, these polynomials are\\nconstrained so that they join smoothly at the region boundaries, or\\nknots. Provided that the interval is divided into enough regions, this\\ncan produce an extremely flexible fit.\\n• Smoothing splines are similar to regression splines, but arise in a\\nslightly different situation. Smoothing splines result from minimizing\\na residual sum of squares criterion subject to a smoothness penalty.\\n• Local regression is similar to splines, but differs in an important way.\\nThe regions are allowed to', doc_id='07583cca-8eba-46d0-91ea-605ede74dc22', embedding=None, doc_hash='3f917d2f6f2a158021719f9cdbe0c20702f1784bc55b923bf230cfccd4552c4f', extra_info=None, node_info={'start': 678822, 'end': 682648, '_node_type': <NodeType.TEXT: '1'>}, relationships={<DocumentRelationship.SOURCE: '1'>: 'a2db4d93-478d-41f9-854e-ac00ecdc385e', <DocumentRelationship.PREVIOUS: '2'>: 'aa507997-3a52-4a68-8d43-756cfdae3136', <DocumentRelationship.NEXT: '3'>: 'cbd1f69f-5c10-4508-8501-24f8291c2b05'}), score=0.8509497244575328)], extra_info={'f1ffc41d-678e-42b1-aa62-3f6d32228432': None, '07583cca-8eba-46d0-91ea-605ede74dc22': None})"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query_engine.query(\"Can you give a brief intro to linear regression?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Store the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "index_txt.storage_context.persist(persist_dir=\"./storage\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import StorageContext, load_index_from_storage\n",
    "\n",
    "# rebuild storage context\n",
    "storage_context = StorageContext.from_defaults(persist_dir=\"./storage\")\n",
    "\n",
    "# load index\n",
    "index = load_index_from_storage(storage_context)\n",
    "islr_query = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = islr_query.query(\"Can you give a brief intro to linear regression?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NodeWithScore(node=Node(text='. . . ,n. In other words, we want to find an intercept ˆ β0 and a slope ˆ β1 such\\nthat the resulting line is as close as possible to the n = 200 data points.\\nThere are a number of ways of measuring closeness. However, by far the\\nmost common approach involves minimizing the least squares criterion,\\nleast squares\\nand we take that approach in this chapter. Alternative approaches will be\\nconsidered in Chapter 6.\\nLet ˆyi = ˆ β0 + ˆ β1xi be the prediction for Y based on the ith value of X.\\nThen ei = yi−ˆyi represents the ith residual—this is the difference between\\nresidual\\n62 3. Linear Regression\\n0 50 100 150 200 250 300\\n5 10 15 20 25\\nTV\\nSales\\nFIGURE 3.1. For the Advertising data, the least squares fit for the regression\\nof sales onto TV is shown. The fit is found by minimizing the residual sum of\\nsquares. Each grey line segment represents a residual. In this case a linear fit\\ncaptures the essence of the relationship, although it overestimates the trend in the\\nleft of the plot.\\nthe ith observed response value and the ith response value that is predicted\\nby our linear model. We define the residual sum of squares (RSS) as\\nresidual sum\\nof squares RSS = e21\\n+ e22\\n+ · · · + e2\\nn,\\nor equivalently as\\nRSS = (y1− ˆ β0− ˆ β1x1)2+(y2− ˆ β0− ˆ β1x2)2+· · ·+(yn− ˆ β0− ˆ β1xn)2. (3.3)\\nThe least squares approach chooses ˆ β0 and ˆ β1 to minimize the RSS. Using\\nsome calculus, one can show that the minimizers are\\nˆ β1 =\\n)n\\ni=1)(xi − x¯)(yi − y¯) n\\ni=1(xi − ¯x)2 ,\\nˆ β0 = ¯y − ˆ β1¯x,\\n(3.4)\\nwhere ¯y ≡ 1n\\n)n\\ni=1 yi and ¯x ≡ 1n\\n)n\\ni=1 xi are the sample means. In other\\nwords, (3.4) defines the least squares coefficient estimates for simple linear\\nregression.\\nFigure 3.1 displays the simple linear regression fit to the Advertising\\ndata, where ˆ β0 = 7.03 and ˆ β1 = 0.0475. In other words, according to\\nthis approximation, an additional $1,000 spent on TV advertising is associated\\nwith selling approximately 47.5 additional units of the product. In\\n3.1 Simple Linear Regression 63\\nβ0\\nβ1\\n2.11\\n2.15\\n2.2\\n2.3\\n2.5\\n2.5\\n3\\n3\\n5 6 7 8 9\\n0.03 0.04 0.05 0.06\\n●\\nRSS\\nβ1 β0\\nFIGURE 3.2. Contour and three-dimensional plots of the RSS on the\\nAdvertising data, using sales as the response and TV as the predictor. The\\nred dots correspond to the least squares estimates ˆβ0 and ˆβ1, given by (3.4).\\nFigure 3.2, we have computed RSS for a number of values of β0 and β1,\\nusing the advertising data with sales as the response and TV as the predictor.\\nIn each plot, the red dot represents the pair of least squares estimates\\n( ˆ β0, ˆ β1) given by (3.4). These values clearly minimize the RSS.\\n3.1.2 Assessing the Accuracy of the Coefficient Estimates\\nRecall from (2.1) that we assume that the true relationship between X and\\nY takes the form Y = f(X) + ϵ for some unknown function f, where ϵ\\nis a mean-zero random error term. If f is to be approximated by a linear\\nfunction, then we can write this relationship as\\nY = β0 + β1X + ϵ.', doc_id='f1ffc41d-678e-42b1-aa62-3f6d32228432', embedding=None, doc_hash='9144d43a170e3498c7b394a04a369bd769edf7342fbcde6806bc2522d6a228e5', extra_info=None, node_info={'start': 155478, 'end': 158388, '_node_type': '1'}, relationships={<DocumentRelationship.SOURCE: '1'>: 'a2db4d93-478d-41f9-854e-ac00ecdc385e', <DocumentRelationship.PREVIOUS: '2'>: '1a990ac3-9da6-4789-a63a-ed581e847763', <DocumentRelationship.NEXT: '3'>: 'ccda122c-aa3d-4d08-9b8f-246967a2c446'}), score=0.8562101371814279)\n"
     ]
    }
   ],
   "source": [
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"NodeWithScore(node=Node(text='. . . ,n. In other words, we want to find an intercept ˆ β0 and a slope ˆ β1 such\\\\nthat the resulting line is as close as possible to the n = 200 data points.\\\\nThere are a number of ways of measuring closeness. However, by far the\\\\nmost common approach involves minimizing the least squares criterion,\\\\nleast squares\\\\nand we take that approach in this chapter. Alternative approaches will be\\\\nconsidered in Chapter 6.\\\\nLet ˆyi = ˆ β0 + ˆ β1xi be the prediction for Y based on the ith value of X.\\\\nThen ei = yi−ˆyi represents the ith residual—this is the difference between\\\\nresidual\\\\n62 3. Linear Regression\\\\n0 50 100 150 200 250 300\\\\n5 10 15 20 25\\\\nTV\\\\nSales\\\\nFIGURE 3.1. For the Advertising data, the least squares fit for the regression\\\\nof sales onto TV is shown. The fit is found by minimizing the residual sum of\\\\nsquares. Each grey line segment represents a residual. In this case a linear fit\\\\ncaptures the essence of the relationship, although it overestimates the trend in the\\\\nleft of the plot.\\\\nthe ith observed response value and the ith response value that is predicted\\\\nby our linear model. We define the residual sum of squares (RSS) as\\\\nresidual sum\\\\nof squares RSS = e21\\\\n+ e22\\\\n+ · · · + e2\\\\nn,\\\\nor equivalently as\\\\nRSS = (y1− ˆ β0− ˆ β1x1)2+(y2− ˆ β0− ˆ β1x2)2+· · ·+(yn− ˆ β0− ˆ β1xn)2. (3.3)\\\\nThe least squares approach chooses ˆ β0 and ˆ β1 to minimize the RSS. Using\\\\nsome calculus, one can show that the minimizers are\\\\nˆ β1 =\\\\n)n\\\\ni=1)(xi − x¯)(yi − y¯) n\\\\ni=1(xi − ¯x)2 ,\\\\nˆ β0 = ¯y − ˆ β1¯x,\\\\n(3.4)\\\\nwhere ¯y ≡ 1n\\\\n)n\\\\ni=1 yi and ¯x ≡ 1n\\\\n)n\\\\ni=1 xi are the sample means. In other\\\\nwords, (3.4) defines the least squares coefficient estimates for simple linear\\\\nregression.\\\\nFigure 3.1 displays the simple linear regression fit to the Advertising\\\\ndata, where ˆ β0 = 7.03 and ˆ β1 = 0.0475. In other words, according to\\\\nthis approximation, an additional $1,000 spent on TV advertising is associated\\\\nwith selling approximately 47.5 additional units of the product. In\\\\n3.1 Simple Linear Regression 63\\\\nβ0\\\\nβ1\\\\n2.11\\\\n2.15\\\\n2.2\\\\n2.3\\\\n2.5\\\\n2.5\\\\n3\\\\n3\\\\n5 6 7 8 9\\\\n0.03 0.04 0.05 0.06\\\\n●\\\\nRSS\\\\nβ1 β0\\\\nFIGURE 3.2. Contour and three-dimensional plots of the RSS on the\\\\nAdvertising data, using sales as the response and TV as the predictor. The\\\\nred dots correspond to the least squares estimates ˆβ0 and ˆβ1, given by (3.4).\\\\nFigure 3.2, we have computed RSS for a number of values of β0 and β1,\\\\nusing the advertising data with sales as the response and TV as the predictor.\\\\nIn each plot, the red dot represents the pair of least squares estimates\\\\n( ˆ β0, ˆ β1) given by (3.4). These values clearly minimize the RSS.\\\\n3.1.2 Assessing the Accuracy of the Coefficient Estimates\\\\nRecall from (2.1) that we assume that the true relationship between X and\\\\nY takes the form Y = f(X) + ϵ for some unknown function f, where ϵ\\\\nis a mean-zero random error term. If f is to be approximated by a linear\\\\nfunction, then we can write this relationship as\\\\nY = β0 + β1X + ϵ.', doc_id='f1ffc41d-678e-42b1-aa62-3f6d32228432', embedding=None, doc_hash='9144d43a170e3498c7b394a04a369bd769edf7342fbcde6806bc2522d6a228e5', extra_info=None, node_info={'start': 155478, 'end': 158388, '_node_type': '1'}, relationships={<DocumentRelationship.SOURCE: '1'>: 'a2db4d93-478d-41f9-854e-ac00ecdc385e', <DocumentRelationship.PREVIOUS: '2'>: '1a990ac3-9da6-4789-a63a-ed581e847763', <DocumentRelationship.NEXT: '3'>: 'ccda122c-aa3d-4d08-9b8f-246967a2c446'}), score=0.8562101371814279)\""
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(response.source_nodes[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'nodes'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32mc:\\Users\\toekle\\OneDrive - Capgemini\\Documents\\Kurs\\ML_AI\\llama_index\\test_llama_index.ipynb Cell 13\u001b[0m in \u001b[0;36m1\n\u001b[1;32m----> <a href='vscode-notebook-cell:/c%3A/Users/toekle/OneDrive%20-%20Capgemini/Documents/Kurs/ML_AI/llama_index/test_llama_index.ipynb#X46sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m response\u001b[39m.\u001b[39;49msource_nodes\u001b[39m.\u001b[39;49mnodes[\u001b[39m0\u001b[39m]\u001b[39m.\u001b[39mtext\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'list' object has no attribute 'nodes'"
     ]
    }
   ],
   "source": [
    "response.source_nodes.nodes[0].text"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}