Smitha&saisharan.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "##Install All packages\n"
      ],
      "metadata": {
        "id": "36Y-BtYs7Q9e"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "95HmH7LEdd8K"
      },
      "outputs": [],
      "source": [
        "!pip install -q --upgrade langchain pypdf chromadb google-generativeai langchain-google-genai python-dotenv"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Import All required Packages"
      ],
      "metadata": {
        "id": "0sXWmH2N8BbY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import google.generativeai as genai\n",
        "from langchain import PromptTemplate\n",
        "from langchain.chains.question_answering import load_qa_chain\n",
        "from langchain.document_loaders import PyPDFDirectoryLoader\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "from langchain.vectorstores import Chroma\n",
        "from google.colab import userdata\n",
        "from dotenv import load_dotenv"
      ],
      "metadata": {
        "id": "gTOWYrwXf2IE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Set up API key"
      ],
      "metadata": {
        "id": "MHIcvuRZ8pdR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n",
        "genai.configure(api_key=GOOGLE_API_KEY)"
      ],
      "metadata": {
        "id": "HshOrLsXfM-M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!echo -e 'GOOGLE_API_KEY=AIzaSyB7-Z9AivYe01bpPGiQWk_jqIaY7x9OiUU' > .env"
      ],
      "metadata": {
        "id": "Id0SA646i7-a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "load_dotenv()"
      ],
      "metadata": {
        "id": "XXEB_B55fetB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "71805a6c-e5b7-42f6-ccce-6771318f2946"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Load PDF files from Folder"
      ],
      "metadata": {
        "id": "in-2VZzv9Zp9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "loader = PyPDFDirectoryLoader(\"/content/sample_data/data\")\n",
        "data = loader.load()"
      ],
      "metadata": {
        "id": "z0hahDgdf8Se"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ge54_Xigg78t",
        "outputId": "7465da6c-02bf-49f9-c375-6591d8cf6661"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='/two.ltab/two.ltab MAY /two.lprp/zero.lprp/one.lprp/nine.lprp  /  HOMETEXTILESTODAY.COMIn Walmart Inc.’s new annual report, Doug McMillon reﬂ  ects on lessons \\nlearned during his ﬁ  ve years as the company’s chief executive ofﬁ  cer.\\nThe report, titled “Deﬁ  ning the Future of Retail,” was recently posted to \\nthe company’s investor relations site ahead of the retailer’s June 5 annual \\nshareholder meeting\\nMcMillon writes that Walmart is focused on long-term success.\\n“We’re playing the long game,” he said. “Managing our business on a dai-\\nly basis is important, but our most important strategic decisions are made \\nin light of what we want our company \\nto become for the next generation.”\\nA leader during times of transforma-\\ntion has to be curious, introduce new \\nideas and ask questions. “You can’t \\npush a rope, but you can pull it. In \\nother words, sometimes you just can’t \\nlead from behind,” he added.\\nIn pursuing growth, companies also \\nhave to get comfortable with “an intel-\\nligent level of risk.” McMillon pointed \\nto Walmart’s investments in higher \\nwages, associate education, pricing \\nand e-commerce.“We acquired Flipkart, Jet and oth-\\ners, and we partnered with global \\ntechnology companies in places like \\nChina and Japan. We don’t know \\nwhat Sam [company founder Sam \\nWalton] would have done in these \\nmoments, but we know he would have been adapting — and he would \\nhave been aggressive”\\nHe added that Walmart associates also feel more comfortable taking risk.\\n“They’re launching minimum viable products to test and learn from. \\nThese have enough function to satisfy early adopters, whose feedback \\ninforms future design. Result: We go from Product 1.0 to Product 2.0 a lot \\nfaster. This is a powerful unlock.” \\nFinally, he said, building trust by communicating the work the company \\nand its people are doing is important to the company’s long-term health.\\n“In our supply chain, we are eliminating waste, using more renewable en-\\nergy, reducing carbon emissions and making our items and the packages \\nthey come in healthier and more sustainable,” said McMillon. “Of course \\nwe aren’t perfect. We make mistakes. But, if the world could see all of the \\nhard-working, well-intentioned people inside our company who are mak-\\ning things better in their communities and in the world, I’m convinced they \\nwould be moved by it all. I am.”  HTTRETAIL IN DETAIL\\nDoug McMillonWalmart CEO Looks \\nat the Long Game\\n...our most \\nimportant strategic \\ndecisions are made \\nin light of what we \\nwant our company \\nto become for the \\nnext generation.\\n—Doug McMillon\\n CEO, Walmart\\nApril proved to be a month of change for \\nretail execs. Maybe ﬁ  scal year closings in \\nFebruary and March made it a good time \\nto move on, or perhaps it’s a reﬂ  ection \\nof the broader disruption in the retail \\nmarket. Here’s a scorecard of who’s going \\nwhere.\\nANTHROPOLOGIE\\nAndrew Carnie, a key player in develop-\\ning the retailer’s souped-up home busi-\\nness, stepped down as president of home, \\ngarden and international April 26 to return \\nto the UK. Hillary Super was promoted \\nto global president of the Anthropologie Group. In addition to her previous \\nresponsibilities overseeing the women’s apparel and accessories categories \\nand the BHLDN bridal business, she assumes oversight of the business seg-\\nments previously reporting to Carnie.\\nBURLINGTON STORES\\nChairman, president and CEO Thomas Kingsbury announced he will cede \\nhis role around Sept. 16 after more than 10 years at the helm. He will serve as executive chairman of the board during a transition. Michael O’Sullivan, \\nwho recently exited Ross Stores as president and chief operating ofﬁ  cer, will \\nsucceed Kingsbury and will take a seat on the board. Jennifer Vecchio, chief \\nmerchandising ofﬁ  cer and principal, was appointed to the newly created role \\nof president, chief merchandising ofﬁ  cer. She continues to oversee merchan-\\ndising and planning and also takes on marketing and strategy.\\nJCPENNEY\\nMacy’s Inc. veteran Laurene Gandolfo joined the company April 22 as senior \\nvice president for home product design and development, a new position. \\nShe reports to Michelle Wlazlo, executive vice president, chief merchant. For-\\nmer T arget executive Trish Adams joined Penney as a strategic advisor, sup-\\nporting the company’s transformation ofﬁ  ce led by Truett Horne, senior vice \\npresident, chief transformation ofﬁ  cer. In a third hire, Penney named Steve \\nWhaley senior vice president, principal accounting ofﬁ  cer and controller.\\nROSS STORES\\nAfter O’Sullivan decamped to Burlington, the company announced it expects \\nto make management changes over the coming months. In the meantime, \\nMike Kobayashi, group EVP for supply chain, merchant operations and \\ntechnology, reports to CEO Barbara Rentler. EVPs for stores and ﬁ  nance who \\npreviously reported to Sullivan also report to Rentler in the interim.  HTT\\nTHE EXECUTIVE SHUFFLE\\nMacy’s Inc. veteran Laurene \\nGandolfo took on a newly created \\nhome position at JCPenney.  ', metadata={'source': '/content/sample_data/data/walmart.pdf', 'page': 0}),\n",
              " Document(page_content=\"Copyright\\nof\\nHome\\nTextiles\\nToday\\nis\\nthe\\nproperty\\nof\\nProgressive\\nBusiness\\nMedia\\nand\\nits\\ncontent\\nmay\\nnot\\nbe\\ncopied\\nor\\nemailed\\nto\\nmultiple\\nsites\\nor\\nposted\\nto\\na\\nlistserv\\nwithout\\nthe\\ncopyright\\nholder's\\nexpress\\nwritten\\npermission.\\nHowever,\\nusers\\nmay\\nprint,\\ndownload,\\nor\\nemail\\narticles\\nfor\\nindividual\\nuse.\\n\", metadata={'source': '/content/sample_data/data/walmart.pdf', 'page': 1}),\n",
              " Document(page_content='Econometric Computing with HC and HAC Covariance\\nMatrix Estimators\\nAchim Zeileis\\nUniversit ¨at Innsbruck\\nAbstract\\nThis introduction to the Rpackagesandwich is a (slightly) modiﬁed version of Zeileis\\n(2004), published in the Journal of Statistical Software . A follow-up paper on object\\nobject-oriented computation of sandwich estimators is available in ( Zeileis 2006b ).\\nData described by econometric models typically contains autocorrel ation and/or het-\\neroskedasticity of unknown form and for inference in such models it is essential to use\\ncovariance matrix estimators that can consistently estimate the covar iance of the model\\nparameters. Hence, suitable heteroskedasticity-consistent (HC) an d heteroskedasticity\\nand autocorrelation consistent (HAC) estimators have been receiving attention in the\\neconometric literature over the last 20 years. To apply these estimat ors in practice,\\nan implementation is needed that preferably translates the conceptu al properties of the\\nunderlying theoretical frameworks into computational tools. In this paper, such an imple-\\nmentation in the package sandwich in theRsystem for statistical computing is described\\nand it is shown how the suggested functions provide reusable compone nts that build on\\nreadily existing functionality and how they can be integrated easily into new inferential\\nprocedures or applications. The toolbox contained in sandwich is extremely ﬂexible and\\ncomprehensive, including speciﬁc functions for the most importan t HC and HAC estima-\\ntors from the econometric literature. Several real-world data sets are used to illustrate\\nhow the functionality can be integrated into applications.\\nKeywords : covariance matrix estimators, heteroskedasticity, autocorrelation , estimating func-\\ntions, econometric computing, R.\\n1. Introduction\\nThis paper combines two topics that play an important role in applied ec onometrics: compu-\\ntational tools and robust covariance estimation.\\nWithout the aid of statistical and econometric software modern data analys is would not be\\npossible: hence, both practitioners and (applied) researchers rel y on computational tools that\\nshould preferably implement state-of-the-art methodology and be num erically reliable, easy\\nto use, ﬂexible and extensible.\\nIn many situations, economic data arises from time-series or cross-sec tional studies which\\ntypically exhibit some form of autocorrelation and/or heteroskedasticit y. If the covariance\\nstructure were known, it could be taken into account in a (parametri c) model, but more\\noften than not the form of autocorrelation and heteroskedasticity is unkn own. In such cases,\\nmodel parameters can typically still be estimated consistently us ing the usual estimating', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 0}),\n",
              " Document(page_content='2 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nfunctions, but for valid inference in such models a consistent co variance matrix estimate is\\nessential. Over the last 20 years several procedures for heterosked asticity consistent (HC) and\\nfor heteroskedasticity and autocorrelation consistent (HAC) covarianc e estimation have been\\nsuggested in the econometrics literature ( White 1980 ;MacKinnon and White 1985 ;Newey\\nandWest1987 ,1994;Andrews1991 , amongothers)andarenowroutinelyusedineconometric\\nanalyses.\\nMany statistical and econometric software packages implement various HC and HAC esti-\\nmators for certain inference procedures, so why is there a need for a paper about economet-\\nric computing with HC and HAC estimators? Typically, only certain spec ial cases of such\\nestimators—and not the general framework they are taken from—are implement ed in sta-\\ntistical and econometric software packages and sometimes they are only avai lable as options\\nto certain inference functions. It is desirable to improve on thi s for two reasons: First, the\\nliterature suggested conceptual frameworks for HC and HAC estimation and it would only\\nbe natural to translate these conceptual properties into computational tools that reﬂect the\\nﬂexibility of the general framework. Second, it is important, particu larly for applied research,\\nto have covariance matrices not only as options to certain tests but as st and-alone functions\\nwhich can be used as modular building blocks and plugged into various i nference procedures.\\nThis is becoming more and more relevant, because today, as Cribari-Neto and Zarkos (2003)\\npoint out, applied researchers typically cannot wait until a certain p rocedure becomes avail-\\nable in the software package of their choice but are often forced to program ne w techniques\\nthemselves. Thus, just as suitable covariance estimators are routin ely plugged into formu-\\nlas in theoretical work, programmers should be enabled to plug in imple mentations of such\\nestimators in computational work. Hence, the aim of this paper is to prese nt an economet-\\nric computing approach to HC and HAC estimation that provides reusable com ponents that\\ncan be used as modular building blocks in implementing new infer ential techniques and in\\napplications.\\nAll functions described are available in the package sandwich implemented in the Rsystem\\nfor statistical computing ( RDevelopment Core Team 2008 ) which is currently not the most\\npopular environment for econometric computing but which is ﬁnding i ncreasing attention\\namong econometricians ( Cribari-Neto and Zarkos 1999 ;Racine and Hyndman 2002 ). BothR\\nitself and the sandwich package (as well as all other packages used in this paper) are available\\nat no cost under the terms of the general public licence (GPL) from the comprehensive R\\narchive network (CRAN, http://CRAN.R-project.org/ ).Rhas no built-in support for HC\\nand HAC estimation and at the time we started writing sandwich there was only one package\\nthat implements HC (but not HAC) estimators (the carpackage Fox 2002 ) but which does\\nnot allow for as much ﬂexibility as the tools presented here. sandwich provides the functions\\nvcovHCandvcovHAC implementing general classes of HC and HAC estimators. The names of\\nthe functions are chosen to correspond to vcov,R’s generic function for extracting covariance\\nmatrices from ﬁtted model objects.\\nBelow, we focus on the general linear regression model estimated by or dinary least squares\\n(OLS), which is typically ﬁtted in Rusing the function lmfrom which the standard covariance\\nmatrix (assuming spherical errors) can be extracted by vcov. Using the tools from sandwich ,\\nHC and HAC covariances matrices can now be extracted from the same ﬁtted m odels using\\nvcovHCandvcovHAC.Duetotheobjectorientationof R,thesefunctionsarenotonlylimitedto\\nthe linear regression model but can be easily extended to other mode ls. The HAC estimators\\nare already available for generalized linear models (ﬁtted by glm) and robust regression (ﬁtted', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 1}),\n",
              " Document(page_content='Achim Zeileis 3\\nbyrlmin package MASS). Another important feature of Rthat is used repeatedly below is\\nthat functions are ﬁrst-level objects—i.e., functions can take func tions as arguments and\\nreturn functions—which is particularly useful for deﬁning certai n procedures for data-driven\\ncomputations such as the deﬁnition of the structure of covariance matri ces in HC estimation\\nand weighting schemes for HAC estimation.\\nThe remainder of this paper is structured as follows: To ﬁx notations, Section2describes\\nthe linear regression model used and motivates the following section s. Section 3gives brief\\nliterature reviews and describes the conceptual frameworks for HC and HAC estimation re-\\nspectively and then shows how the conceptual properties are turne d into computational tools\\ninsandwich . Section 4provides some illustrations and applications of these tools before\\na summary is given in Section 5. More details about the Rcode used are provided in an\\nappendix.\\n2. The linear regression model\\nTo ﬁx notations, we consider the linear regression model\\nyi=x⊤\\niβ+ui(i= 1,...,n), (1)\\nwith dependent variable yi,k-dimensional regressor xiwith coeﬃcient vector βand error\\ntermui. In the usual matrix notation comprising all nobservations this can be formulated\\nasy=Xβ+u.\\nInthegenerallinearmodel, itistypicallyassumedthattheerrorsh avezeromeanandvariance\\nVAR[u] = Ω. Under suitable regularity conditions (see e.g., Greene 1993 ;White 2000 ), the\\ncoeﬃcients βcan be consistently estimated by OLS giving the well-known OLS esti matorˆβ\\nwith corresponding OLS residuals ˆ ui:\\nˆβ=(\\nX⊤X)−1X⊤y (2)\\nˆu= (In−H)y= (In−X(\\nX⊤X)−1X⊤)y (3)\\nwhereInis then-dimensional identity matrix and His usually called hat matrix. The\\nestimates ˆβare unbiased and asymptotically normal ( White 2000 ). Their covariance matrix\\nΨ is usually denoted in one of the two following ways:\\nΨ =VAR[ˆβ] =(\\nX⊤X)−1X⊤ΩX(\\nX⊤X)−1(4)\\n=(1\\nnX⊤X)−11\\nnΦ(1\\nnX⊤X)−1\\n(5)\\nwhereΦ = n−1X⊤ΩXisessentiallythecovariancematrixofthescoresorestimatingfunc tions\\nVi(β) =xi(yi−x⊤\\niβ). The estimating functions evaluated at the parameter estimates ˆVi=\\nVi(ˆβ) have then sum zero.\\nFor inference in the linear regression model, it is essential to ha ve a consistent estimator for\\nΨ. What kind of estimator should be used for Ψ depends on the assumptions about Ω: In the\\nclassical linear model independent and homoskedastic errors with var ianceσ2are assumed', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 2}),\n",
              " Document(page_content='4 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nyielding Ω = σ2Inand Ψ = σ2(X⊤X)−1which can be consistently estimated by plugging\\nin the usual OLS estimator ˆ σ2= (n−k)−1∑n\\ni=1ˆu2\\ni. But if the independence and/or ho-\\nmoskedasticity assumption is violated, inference based on this esti matorˆΨconst= ˆσ(X⊤X)−1\\nwill be biased. HC and HAC estimators tackle this problem by plugging an es timateˆΩ or\\nˆΦ into (4) or (5) respectively which are consistent in the presence of heteroske dasticity and\\nautocorrelation respectively. Such estimators and their implemen tation are described in the\\nfollowing section.\\n3. Estimating the covariance matrix Ψ\\n3.1. Dealing with heteroskedasticity\\nIf it is assumed that the errors uiare independent but potentially heteroskedastic—a situation\\nwhich typically arises with cross-sectional data—their covariance mat rix Ω is diagonal but has\\nnonconstant diagonal elements. Therefore, various HC estimators ˆΨHChave been suggested\\nwhichareconstructedbyplugginganestimateoftype ˆΩ = diag( ω1,...,ω n)intoEquation( 4).\\nThese estimators diﬀer in their choice of the ωi, an overview of the most important cases is\\ngiven in the following:\\nconst : ωi= ˆσ2\\nHC0 :ωi= ˆu2\\ni\\nHC1 :ωi=n\\nn−kˆu2\\ni\\nHC2 :ωi=ˆu2\\ni\\n1−hi\\nHC3 :ωi=ˆu2\\ni\\n(1−hi)2\\nHC4 :ωi=ˆu2\\ni\\n(1−hi)δi\\nwherehi=Hiiare the diagonal elements of the hat matrix, ¯his their mean and δi=\\nmin{4,hi/¯h}.\\nThe ﬁrst equation above yields the standard estimator ˆΨconstfor homoskedastic errors. All\\nothers produce diﬀerent kinds of HC estimators. The estimator HC0 was s uggested in the\\neconometrics literature by White(1980) and is justiﬁed by asymptotic arguments. The es-\\ntimators HC1, HC2 and HC3 were suggested by MacKinnon and White (1985) to improve\\nthe performance in small samples. A more extensive study of small samp le behaviour was\\ncarried out by Long and Ervin (2000) which arrive at the conclusion that HC3 provides the\\nbest performance in small samples as it gives less weight to inﬂuenti al observations. Re-\\ncently,Cribari-Neto (2004) suggested the estimator HC4 to further improve small sample\\nperformance, especially in the presence of inﬂuential observations .\\nAlloftheseHCestimators ˆΨHChaveincommonthattheyaredeterminedby ω= (ω1,...,ω n)⊤\\nwhich in turn can be computed based on the residuals ˆ u, the diagonal of the hat matrix h\\nand the degrees of freedom n−k. To translate these conceptual properties of this class of\\nHC estimators into a computational tool, a function is required which t akes a ﬁtted regres-', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 3}),\n",
              " Document(page_content='Achim Zeileis 5\\nsion model and the diagonal elements ωas inputs and returns the corresponding ˆΨHC. In\\nsandwich , this is implemented in the function vcovHCwhich takes the following arguments:\\nvcovHC(lmobj, omega = NULL, type = \"HC3\", ...)\\nTheﬁrstargument lmobjisanobjectasreturnedby lm,R’sstandardfunctionforﬁttinglinear\\nregression models. The argument omegacan either be the vector ωor a function for data-\\ndriven computation of ωbased on the residuals ˆ u, the diagonal of the hat matrix hand the\\nresidual degrees of freedom n−k. Thus, it has tobe of the form omega(residuals, diaghat,\\ndf): e.g., for computing HC3 omegais set to function(residuals, diaghat, df)\\nresiduals^2/(1 - diaghat)^2 .\\nAs a convenience option, a typeargument can be set to \"const\",\"HC0\"(or equivalently\\n\"HC\"),\"HC1\",\"HC2\",\"HC3\"(the default) or \"HC4\"and then vcovHCuses the corresponding\\nomegafunction. As soon as omegais speciﬁed by the user, typeis ignored.\\nIn summary, by specfying ω—either as a vector or as a function— vcovHCcan compute arbi-\\ntraryHCcovariancematrixestimatesfromtheclassofestimatorsoutline dabove. InSection 4,\\nit will be illustrated how this function can be used as a building b lock when doing inference\\nin linear regression models.\\n3.2. Dealing with autocorrelation\\nIftheerrorterms uiarenotindependent, Ωisnotdiagonalandwithoutfurtherspeciﬁcationof\\naparameticmodelforthetypeofdependenceitistypicallyburden sometoestimateΩdirectly.\\nHowever, if the form of heteroskedasticity and autocorrelation is unkn own, a solution to this\\nproblem is to estimate Φ instead which is essentially the covarianc e matrix of the estimating\\nfunctions1. This is what HAC estimators do: ˆΨHACis computed by plugging an estimate ˆΦ\\ninto Equation ( 5) with\\nˆΦ =1\\nnn∑\\ni,j=1w|i−j|ˆViˆV⊤\\nj (6)\\nwherew= (w0,...,w n−1)⊤is a vector of weights. An additional ﬁnite sample adjustment can\\nbe applied by multiplication with n/(n−k). For many data structures, it is a reasonable as-\\nsumption that the autocorrelations should decrease with increasing l agℓ=|i−j|—otherwise\\nβcan typically not be estimated consistently by OLS—so that it is rather intuitive that the\\nweightswℓshould also decrease. Starting from White and Domowitz (1984) andNewey and\\nWest(1987), diﬀerent choices for the vector of weights whave been suggested in the econo-\\nmetrics literature which have been placed by Andrews (1991) in a more general framework\\nof choosing the weights by kernel functions with automatic bandwidth selection. Andrews\\nand Monahan (1992) show that the bias of the estimators can be reduced by prewhitening\\nthe estimating functions ˆViusing a vector autoregression (VAR) of order pand applying the\\nestimator in Equation ( 6) to the VAR( p) residuals subsequently. Lumley and Heagerty (1999)\\nsuggest an adaptive weighting scheme where the weights are chosen based on the estimated\\nautocorrelations of the residuals ˆ u.\\n1Due to the use of estimating functions, this approach is not only feasible in linear models estimated by OLS,\\nbut also in nonlinear models using other estimating function s such as maximum likelihood (ML), generalized\\nmethods of moments (GMM) or Quasi-ML.', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 4}),\n",
              " Document(page_content='6 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nAll the estimators mentionedabove are of the form ( 6), i.e., a weighted sum of laggedproducts\\nof the estimating functions corresponding to a ﬁtted regression mod el. Therefore, a natural\\nimplementation for this class of HAC estimators is the following:\\nvcovHAC(lmobj, weights,\\nprewhite = FALSE, adjust = TRUE, sandwich = TRUE,\\norder.by, ar.method, data)\\nThe most important arguments are again the ﬁtted linear model2lmobj—from which the es-\\ntimating functions ˆVican easily be extracted using the generic function estfun(lmobj) —and\\nthe argument weights which specifys w. The latter can be either the vector wdirectly or a\\nfunction to compute it from lmobj.3The argument prewhite speciﬁes wether prewhitening\\nshould be used or not4andadjustdetermines wether a ﬁnite sample correction by multipli-\\ncation with n/(n−k) should be made or not. By setting sandwich it can be controlled wether\\nthe full sandwich estimator ˆΨHACor only the“meat” ˆΦ/nof the sandwich should be returned.\\nThe remaining arguments are a bit more technical: order.by speciﬁes by which variable the\\ndata should be ordered (the default is that they are already ordered, as i s natural with time\\nseries data), which ar.method should be used for ﬁtting the VAR( p) model (the default is\\nOLS) and dataprovides a data frame from which order.by can be taken (the default is the\\nenvironment from which vcovHAC is called).5\\nAs already pointed out above, all that is required for specifying an esti matorˆΨHACis the\\nappropriate vector of weights (or a function for data-driven computation of t he weights).\\nFor the most important estimators from the literature mentioned above t here are func-\\ntions for computing the corresponding weights readily available in sandwich . They are\\nall of the form weights(lmobj, order.by, prewhite, ar.method, data) , i.e., functions\\nthat compute the weights depending on the ﬁtted model object lmobjand the arguments\\norder.by ,prewhite ,datawhich are only needed for ordering and prewhitening. The func-\\ntionweightsAndrews implements the class of weights of Andrews (1991) andweightsLumley\\nimplements the class of weights of Lumley and Heagerty (1999). Both functions have con-\\nvenience interfaces: kernHAC callsvcovHAC withweightsAndrews (and diﬀerent defaults for\\nsome parameters) and weavecallsvcovHAC withweightsLumley . Finally, a third convenience\\ninterface to vcovHAC is available for computing the estimator(s) of Newey and West (1987,\\n1994).\\n•Newey and West (1987) suggested to use linearly decaying weights\\nwℓ= 1−ℓ\\nL+1(7)\\nwhereLis the maximum lag, all other weights are zero. This is implemented in t he\\nfunction NeweyWest(lmobj, lag = NULL, ...) wherelagspeciﬁes Land...are\\n2Note, that not only HAC estimators for ﬁtted linearmodels can be computed with vcovHAC. SeeZeileis\\n(2006b) for details.\\n3Ifweights is a vector with less than nelements, the remaining weights are assumed to be zero.\\n4The order pis set to as.integer(prewhite) , hence both prewhite = 1 andprewhite = TRUE lead to a\\nVAR(1) model, but also prewhite = 2 is possible.\\n5Moredetailedtechnicaldocumentationoftheseandotherargu mentsofthefunctionsdescribedareavailable\\nin the reference manual included in sandwich .', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 5}),\n",
              " Document(page_content='Achim Zeileis 7\\n(here, and in the following) further arguments passed to other funct ions, detailed infor-\\nmation is always available in the reference manual. If lagis set toNULL(the default)\\nthe non-parametric bandwidth selection procedure of Newey and West (1994) is used.\\nThis is also available in a stand-alone function bwNeweyWest , see also below.\\n0.0 0.5 1.0 1.5 2.0 2.5 3.00.0 0.2 0.4 0.6 0.8 1.0\\nxK(x)Truncated\\nBartlett\\nQuadratic SpectralParzen\\nTukey−Hanning\\nFigure 1: Kernel functions for kernel-based HAC estimation.\\n•Andrews (1991) placed this and other estimators in a more general class of kernel-\\nbased HAC estimators with weights of the form wℓ=K(ℓ/B) whereK(·) is a kernel\\nfunction and Bthe bandwidth parameter used. The kernel functions considered are the\\ntruncated, Bartlett, Parzen, Tukey-Hanning and quadratic spectral k ernel which are\\ndepicted in Figure 1. The Bartlett kernel leads to the weights used by Newey and West\\n(1987) in Equation ( 7) when the bandwidth Bis set toL+1. The kernel recommended\\nbyAndrews (1991) and probably most used in the literature is the quadratic spectral\\nkernel which leads to the following weights:\\nwℓ=3\\nz2(sin(z)\\nz−cos(z))\\n, (8)\\nwherez= 6π/5·ℓ/B. The deﬁnitions for the remaining kernels can be found in Andrews\\n(1991). All kernel weights mentioned above are available in weightsAndrews(lmobj,\\nkernel, bw, ...) wherekernelspeciﬁes one of the kernels via a character string\\n(\"Truncated\" ,\"Bartlett\" ,\"Parzen\" ,\"Tukey-Hanning\" or\"Quadratic Spectral\" )\\nandbwthe bandwidth either as a scalar or as a function. The automatic bandwidth\\nselection described in Andrews (1991) via AR(1) or ARMA(1,1) approximations is im-\\nplemented in a function bwAndrews which is set as the default in weightsAndrews .\\nFor the Bartlett, Parzen and quadratic spectral kernels, Newey and West (1994) sug-\\ngested a diﬀerent nonparametric bandwidth selection procedure, w hich is implemented\\ninbwNeweyWest and which can also be passed to weightsAndrews . As the ﬂexibility of', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 6}),\n",
              " Document(page_content='8 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nthis conceptual framework of estimators leads to a lot of knobs and switche s in the com-\\nputational tools, a convenience function kernHAC for kernel-based HAC estimation has\\nbeen added to sandwich that calls vcovHAC based on weightsAndrews andbwAndrews\\nwithdefaultsasmotivatedby Andrews (1991)andAndrewsandMonahan (1992): byde-\\nfault, it computes a quadratic spectral kernel HAC estimator with VAR( 1) prewhitening\\nand automatic bandwidth selection based on an AR(1) approximation. But of cou rse,\\nall the options described above can also be changed by the user when call ingkernHAC.\\n•Lumley and Heagerty (1999) suggested a diﬀerent approach for specifying the weights\\nin (6) based on some estimate ˆ ̺ℓof the autocorrelation of the residuals ˆ uiat lag\\n0 = 1,...,n−1. They suggest either to use truncated weights wℓ=I{nˆ̺2\\nℓ> C}\\n(whereI(·) is the indicator function) or smoothed weights wℓ= min{1,Cnˆ̺2\\nℓ}, where\\nfor both a suitable constant Chas to be speciﬁed. Lumley and Heagerty (1999) suggest\\nusing a default of C= 4 and C= 1 for the truncated and smoothed weights respec-\\ntively. Note, that the truncated weights are equivalent to the trunc ated kernel from\\nthe framework of Andrews (1991) but using a diﬀerent method for computing the trun-\\ncation lag. To ensure that the weights |wℓ|are decreasing, the autocorrelations have\\nto be decreasing for increasing lag ℓwhich can be achieved by using isotonic regres-\\nsion methods. In sandwich , these two weighting schemes are implemented in a function\\nweightsLumley withaconvenienceinterface weave(whichstandsforw eightede mpirical\\nadaptive v ariance e stimators) which again sets up the weights and then calls vcovHAC.\\nIts most important arguments are weave(lmobj, method, C, ...) wheremethodcan\\nbe either \"truncate\" or\"smooth\" andCis by default 4 or 1 respectively.\\nTo sum up, vcovHAC provides a simple yet ﬂexible interface for general HAC estimation as\\ndeﬁned in Equation ( 6). Arbitrary weights can be supplied either as vectors or functions for\\ndata-driven computation of the weights. As the latter might easily become rather complex,\\nin particular due to the automatic choice of bandwidth or lag truncation par ameters, three\\nstrategies suggested in the literature are readily available in sandwich : First, the Bartlett ker-\\nnelweightssuggestedby NeweyandWest (1987,1994)areusedin NeweyWest whichbydefault\\nusesthebandwidthselectionfunction bwNeweyWest . Second, theweightingschemeintroduced\\nbyAndrews (1991) for kernel-based HAC estimation with automatic bandwidth selection i s\\nimplemented in weightsAndrews andbwAndrews with corresponding convenience interface\\nkernHAC. Third, the weighted empirical adaptive variance estimation scheme suggested by\\nLumley and Heagerty (1999) is available in weightsLumley with convenience interface weave.\\nIt is illustrated in the following section how these functions can be easily used in applications.\\n4. Applications and illustrations\\nIn econometric analyses, the practitioner is only seldom interested in the covariance matrix ˆΨ\\n(orˆΩ orˆΦ)per se, but mainly wants to compute them to use them for inferential proce dures.\\nTherefore, it is important that the functions vcovHCandvcovHAC described in the previous\\nsection can be easily supplied to other procedures such that the us er does not necessarily have\\nto compute the variances in advance.\\nA typical ﬁeld of application for HC and HAC covariances are partial torztests for assessing\\nwhether a parameter βjis signiﬁcantly diﬀerent from zero. Exploiting the (asymptotic)', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 7}),\n",
              " Document(page_content='Achim Zeileis 9\\nnormality of the estimates, these tests are based on the tratioˆβj/√\\nˆΨjjand either use\\nthe asymptotic normal distribution or the tdistribution with n−kdegrees of freedom for\\ncomputing pvalues (White 2000 ). This procedure is available in the Rpackagelmtest(Zeileis\\nand Hothorn 2002 ) in the generic function coeftest which has a default method applicable\\nto ﬁtted\"lm\"objects.\\ncoeftest(lmobj, vcov = NULL, df = NULL, ...)\\nwherevcovspeciﬁesthecovarianceseitherasamatrix(correspondingtothec ovariancematrix\\nestimate) or as a function computing it from lmobj(corresponding to the covariance matrix\\nestimator). By default, it uses the vcovmethod which computes ˆΨconstassuming spherical\\nerrors. The dfargument determines the degrees of freedom: if dfis ﬁnite and positive,\\natdistribution with dfdegrees of freedom is used, otherwise a normal approximation is\\nemployed. The default is to set dfton−k.\\nInference based on HC and HAC estimators is illustrated in the followin g using three real-\\nworld data sets: testing coeﬃcients in two models from Greene(1993) and a structural change\\nproblem from Bai and Perron (2003).\\nTo make the results exactly reproducible for the reader, the comman ds for the inferential\\nprocedures is given along with their output within the text. A ful l list of commands, including\\nthose which produce the ﬁgures in the text, are provided (without output) in the appendix\\nalong with the versions of Rand the packages used. Before we start with the examples, the\\nsandwich andlmtestpackage have to be loaded:\\nR> library(\"sandwich\")\\nR> library(\"lmtest\")\\n4.1. Testing coeﬃcients in cross-sectional data\\nA quadratic regression model for per capita expenditures on public s chools explained by\\nper capita income in the United States in 1979 has been analyzed by Greene(1993) and\\nre-analyzed in Cribari-Neto (2004). The corresponding cross-sectional data for the 51 US\\nstates is given in Table 14.1 in Greene(1993) and available in sandwich in the data frame\\nPublicSchools which can be loaded by:\\nR> data(\"PublicSchools\")\\nR> ps <- na.omit(PublicSchools)\\nR> ps$Income <- ps$Income * 0.0001\\nwhere the second line omits a missing value ( NA) in Wisconsin and assigns the result to a\\nnew data frame psand the third line transforms the income to be in USD 10 ,000. The\\nquadratic regression can now easily be ﬁt using the function lmwhich ﬁts linear regression\\nmodels speciﬁed by a symbolic formula via OLS.\\nR> fm.ps <- lm(Expenditure ~ Income + I(Income^2), data = ps)\\nThe ﬁtted \"lm\"objectfm.psnow contains the regression of the variable Expenditure on\\nthe variable Incomeand its sqared value, both variables are taken from the data frame ps.', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 8}),\n",
              " Document(page_content='10 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nThe question in this data set is whether the quadratic term is reall y needed, i.e., whether\\nthe coeﬃcient of I(Income^2) is signiﬁcantly diﬀerent from zero. The partial quasi- ttests\\n(orztests) for all coeﬃcients can be computed using the function coeftest .Greene(1993)\\nassesses the signiﬁcance using the HC0 estimator of White(1980).\\nR> coeftest(fm.ps, df = Inf, vcov = vcovHC(fm.ps, type = \"HC0 \"))\\nz test of coefficients:\\nEstimate Std. Error z value Pr(>|z|)\\n(Intercept) 833 461 1.81 0.071 .\\nIncome -1834 1243 -1.48 0.140\\nI(Income^2) 1587 830 1.91 0.056 .\\n---\\nSignif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\\nThevcovargument speciﬁes the covariance matrix as a matrix (as opposed to a func tion)\\nwhich is returned by vcovHC(fm.ps, type = \"HC0\") . Asdfis set to inﬁnity ( Inf) a normal\\napproximation is used for computing the pvalues which seem to suggest that the quadratic\\nterm might be weakly signiﬁcant. In his analysis, Cribari-Neto (2004) uses his HC4 estimator\\n(among others) giving the following result:\\nR> coeftest(fm.ps, df = Inf, vcov = vcovHC(fm.ps, type = \"HC4 \"))\\nz test of coefficients:\\nEstimate Std. Error z value Pr(>|z|)\\n(Intercept) 833 3008 0.28 0.78\\nIncome -1834 8183 -0.22 0.82\\nI(Income^2) 1587 5489 0.29 0.77\\nThe quadratic term is clearly non-signiﬁcant. The reason for this resu lt is depicted in Figure 2\\nwhich shows the data along with the ﬁtted linear and quadratic model—t he latter being\\nobviously heavily inﬂuenced by a single outlier: Alaska. Thus, the i mproved performance of\\nthe HC4 as compared to the HC0 estimator is due to the correction for high le verage points.\\n4.2. Testing coeﬃcients in time-series data\\nGreene(1993) also anayzes a time-series regression model based on robust covarian ce matrix\\nestimates: his Table 15.1 provides data on the nominal gross national produ ct (GNP), nom-\\ninal gross private domestic investment, a price index and an interes t rate which is used to\\nformulate a model that explains real investment by real GNP and real int erest. The corre-\\nsponding transformed variables RealInv,RealGNP andRealInt are stored in the data frame\\nInvestment insandwich which can be loaded by:\\nR> data(\"Investment\")\\nSubsequently, the ﬁtted linear regression model is computed by:', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 9}),\n",
              " Document(page_content='Achim Zeileis 11\\n0.6 0.7 0.8 0.9 1.0 1.1300 400 500 600 700 800\\nper capita incomeper capita spending on public schoolsAlaska\\nFigure 2: Expenditure on public schools and income with ﬁtted model s.\\nR> fm.inv <- lm(RealInv ~ RealGNP + RealInt, data = Investmen t)\\nand the signiﬁcance of the coeﬃcients can again be assessed by partial ztests using coeftest .\\nGreene(1993) uses the estimator of Newey and West (1987) without prewhitening and with\\nlagL= 4 for this purpose which is here passed as a matrix (as opposed to a func tion) to\\ncoeftest .\\nR> coeftest(fm.inv, df = Inf, vcov = NeweyWest(fm.inv, lag = 4, prewhite = FALSE))\\nz test of coefficients:\\nEstimate Std. Error z value Pr(>|z|)\\n(Intercept) -12.5336 18.9583 -0.66 0.51\\nRealGNP 0.1691 0.0168 10.10 <2e-16 ***\\nRealInt -1.0014 3.3424 -0.30 0.76\\n---\\nSignif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\\nIf alternatively the automatic bandwidth selection procedure of Newey and West (1994) with\\nprewhitening should be used, this can be passed as a function to coeftest .\\nR> coeftest(fm.inv, df = Inf, vcov = NeweyWest)\\nz test of coefficients:\\nEstimate Std. Error z value Pr(>|z|)\\n(Intercept) -12.5336 24.3742 -0.51 0.61', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 10}),\n",
              " Document(page_content='12 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nRealGNP 0.1691 0.0236 7.17 7.4e-13 ***\\nRealInt -1.0014 3.6399 -0.28 0.78\\n---\\nSignif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\\nFor illustration purposes, we show how a new function implementin g a particular HAC esti-\\nmator can be easily set up using the tools provided by sandwich . This is particularly helpful\\nif the same estimator is to be applied several times in the course of an an alysis. Suppose,\\nwe want to use a Parzen kernel with VAR(2) prewhitening, no ﬁnite sam ple adjustment and\\nautomatic bandwidth selection according to Newey and West (1994). First, we set up the\\nfunctionparzenHAC and then pass this function to coeftest .\\nR> parzenHAC <- function(x, ...) kernHAC(x, kernel = \"Parze n\", prewhite = 2,\\n+ adjust = FALSE, bw = bwNeweyWest, ...)\\nR> coeftest(fm.inv, df = Inf, vcov = parzenHAC)\\nz test of coefficients:\\nEstimate Std. Error z value Pr(>|z|)\\n(Intercept) -12.5336 24.6639 -0.51 0.61\\nRealGNP 0.1691 0.0208 8.12 4.7e-16 ***\\nRealInt -1.0014 3.9475 -0.25 0.80\\n---\\nSignif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\\nThe three estimators leads to slightly diﬀerent standard errors, bu t all tests agree that real\\nGNP has a highly signiﬁcant inﬂuence while the real interest rate has n ot. The data along\\nwith the ﬁtted regression are depicted in Figure 3.\\n4.3. Testing and dating structural changes in the presence of\\nheteroskedasticity and autocorrelation\\nTo illustrate that the functionality provided by the covariance es timators implemented in\\nsandwich cannot only be used in simple settings, such as partial quasi- ttests, but also for\\nmore complicated tasks, we employ the real interest time series anal yzed byBai and Perron\\n(2003). This series contains changes in the mean (see Figure 4, right panel) which Bai and\\nPerron(2003) detect using several structural change tests based on Fstatistics and date using\\na dynamic programming algorithm. As the visualization suggests, this seri es exhibits both\\nheteroskedasticity and autocorrelation, hence Bai and Perron (2003) use a quadratic spectral\\nkernel HAC estimator in their analysis. Here, we use the same dating pr ocedure but assess\\nthe signiﬁcance using an OLS-based CUSUM test ( Ploberger and Kr ¨amer 1992 ) based on the\\nsame HAC estimator. The data are available in the package strucchange as the quarterly\\ntime series RealInt containing the US ex-post real interest rate from 1961(1) to 1986(3) and\\nthey are analyzed by a simple regression on the mean.\\nUnder the assumptions in the classical linear model with spherical e rrors, the test statistic of', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 11}),\n",
              " Document(page_content='Achim Zeileis 13\\n 800  900 1000 1100 1200 1300 1400 1500 1600120 140 160 180 200 220 240 260−4−2 0 2 4 6\\nRealGNP\\nRealIntRealInv\\nFigure 3: Investment equation data with ﬁtted model.\\nthe OLS-based CUSUM test is\\nsup\\nj=1,...,n⏐⏐⏐⏐⏐⏐1√\\nnˆσ2j∑\\ni=1ˆui⏐⏐⏐⏐⏐⏐. (9)\\nIf autocorrelation and heteroskedasticity are present in the data, a rob ust variance estimator\\nshould be used: if xiis equal to unity, this can simply be achieved by replacing ˆ σ2withˆΦ or\\nnˆΨ respectively. Here, we use the quadratic spectral kernel HAC est imator of Andrews (1991)\\nwith VAR(1) prewhitening and automatic bandwidth selection based on an AR (1) approx-\\nimation as implemented in the function kernHAC. Thepvalues for the OLS-based CUSUM\\ntest can be computed from the distribution of the supremum of a Brown ian bridge (see e.g.,\\nPloberger and Kr ¨amer 1992 ). This and other methods for testing, dating and monitoring\\nstructural changes are implemented in the Rpackage strucchange (Zeileis, Leisch, Hornik,\\nand Kleiber 2002 ) which contains the function gefpfor ﬁtting and assessing ﬂuctuation pro-\\ncesses including OLS-based CUSUM processes (see Zeileis 2006a , for more details).\\nAfter loading the package and the data,\\nR> library(\"strucchange\")\\nR> data(\"RealInt\")\\nthe command\\nR> ocus <- gefp(RealInt ~ 1, fit = lm, vcov = kernHAC)\\nﬁts the OLS-based CUSUM process for a regression on the mean ( RealInt ~ 1 ), using the\\nfunction lmand estimating the variance using the function kernHAC. The ﬁtted OLS-based\\nCUSUM process can then be visualized together with its 5% critical valu e (horizontal lines)', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 12}),\n",
              " Document(page_content='14 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nbyplot(scus) which leads to a similar plot as in the left panel of Figure 4(see the appendix\\nfor more details). As the process crosses its boundary, there is a sign iﬁcant change in the\\nmean, while the clear peak in the process conveys that there is at leas t one strong break in\\nthe early 1980s. A formal signiﬁcance test can also be carried out by sctest(ocus) which\\nleads to a highly signiﬁcant pvalue of 0.0082. Similarly, the same quadratic spectral kernel\\nHAC estimator could also be used for computing and visualizing the sup Ftest ofAndrews\\n(1993), the code is provided in the appendix.\\nFinally, the breakpoints in this model along with their conﬁdence i ntervals can be computed\\nby:\\nR> bp <- breakpoints(RealInt ~ 1)\\nR> confint(bp, vcov = kernHAC)\\nConfidence intervals for breakpoints\\nof optimal 3-segment partition:\\nCall:\\nconfint.breakpointsfull(object = bp, vcov. = kernHAC)\\nBreakpoints at observation number:\\n2.5 % breakpoints 97.5 %\\n1 37 47 48\\n2 77 79 81\\nCorresponding to breakdates:\\n2.5 % breakpoints 97.5 %\\n1 1970(1) 1972(3) 1972(4)\\n2 1980(1) 1980(3) 1981(1)\\nThe dating algorithm breakpoints implements the procedure described in Bai and Perron\\n(2003) and estimates the timing of the structural changes by OLS. Therefore , in this step\\nno covariance matrix estimate is required, but for computing the con ﬁdence intervals using a\\nconsistent covariance matrix estimator is again essential. The confint method for computing\\nconﬁdenceintervalstakesagaina vcovargumentwhichhastobeafunction(andnotamatrix)\\nbecause it has to be applied to several segments of the data. By default, it computes the\\nbreakpoints for the minimum BIC partition which gives in this case tw o breaks.6The ﬁtted\\nthree-segment model along with the breakpoints and their conﬁdence intervals is depicted in\\nthe right panel of Figure 4.\\n5. Summary\\nThis paper brieﬂy reviews a class of heteroskedasticity-consiste nt (HC) and a class of het-\\neroskedasticity and autocorrelation consistent (HAC) covariance matri x estimators suggested\\n6By choosing the number of breakpoints with sequential tests a nd not the BIC, Bai and Perron (2003)\\narrive at a model with an additional breakpoint which has rather w ide conﬁdence intervals (see also Zeileis\\nand Kleiber 2005 )', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 13}),\n",
              " Document(page_content='Achim Zeileis 15\\n1960 1965 1970 1975 1980 1985−1.5 −0.5 0.5\\nTimeEmpirical fluctuation process\\nTimeReal interest rate\\n1960 1965 1970 1975 1980 1985−5 0 5 10\\nFigure 4: OLS-based CUSUM test (left) and ﬁtted model (right) for real in terest data.\\nin the econometric literature over the last 20 years and introduces u niﬁed computational tools\\nthat reﬂect the ﬂexibility and the conceptual ideas of the underlyi ng theoretical frameworks.\\nBased on these general tools, a number of special cases of HC and HAC estimator s is provided\\nincluding the most popular in applied econometric research. All the f unctions suggested are\\nimplemented in the package sandwich in theRsystem for statistical computing and designed\\nin such a way that they build on readily available model ﬁtting func tions and provide build-\\ning blocks that can be easily integrated into other programs or application s. To achieve this\\nﬂexibility, the object orientation mechanism of Rand the fact that functions are ﬁrst-level\\nobjects are of prime importance.\\nAcknowledgments\\nWe are grateful to Thomas Lumley for putting his code in the weavepackage at disposal and\\nfor advice in the design of sandwich , and to Christian Kleiber for helpful suggestions in the\\ndevelopment of sandwich .\\nReferences\\nAndrews DWK (1991). “Heteroskedasticity and Autocorrelation Consistent Co variance Ma-\\ntrix Estimation.” Econometrica ,59, 817–858. doi:10.2307/2938229 .\\nAndrewsDWK(1993).“TestsforParameterInstabilityandStructuralChange withUnknown\\nChange Point.” Econometrica ,61, 821–856. doi:10.2307/2951764 .\\nAndrews DWK, Monahan JC (1992). “An Improved Heteroskedasticity and Autocorr elation\\nConsistent Covariance Matrix Estimator.” Econometrica ,60(4), 953–966. doi:10.2307/\\n2951574.', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 14}),\n",
              " Document(page_content='16 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nBai J, Perron P (2003). “Computation and Analysis of Multiple Structural Change Models.”\\nJournal of Applied Econometrics ,18, 1–22.doi:10.1002/jae.659 .\\nCribari-Neto F (2004). “Asymptotic Inference Under Heteroskedasticity of Unknown Form.”\\nComputational Statistics & Data Analysis ,45, 215–233. doi:10.1016/s0167-9473(02)\\n00366-3.\\nCribari-Neto F, Zarkos SG (1999).“ R: Yet Another Econometric Programming Environment.”\\nJournal of Applied Econometrics ,14, 319–329. doi:10.1002/(sici)1099-1255(199905/\\n06)14:3<319::aid-jae533>3.0.co;2-q .\\nCribari-Neto F, Zarkos SG (2003).“Econometric and Statistical Computing Usi ngOx.”Com-\\nputational Economics ,21, 277–295. doi:10.1023/a:1023902027800 .\\nFox J (2002). AnRandS-PLUS Companion to Applied Regression . Sage Publications,\\nThousand Oaks.\\nGreene WH (1993). Econometric Analysis . 2nd edition. Macmillan Publishing Company, New\\nYork.\\nLongJS,ErvinLH(2000).“UsingHeteroscedasticityConsistentStandardErr orsintheLinear\\nRegression Model.” The American Statistician ,54, 217–224. doi:10.1080/00031305.\\n2000.10474549 .\\nLumley T, Heagerty P (1999). “Weighted Empirical Adaptive Variance Estimators for Cor-\\nrelated Data Regression.” Journal of the Royal Statistical Society B ,61, 459–477. doi:\\n10.1111/1467-9868.00187 .\\nMacKinnon JG, White H (1985). “Some Heteroskedasticity-Consistent Covar iance Matrix\\nEstimators with Improved Finite Sample Properties.” Journal of Econometrics ,29, 305–\\n325.doi:10.1016/0304-4076(85)90158-7 .\\nNewey WK, West KD(1987).“A Simple, Positive-Deﬁnite, Heteroskedasti city and Autocorre-\\nlation Consistent Covariance Matrix.” Econometrica ,55, 703–708. doi:10.2307/1913610 .\\nNewey WK, West KD (1994). “Automatic Lag Selection in Covariance Matrix Estim ation.”\\nReview of Economic Studies ,61, 631–653. doi:10.2307/2297912 .\\nPloberger W, Kr ¨amer W (1992). “The CUSUM Test with OLS Residuals.” Econometrica ,60,\\n271–285. doi:10.2307/2951597 .\\nRacine J, Hyndman R (2002). “Using Rto Teach Econometrics.” Journal of Applied Econo-\\nmetrics,17, 175–189. doi:10.1002/jae.657 .\\nRDevelopment Core Team (2008). R: A Language and Environment for Statistical Comput-\\ning.RFoundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL\\nhttps://www.R-project.org/ .\\nWhite H (1980). “A Heteroskedasticity-Consistent Covariance Matrix and a Direct Test for\\nHeteroskedasticity.” Econometrica ,48, 817–838. doi:10.2307/1912934 .\\nWhite H (2000). Asymptotic Theory for Econometricians . Revised edition. Academic Press,\\nNew York.', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 15}),\n",
              " Document(page_content='Achim Zeileis 17\\nWhite H, Domowitz I (1984). “Nonlinear Regression with Dependent Observat ions.”Econo-\\nmetrica,52, 143–161. doi:10.2307/1911465 .\\nZeileisA(2004).“EconometricComputingwithHCandHACCovarianceMatrixE stimators.”\\nJournal of Statistical Software ,11(10), 1–17. doi:10.18637/jss.v011.i10 .\\nZeileis A (2006a). “Implementing a Class of Structural Change Tests: An Ec onometric\\nComputing Approach.” Computational Statistics & Data Analysis ,50, 2987–3008. doi:\\n10.1016/j.csda.2005.07.001 .\\nZeileis A (2006b). “Object-Oriented Computation of Sandwich Estimators .”Journal of Sta-\\ntistical Software ,16(9), 1–16. doi:10.18637/jss.v016.i09 .\\nZeileis A, Hothorn T (2002). “Diagnostic Checking in Regression Relationship s.”RNews,\\n2(3), 7–10. URL https://CRAN.R-project.org/doc/Rnews/ .\\nZeileis A, Kleiber C (2005).“Validating Multiple Structural Change Mod els – A Case Study.”\\nJournal of Applied Econometrics ,20, 685–690. doi:10.1002/jae.856 .\\nZeileis A, Leisch F, Hornik K, Kleiber C (2002). “ strucchange : AnRPackage for Testing\\nfor Structural Change in Linear Regression Models.” Journal of Statistical Software ,7(2),\\n1–38.doi:10.18637/jss.v007.i02 .', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 16}),\n",
              " Document(page_content='18 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nA.Rcode\\nThepackages sandwich ,lmtestandstrucchange arerequiredfortheapplicationsinthispaper.\\nFurthermore, the packages depend on zoo. For the computations in this paper R4.0.2 and\\nsandwich 3.0–0,lmtest0.9–38,strucchange 1.5–2 and zoo1.8–9 have been used. Ritself and\\nall packages used are available from CRAN at http://CRAN.R-project.org/ .\\nTo make the packages available for the examples the following commands are necessary:\\nlibrary(\"sandwich\")\\nlibrary(\"lmtest\")\\nlibrary(\"strucchange\")\\nA.1. Testing coeﬃcients in cross-sectional data\\nLoad public schools data, omit NAin Wisconsin and scale income:\\ndata(\"PublicSchools\")\\nps <- na.omit(PublicSchools)\\nps$Income <- ps$Income * 0.0001\\nFit quadratic regression model:\\nfm.ps <- lm(Expenditure ~ Income + I(Income^2), data = ps)\\nCompare standard errors:\\nsqrt(diag(vcov(fm.ps)))\\nsqrt(diag(vcovHC(fm.ps, type = \"const\")))\\nsqrt(diag(vcovHC(fm.ps, type = \"HC0\")))\\nsqrt(diag(vcovHC(fm.ps, type = \"HC3\")))\\nsqrt(diag(vcovHC(fm.ps, type = \"HC4\")))\\nTest coeﬃcient of quadratic term:\\ncoeftest(fm.ps, df = Inf, vcov = vcovHC(fm.ps, type = \"HC0\") )\\ncoeftest(fm.ps, df = Inf, vcov = vcovHC(fm.ps, type = \"HC4\") )\\nVisualization:\\nplot(Expenditure ~ Income, data = ps,\\nxlab = \"per capita income\",\\nylab = \"per capita spending on public schools\")\\ninc <- seq(0.5, 1.2, by = 0.001)\\nlines(inc, predict(fm.ps, data.frame(Income = inc)), col = 4, lty = 2)\\nfm.ps2 <- lm(Expenditure ~ Income, data = ps)\\nabline(fm.ps2, col = 4)\\ntext(ps[2,2], ps[2,1], rownames(ps)[2], pos = 2)', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 17}),\n",
              " Document(page_content='Achim Zeileis 19\\nA.2. Testing coeﬃcients in time-series data\\nLoad investment equation data:\\ndata(\"Investment\")\\nFit regression model:\\nfm.inv <- lm(RealInv ~ RealGNP + RealInt, data = Investment)\\nTest coeﬃcients using Newey-West HAC estimator with user-deﬁne d and data-driven band-\\nwidth and with Parzen kernel:\\ncoeftest(fm.inv, df = Inf, vcov = NeweyWest(fm.inv, lag = 4, prewhite = FALSE))\\ncoeftest(fm.inv, df = Inf, vcov = NeweyWest)\\nparzenHAC <- function(x, ...) kernHAC(x, kernel = \"Parzen\" , prewhite = 2,\\nadjust = FALSE, bw = bwNeweyWest, ...)\\ncoeftest(fm.inv, df = Inf, vcov = parzenHAC)\\nTime-series visualization:\\nplot(Investment[, \"RealInv\"], type = \"b\", pch = 19, ylab = \"R eal investment\")\\nlines(ts(fitted(fm.inv), start = 1964), col = 4)\\n3-dimensional visualization:\\nlibrary(\"scatterplot3d\")\\ns3d <- scatterplot3d(Investment[,c(5,7,6)],\\ntype = \"b\", angle = 65, scale.y = 1, pch = 16)\\ns3d$plane3d(fm.inv, lty.box = \"solid\", col = 4)\\nA.3. Testing and dating structural changes in the presence of\\nheteroskedasticity and autocorrelation\\nLoad real interest series:\\ndata(\"RealInt\")\\nOLS-based CUSUM test with quadratic spectral kernel HAC estimate:\\nocus <- gefp(RealInt ~ 1, fit = lm, vcov = kernHAC)\\nplot(ocus, aggregate = FALSE)\\nsctest(ocus)\\nsupFtest with quadratic spectral kernel HAC estimate:\\nfs <- Fstats(RealInt ~ 1, vcov = kernHAC)\\nplot(fs)\\nsctest(fs)', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 18}),\n",
              " Document(page_content='20 Econometric Computing with HC and HAC Covariance Matrix Estimators\\nBreakpoint estimation and conﬁdence intervals with quadratic spectr al kernel HAC estimate:\\nbp <- breakpoints(RealInt ~ 1)\\nconfint(bp, vcov = kernHAC)\\nplot(bp)\\nVisualization:\\nplot(RealInt, ylab = \"Real interest rate\")\\nlines(ts(fitted(bp), start = start(RealInt), freq = 4), co l = 4)\\nlines(confint(bp, vcov = kernHAC))\\nA.4. Integrating covariance matrix estimators in other function s\\nIf programmers want to allow for the same ﬂexibility regarding the spec iﬁcation of covariance\\nmatrices in their own functions as illustrated in coeftest , only a few simple additions have\\nto be made which are illustrated in the following. Say, a function foo(lmobj, vcov = NULL,\\n...)wants to compute some quantity involving the standard errors associate d with the \"lm\"\\nobjectlmobj. Then,vcovshould use by default the standard vcovmethod for \"lm\"objects,\\notherwise vcovis assumed to be either a function returning the covariance matrix estimate\\nor the estimate itself. The following piece of code is suﬃcient for computing the standard\\nerrors.\\nif(is.null(vcov)) {\\nse <- vcov(lmobj)\\n} else {\\nif (is.function(vcov))\\nse <- vcov(lmobj)\\nelse\\nse <- vcov\\n}\\nse <- sqrt(diag(se))\\nIn the ﬁrst step the default method is called: note, that Rcan automatically distinguish\\nbetween the variable vcov(which is NULL) and the generic function vcov(from the stats\\npackage which dispatches to the \"lm\"method) that is called here. Otherwise, it is just\\ndistinguished between a function or non-function. In the ﬁnal step the square root of the\\ndiagonal elements is computed and stored in the vector sewhich can subsequently used for\\nfurther computation in foo().\\nAﬃliation:\\nAchim Zeileis\\nDepartment of Statistics\\nFaculty of Economics and Statistics\\nUniversit ¨at Innsbruck', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 19}),\n",
              " Document(page_content='Achim Zeileis 21\\nUniversit ¨atsstr. 15\\n6020 Innsbruck, Austria\\nE-mail:Achim.Zeileis@R-project.org\\nURL:http://eeecon.uibk.ac.at/~zeileis/', metadata={'source': '/content/sample_data/data/sandwich.pdf', 'page': 20})]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Split data into Text chunks"
      ],
      "metadata": {
        "id": "a2jf_2oI_IzU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)\n",
        "content = \"\\n\\n\".join(str(page.page_content) for page in data)"
      ],
      "metadata": {
        "id": "hIonwQkrhgRJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "texts = text_splitter.split_text(content)"
      ],
      "metadata": {
        "id": "oUcIw59ohn6O"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(texts))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UpHwXgTghrzi",
        "outputId": "7e29ae47-8c8e-43db-b8e0-eedac974af84"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "66\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "texts[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        },
        "id": "p_iZanP1hv4W",
        "outputId": "adc41b54-2a82-42f6-e8eb-031d8dfa4328"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'/two.ltab/two.ltab MAY /two.lprp/zero.lprp/one.lprp/nine.lprp  /  HOMETEXTILESTODAY.COMIn Walmart Inc.’s new annual report, Doug McMillon reﬂ  ects on lessons \\nlearned during his ﬁ  ve years as the company’s chief executive ofﬁ  cer.\\nThe report, titled “Deﬁ  ning the Future of Retail,” was recently posted to \\nthe company’s investor relations site ahead of the retailer’s June 5 annual \\nshareholder meeting\\nMcMillon writes that Walmart is focused on long-term success.\\n“We’re playing the long game,” he said. “Managing our business on a dai-\\nly basis is important, but our most important strategic decisions are made \\nin light of what we want our company \\nto become for the next generation.”\\nA leader during times of transforma-\\ntion has to be curious, introduce new \\nideas and ask questions. “You can’t \\npush a rope, but you can pull it. In \\nother words, sometimes you just can’t \\nlead from behind,” he added.\\nIn pursuing growth, companies also \\nhave to get comfortable with “an intel-'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Download Embeddings model"
      ],
      "metadata": {
        "id": "UOGeZ6gh_jxQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_google_genai import GoogleGenerativeAIEmbeddings"
      ],
      "metadata": {
        "id": "BHrE4PWQh27_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "embeddings = GoogleGenerativeAIEmbeddings(model = \"models/embedding-001\")"
      ],
      "metadata": {
        "id": "vKO83B2ViBZW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Create embeddings for each of the Text chunks and Store them in Vectorstore-chromadb"
      ],
      "metadata": {
        "id": "I4UxjePB_5-E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "vector_store = Chroma.from_texts(texts, embeddings).as_retriever()"
      ],
      "metadata": {
        "id": "zcHoggHhkISa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Create a Prompt Template"
      ],
      "metadata": {
        "id": "ULufU9vFA6bZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prompt_template = \"\"\"\n",
        "  Please answer the question in as much detail as possible based on the provided context.\n",
        "  Ensure to include all relevant details. If the answer is not available in the provided context,\n",
        "  kindly respond with \"The answer is not available in the context.\" Please avoid providing incorrect answers.\n",
        "\\n\\n\n",
        "  Context:\\n {context}?\\n\n",
        "  Question: \\n{question}\\n\n",
        "\n",
        "  Answer:\n",
        "\"\"\"\n",
        "\n",
        "prompt = PromptTemplate(template = prompt_template, input_variables = [\"context\", \"question\"])"
      ],
      "metadata": {
        "id": "P8O5EtjokYg5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Load the model"
      ],
      "metadata": {
        "id": "H05iZ958BBo1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain_google_genai import ChatGoogleGenerativeAI\n",
        "model = ChatGoogleGenerativeAI(model=\"gemini-pro\",\n",
        "                             temperature=0.3)\n"
      ],
      "metadata": {
        "id": "U0dpJOMXkiH7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chain = load_qa_chain(model, chain_type=\"stuff\", prompt=prompt)"
      ],
      "metadata": {
        "id": "khMcdKz_knMt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "question = input(\"Enter your question: \")\n",
        "docs = vector_store.get_relevant_documents(question)\n",
        "\n",
        "response = chain(\n",
        "    {\"input_documents\":docs, \"question\": question}\n",
        "    , return_only_outputs=True)\n",
        "response"
      ],
      "metadata": {
        "id": "VF0a7azBkrsV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 138
        },
        "outputId": "58581c19-bdfe-4bf5-ad79-0bad957caddc"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your question: what is linear regression model?\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'output_text': 'The linear regression model is a statistical model that attempts to determine the relationship between one or more independent variables and a dependent variable. It is typically used to predict the value of the dependent variable based on the values of the independent variables. In the linear regression model, the dependent variable is assumed to be a linear function of the independent variables, plus an error term. The error term represents the difference between the observed value of the dependent variable and the value predicted by the model.'}"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    }
  ]
}