Zestimate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Project in R -Build a machine learning algorithm to predict the future sale prices of homes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Zillow is asking you to predict the log-error between their Zestimate and the actual sale price, given all the features of a home. The log error is defined as:\n",
    "\n",
    "## logerror = log(Zestimate) &minus; log(SalePrice)- log(SalePrice)\n",
    "\n",
    "## and it is recorded in the transactions file train.csv. In this project, you are going to predict the log error for the months in Fall 2017.\n",
    "\n",
    "## \"Zestimates\" are estimated home values based on 7.5 million statistical and machine learning models that analyze hundreds of data points on each property. And, by continually improving the median margin of error (from 14% at the onset to 5% today), Zillow has since become established as one of the largest, most trusted marketplaces for real estate information in the U.S. and a leading example of impactful machine learning.\n",
    "\n",
    "## In this data science project, we will develop a machine learning algorithm that makes predictions about the future sale prices of homes. We will also build a model to improve the Zestimate residual error. And finally, we'll build a home valuation algorithm from the ground up, using external data sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "https://s3.amazonaws.com/hackerday.datascience/82/properties_2016.csv\n",
    "https://s3.amazonaws.com/hackerday.datascience/82/sample_submission.csv\n",
    "https://s3.amazonaws.com/hackerday.datascience/82/zillow_data_dictionary.xlsx\n",
    "https://s3.amazonaws.com/hackerday.datascience/82/train_2016.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "properties<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/properties_2016.csv',header=T,sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>parcelid</th><th scope=col>airconditioningtypeid</th><th scope=col>architecturalstyletypeid</th><th scope=col>basementsqft</th><th scope=col>bathroomcnt</th><th scope=col>bedroomcnt</th><th scope=col>buildingclasstypeid</th><th scope=col>buildingqualitytypeid</th><th scope=col>calculatedbathnbr</th><th scope=col>decktypeid</th><th scope=col>ellip.h</th><th scope=col>numberofstories</th><th scope=col>fireplaceflag</th><th scope=col>structuretaxvaluedollarcnt</th><th scope=col>taxvaluedollarcnt</th><th scope=col>assessmentyear</th><th scope=col>landtaxvaluedollarcnt</th><th scope=col>taxamount</th><th scope=col>taxdelinquencyflag</th><th scope=col>taxdelinquencyyear</th><th scope=col>censustractandblock</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>10754147</td><td>NA</td><td>NA</td><td>NA</td><td>0</td><td>0</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>&#8943;</td><td>NA</td><td></td><td>NA</td><td>9</td><td>2015</td><td>9</td><td>NA</td><td></td><td>NA</td><td>NA</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>10759547</td><td>NA</td><td>NA</td><td>NA</td><td>0</td><td>0</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>&#8943;</td><td>NA</td><td></td><td>NA</td><td>27516</td><td>2015</td><td>27516</td><td>NA</td><td></td><td>NA</td><td>NA</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>10843547</td><td>NA</td><td>NA</td><td>NA</td><td>0</td><td>0</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td><td>&#8943;</td><td>NA</td><td></td><td>650756</td><td>1413387</td><td>2015</td><td>762631</td><td>20800.37</td><td></td><td>NA</td><td>NA</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>10859147</td><td>NA</td><td>NA</td><td>NA</td><td>0</td><td>0</td><td>3</td><td>7</td><td>NA</td><td>NA</td><td>&#8943;</td><td>1</td><td></td><td>571346</td><td>1156834</td><td>2015</td><td>585488</td><td>14557.57</td><td></td><td>NA</td><td>NA</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>10879947</td><td>NA</td><td>NA</td><td>NA</td><td>0</td><td>0</td><td>4</td><td>NA</td><td>NA</td><td>NA</td><td>&#8943;</td><td>NA</td><td></td><td>193796</td><td>433491</td><td>2015</td><td>239695</td><td>5725.17</td><td></td><td>NA</td><td>NA</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>10898347</td><td>NA</td><td>NA</td><td>NA</td><td>0</td><td>0</td><td>4</td><td>7</td><td>NA</td><td>NA</td><td>&#8943;</td><td>1</td><td></td><td>176383</td><td>283315</td><td>2015</td><td>106932</td><td>3661.28</td><td></td><td>NA</td><td>NA</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllllllllllllllllllllllllllllllllllllllllllllllll}\n",
       "  & parcelid & airconditioningtypeid & architecturalstyletypeid & basementsqft & bathroomcnt & bedroomcnt & buildingclasstypeid & buildingqualitytypeid & calculatedbathnbr & decktypeid & ellip.h & numberofstories & fireplaceflag & structuretaxvaluedollarcnt & taxvaluedollarcnt & assessmentyear & landtaxvaluedollarcnt & taxamount & taxdelinquencyflag & taxdelinquencyyear & censustractandblock\\\\\n",
       "\\hline\n",
       "\t1 & 10754147 & NA & NA & NA & 0 & 0 & NA & NA & NA & NA & &#8943; & NA &  & NA & 9 & 2015 & 9 & NA &  & NA & NA\\\\\n",
       "\t2 & 10759547 & NA & NA & NA & 0 & 0 & NA & NA & NA & NA & &#8943; & NA &  & NA & 27516 & 2015 & 27516 & NA &  & NA & NA\\\\\n",
       "\t3 & 10843547 & NA & NA & NA & 0 & 0 & NA & NA & NA & NA & &#8943; & NA &  & 650756 & 1413387 & 2015 & 762631 & 20800.37 &  & NA & NA\\\\\n",
       "\t4 & 10859147 & NA & NA & NA & 0 & 0 & 3 & 7 & NA & NA & &#8943; & 1 &  & 571346 & 1156834 & 2015 & 585488 & 14557.57 &  & NA & NA\\\\\n",
       "\t5 & 10879947 & NA & NA & NA & 0 & 0 & 4 & NA & NA & NA & &#8943; & NA &  & 193796 & 433491 & 2015 & 239695 & 5725.17 &  & NA & NA\\\\\n",
       "\t6 & 10898347 & NA & NA & NA & 0 & 0 & 4 & 7 & NA & NA & &#8943; & 1 &  & 176383 & 283315 & 2015 & 106932 & 3661.28 &  & NA & NA\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  parcelid airconditioningtypeid architecturalstyletypeid basementsqft\n",
       "1 10754147                    NA                       NA           NA\n",
       "2 10759547                    NA                       NA           NA\n",
       "3 10843547                    NA                       NA           NA\n",
       "4 10859147                    NA                       NA           NA\n",
       "5 10879947                    NA                       NA           NA\n",
       "6 10898347                    NA                       NA           NA\n",
       "  bathroomcnt bedroomcnt buildingclasstypeid buildingqualitytypeid\n",
       "1           0          0                  NA                    NA\n",
       "2           0          0                  NA                    NA\n",
       "3           0          0                  NA                    NA\n",
       "4           0          0                   3                     7\n",
       "5           0          0                   4                    NA\n",
       "6           0          0                   4                     7\n",
       "  calculatedbathnbr decktypeid finishedfloor1squarefeet\n",
       "1                NA         NA                       NA\n",
       "2                NA         NA                       NA\n",
       "3                NA         NA                       NA\n",
       "4                NA         NA                       NA\n",
       "5                NA         NA                       NA\n",
       "6                NA         NA                       NA\n",
       "  calculatedfinishedsquarefeet finishedsquarefeet12 finishedsquarefeet13\n",
       "1                           NA                   NA                   NA\n",
       "2                           NA                   NA                   NA\n",
       "3                        73026                   NA                   NA\n",
       "4                         5068                   NA                   NA\n",
       "5                         1776                   NA                   NA\n",
       "6                         2400                   NA                   NA\n",
       "  finishedsquarefeet15 finishedsquarefeet50 finishedsquarefeet6 fips\n",
       "1                   NA                   NA                  NA 6037\n",
       "2                   NA                   NA                  NA 6037\n",
       "3                73026                   NA                  NA 6037\n",
       "4                 5068                   NA                  NA 6037\n",
       "5                 1776                   NA                  NA 6037\n",
       "6                 2400                   NA                  NA 6037\n",
       "  fireplacecnt fullbathcnt garagecarcnt garagetotalsqft hashottuborspa\n",
       "1           NA          NA           NA              NA               \n",
       "2           NA          NA           NA              NA               \n",
       "3           NA          NA           NA              NA               \n",
       "4           NA          NA           NA              NA               \n",
       "5           NA          NA           NA              NA               \n",
       "6           NA          NA           NA              NA               \n",
       "  heatingorsystemtypeid latitude  longitude lotsizesquarefeet poolcnt\n",
       "1                    NA 34144442 -118654084             85768      NA\n",
       "2                    NA 34140430 -118625364              4083      NA\n",
       "3                    NA 33989359 -118394633             63085      NA\n",
       "4                    NA 34148863 -118437206              7521      NA\n",
       "5                    NA 34194168 -118385816              8512      NA\n",
       "6                    NA 34171873 -118380906              2500      NA\n",
       "  poolsizesum pooltypeid10 pooltypeid2 pooltypeid7 propertycountylandusecode\n",
       "1          NA           NA          NA          NA                      010D\n",
       "2          NA           NA          NA          NA                      0109\n",
       "3          NA           NA          NA          NA                      1200\n",
       "4          NA           NA          NA          NA                      1200\n",
       "5          NA           NA          NA          NA                      1210\n",
       "6          NA           NA          NA          NA                      1210\n",
       "  propertylandusetypeid propertyzoningdesc rawcensustractandblock regionidcity\n",
       "1                   269                                  60378002        37688\n",
       "2                   261             LCA11*               60378001        37688\n",
       "3                    47               LAC2               60377030        51617\n",
       "4                    47               LAC2               60371412        12447\n",
       "5                    31               LAM1               60371232        12447\n",
       "6                    31               LAC4               60371252        12447\n",
       "  regionidcounty regionidneighborhood regionidzip roomcnt storytypeid\n",
       "1           3101                   NA       96337       0          NA\n",
       "2           3101                   NA       96337       0          NA\n",
       "3           3101                   NA       96095       0          NA\n",
       "4           3101                27080       96424       0          NA\n",
       "5           3101                46795       96450       0          NA\n",
       "6           3101                46795       96446       0          NA\n",
       "  threequarterbathnbr typeconstructiontypeid unitcnt yardbuildingsqft17\n",
       "1                  NA                     NA      NA                 NA\n",
       "2                  NA                     NA      NA                 NA\n",
       "3                  NA                     NA       2                 NA\n",
       "4                  NA                     NA      NA                 NA\n",
       "5                  NA                     NA       1                 NA\n",
       "6                  NA                     NA      NA                 NA\n",
       "  yardbuildingsqft26 yearbuilt numberofstories fireplaceflag\n",
       "1                 NA        NA              NA              \n",
       "2                 NA        NA              NA              \n",
       "3                 NA        NA              NA              \n",
       "4                 NA      1948               1              \n",
       "5                 NA      1947              NA              \n",
       "6                 NA      1943               1              \n",
       "  structuretaxvaluedollarcnt taxvaluedollarcnt assessmentyear\n",
       "1                         NA                 9           2015\n",
       "2                         NA             27516           2015\n",
       "3                     650756           1413387           2015\n",
       "4                     571346           1156834           2015\n",
       "5                     193796            433491           2015\n",
       "6                     176383            283315           2015\n",
       "  landtaxvaluedollarcnt taxamount taxdelinquencyflag taxdelinquencyyear\n",
       "1                     9        NA                                    NA\n",
       "2                 27516        NA                                    NA\n",
       "3                762631  20800.37                                    NA\n",
       "4                585488  14557.57                                    NA\n",
       "5                239695   5725.17                                    NA\n",
       "6                106932   3661.28                                    NA\n",
       "  censustractandblock\n",
       "1                  NA\n",
       "2                  NA\n",
       "3                  NA\n",
       "4                  NA\n",
       "5                  NA\n",
       "6                  NA"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "head(properties)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "transactions<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/train_2016.csv',header=T,sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>parcelid</th><th scope=col>logerror</th><th scope=col>transactiondate</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>11016594</td><td>0.0276</td><td>2016-01-01</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>14366692</td><td>-0.1684</td><td>2016-01-01</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>12098116</td><td>-0.004</td><td>2016-01-01</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>12643413</td><td>0.0218</td><td>2016-01-02</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>14432541</td><td>-0.005</td><td>2016-01-02</td></tr>\n",
       "\t<tr><th scope=row>6</th><td>11509835</td><td>-0.2705</td><td>2016-01-02</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       "  & parcelid & logerror & transactiondate\\\\\n",
       "\\hline\n",
       "\t1 & 11016594 & 0.0276 & 2016-01-01\\\\\n",
       "\t2 & 14366692 & -0.1684 & 2016-01-01\\\\\n",
       "\t3 & 12098116 & -0.004 & 2016-01-01\\\\\n",
       "\t4 & 12643413 & 0.0218 & 2016-01-02\\\\\n",
       "\t5 & 14432541 & -0.005 & 2016-01-02\\\\\n",
       "\t6 & 11509835 & -0.2705 & 2016-01-02\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  parcelid logerror transactiondate\n",
       "1 11016594   0.0276      2016-01-01\n",
       "2 14366692  -0.1684      2016-01-01\n",
       "3 12098116  -0.0040      2016-01-01\n",
       "4 12643413   0.0218      2016-01-02\n",
       "5 14432541  -0.0050      2016-01-02\n",
       "6 11509835  -0.2705      2016-01-02"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "head(transactions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from transactions and properties datasets please extract the %missing and NA values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'parcelid'</li>\n",
       "\t<li>'airconditioningtypeid'</li>\n",
       "\t<li>'architecturalstyletypeid'</li>\n",
       "\t<li>'basementsqft'</li>\n",
       "\t<li>'bathroomcnt'</li>\n",
       "\t<li>'bedroomcnt'</li>\n",
       "\t<li>'buildingclasstypeid'</li>\n",
       "\t<li>'buildingqualitytypeid'</li>\n",
       "\t<li>'calculatedbathnbr'</li>\n",
       "\t<li>'decktypeid'</li>\n",
       "\t<li>'finishedfloor1squarefeet'</li>\n",
       "\t<li>'calculatedfinishedsquarefeet'</li>\n",
       "\t<li>'finishedsquarefeet12'</li>\n",
       "\t<li>'finishedsquarefeet13'</li>\n",
       "\t<li>'finishedsquarefeet15'</li>\n",
       "\t<li>'finishedsquarefeet50'</li>\n",
       "\t<li>'finishedsquarefeet6'</li>\n",
       "\t<li>'fips'</li>\n",
       "\t<li>'fireplacecnt'</li>\n",
       "\t<li>'fullbathcnt'</li>\n",
       "\t<li>'garagecarcnt'</li>\n",
       "\t<li>'garagetotalsqft'</li>\n",
       "\t<li>'hashottuborspa'</li>\n",
       "\t<li>'heatingorsystemtypeid'</li>\n",
       "\t<li>'latitude'</li>\n",
       "\t<li>'longitude'</li>\n",
       "\t<li>'lotsizesquarefeet'</li>\n",
       "\t<li>'poolcnt'</li>\n",
       "\t<li>'poolsizesum'</li>\n",
       "\t<li>'pooltypeid10'</li>\n",
       "\t<li>'pooltypeid2'</li>\n",
       "\t<li>'pooltypeid7'</li>\n",
       "\t<li>'propertycountylandusecode'</li>\n",
       "\t<li>'propertylandusetypeid'</li>\n",
       "\t<li>'propertyzoningdesc'</li>\n",
       "\t<li>'rawcensustractandblock'</li>\n",
       "\t<li>'regionidcity'</li>\n",
       "\t<li>'regionidcounty'</li>\n",
       "\t<li>'regionidneighborhood'</li>\n",
       "\t<li>'regionidzip'</li>\n",
       "\t<li>'roomcnt'</li>\n",
       "\t<li>'storytypeid'</li>\n",
       "\t<li>'threequarterbathnbr'</li>\n",
       "\t<li>'typeconstructiontypeid'</li>\n",
       "\t<li>'unitcnt'</li>\n",
       "\t<li>'yardbuildingsqft17'</li>\n",
       "\t<li>'yardbuildingsqft26'</li>\n",
       "\t<li>'yearbuilt'</li>\n",
       "\t<li>'numberofstories'</li>\n",
       "\t<li>'fireplaceflag'</li>\n",
       "\t<li>'structuretaxvaluedollarcnt'</li>\n",
       "\t<li>'taxvaluedollarcnt'</li>\n",
       "\t<li>'assessmentyear'</li>\n",
       "\t<li>'landtaxvaluedollarcnt'</li>\n",
       "\t<li>'taxamount'</li>\n",
       "\t<li>'taxdelinquencyflag'</li>\n",
       "\t<li>'taxdelinquencyyear'</li>\n",
       "\t<li>'censustractandblock'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'parcelid'\n",
       "\\item 'airconditioningtypeid'\n",
       "\\item 'architecturalstyletypeid'\n",
       "\\item 'basementsqft'\n",
       "\\item 'bathroomcnt'\n",
       "\\item 'bedroomcnt'\n",
       "\\item 'buildingclasstypeid'\n",
       "\\item 'buildingqualitytypeid'\n",
       "\\item 'calculatedbathnbr'\n",
       "\\item 'decktypeid'\n",
       "\\item 'finishedfloor1squarefeet'\n",
       "\\item 'calculatedfinishedsquarefeet'\n",
       "\\item 'finishedsquarefeet12'\n",
       "\\item 'finishedsquarefeet13'\n",
       "\\item 'finishedsquarefeet15'\n",
       "\\item 'finishedsquarefeet50'\n",
       "\\item 'finishedsquarefeet6'\n",
       "\\item 'fips'\n",
       "\\item 'fireplacecnt'\n",
       "\\item 'fullbathcnt'\n",
       "\\item 'garagecarcnt'\n",
       "\\item 'garagetotalsqft'\n",
       "\\item 'hashottuborspa'\n",
       "\\item 'heatingorsystemtypeid'\n",
       "\\item 'latitude'\n",
       "\\item 'longitude'\n",
       "\\item 'lotsizesquarefeet'\n",
       "\\item 'poolcnt'\n",
       "\\item 'poolsizesum'\n",
       "\\item 'pooltypeid10'\n",
       "\\item 'pooltypeid2'\n",
       "\\item 'pooltypeid7'\n",
       "\\item 'propertycountylandusecode'\n",
       "\\item 'propertylandusetypeid'\n",
       "\\item 'propertyzoningdesc'\n",
       "\\item 'rawcensustractandblock'\n",
       "\\item 'regionidcity'\n",
       "\\item 'regionidcounty'\n",
       "\\item 'regionidneighborhood'\n",
       "\\item 'regionidzip'\n",
       "\\item 'roomcnt'\n",
       "\\item 'storytypeid'\n",
       "\\item 'threequarterbathnbr'\n",
       "\\item 'typeconstructiontypeid'\n",
       "\\item 'unitcnt'\n",
       "\\item 'yardbuildingsqft17'\n",
       "\\item 'yardbuildingsqft26'\n",
       "\\item 'yearbuilt'\n",
       "\\item 'numberofstories'\n",
       "\\item 'fireplaceflag'\n",
       "\\item 'structuretaxvaluedollarcnt'\n",
       "\\item 'taxvaluedollarcnt'\n",
       "\\item 'assessmentyear'\n",
       "\\item 'landtaxvaluedollarcnt'\n",
       "\\item 'taxamount'\n",
       "\\item 'taxdelinquencyflag'\n",
       "\\item 'taxdelinquencyyear'\n",
       "\\item 'censustractandblock'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'parcelid'\n",
       "2. 'airconditioningtypeid'\n",
       "3. 'architecturalstyletypeid'\n",
       "4. 'basementsqft'\n",
       "5. 'bathroomcnt'\n",
       "6. 'bedroomcnt'\n",
       "7. 'buildingclasstypeid'\n",
       "8. 'buildingqualitytypeid'\n",
       "9. 'calculatedbathnbr'\n",
       "10. 'decktypeid'\n",
       "11. 'finishedfloor1squarefeet'\n",
       "12. 'calculatedfinishedsquarefeet'\n",
       "13. 'finishedsquarefeet12'\n",
       "14. 'finishedsquarefeet13'\n",
       "15. 'finishedsquarefeet15'\n",
       "16. 'finishedsquarefeet50'\n",
       "17. 'finishedsquarefeet6'\n",
       "18. 'fips'\n",
       "19. 'fireplacecnt'\n",
       "20. 'fullbathcnt'\n",
       "21. 'garagecarcnt'\n",
       "22. 'garagetotalsqft'\n",
       "23. 'hashottuborspa'\n",
       "24. 'heatingorsystemtypeid'\n",
       "25. 'latitude'\n",
       "26. 'longitude'\n",
       "27. 'lotsizesquarefeet'\n",
       "28. 'poolcnt'\n",
       "29. 'poolsizesum'\n",
       "30. 'pooltypeid10'\n",
       "31. 'pooltypeid2'\n",
       "32. 'pooltypeid7'\n",
       "33. 'propertycountylandusecode'\n",
       "34. 'propertylandusetypeid'\n",
       "35. 'propertyzoningdesc'\n",
       "36. 'rawcensustractandblock'\n",
       "37. 'regionidcity'\n",
       "38. 'regionidcounty'\n",
       "39. 'regionidneighborhood'\n",
       "40. 'regionidzip'\n",
       "41. 'roomcnt'\n",
       "42. 'storytypeid'\n",
       "43. 'threequarterbathnbr'\n",
       "44. 'typeconstructiontypeid'\n",
       "45. 'unitcnt'\n",
       "46. 'yardbuildingsqft17'\n",
       "47. 'yardbuildingsqft26'\n",
       "48. 'yearbuilt'\n",
       "49. 'numberofstories'\n",
       "50. 'fireplaceflag'\n",
       "51. 'structuretaxvaluedollarcnt'\n",
       "52. 'taxvaluedollarcnt'\n",
       "53. 'assessmentyear'\n",
       "54. 'landtaxvaluedollarcnt'\n",
       "55. 'taxamount'\n",
       "56. 'taxdelinquencyflag'\n",
       "57. 'taxdelinquencyyear'\n",
       "58. 'censustractandblock'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] \"parcelid\"                     \"airconditioningtypeid\"       \n",
       " [3] \"architecturalstyletypeid\"     \"basementsqft\"                \n",
       " [5] \"bathroomcnt\"                  \"bedroomcnt\"                  \n",
       " [7] \"buildingclasstypeid\"          \"buildingqualitytypeid\"       \n",
       " [9] \"calculatedbathnbr\"            \"decktypeid\"                  \n",
       "[11] \"finishedfloor1squarefeet\"     \"calculatedfinishedsquarefeet\"\n",
       "[13] \"finishedsquarefeet12\"         \"finishedsquarefeet13\"        \n",
       "[15] \"finishedsquarefeet15\"         \"finishedsquarefeet50\"        \n",
       "[17] \"finishedsquarefeet6\"          \"fips\"                        \n",
       "[19] \"fireplacecnt\"                 \"fullbathcnt\"                 \n",
       "[21] \"garagecarcnt\"                 \"garagetotalsqft\"             \n",
       "[23] \"hashottuborspa\"               \"heatingorsystemtypeid\"       \n",
       "[25] \"latitude\"                     \"longitude\"                   \n",
       "[27] \"lotsizesquarefeet\"            \"poolcnt\"                     \n",
       "[29] \"poolsizesum\"                  \"pooltypeid10\"                \n",
       "[31] \"pooltypeid2\"                  \"pooltypeid7\"                 \n",
       "[33] \"propertycountylandusecode\"    \"propertylandusetypeid\"       \n",
       "[35] \"propertyzoningdesc\"           \"rawcensustractandblock\"      \n",
       "[37] \"regionidcity\"                 \"regionidcounty\"              \n",
       "[39] \"regionidneighborhood\"         \"regionidzip\"                 \n",
       "[41] \"roomcnt\"                      \"storytypeid\"                 \n",
       "[43] \"threequarterbathnbr\"          \"typeconstructiontypeid\"      \n",
       "[45] \"unitcnt\"                      \"yardbuildingsqft17\"          \n",
       "[47] \"yardbuildingsqft26\"           \"yearbuilt\"                   \n",
       "[49] \"numberofstories\"              \"fireplaceflag\"               \n",
       "[51] \"structuretaxvaluedollarcnt\"   \"taxvaluedollarcnt\"           \n",
       "[53] \"assessmentyear\"               \"landtaxvaluedollarcnt\"       \n",
       "[55] \"taxamount\"                    \"taxdelinquencyflag\"          \n",
       "[57] \"taxdelinquencyyear\"           \"censustractandblock\"         "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "colnames(properties)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "    parcelid         airconditioningtypeid architecturalstyletypeid\n",
       " Min.   : 10711789   Min.   : 1.000        Min.   : 2.00           \n",
       " 1st Qu.: 11588354   1st Qu.: 1.000        1st Qu.: 7.00           \n",
       " Median : 12422302   Median : 1.000        Median : 7.00           \n",
       " Mean   : 12858419   Mean   : 1.739        Mean   : 7.17           \n",
       " 3rd Qu.: 13921014   3rd Qu.: 1.000        3rd Qu.: 7.00           \n",
       " Max.   :163187349   Max.   :13.000        Max.   :21.00           \n",
       "                     NA's   :28924         NA's   :39929           \n",
       "  basementsqft     bathroomcnt       bedroomcnt     buildingclasstypeid\n",
       " Min.   :  63.0   Min.   : 0.000   Min.   : 0.000   Min.   :2.0        \n",
       " 1st Qu.: 224.0   1st Qu.: 2.000   1st Qu.: 2.000   1st Qu.:4.0        \n",
       " Median : 600.0   Median : 2.000   Median : 3.000   Median :4.0        \n",
       " Mean   : 629.3   Mean   : 2.196   Mean   : 3.077   Mean   :3.8        \n",
       " 3rd Qu.: 782.0   3rd Qu.: 3.000   3rd Qu.: 4.000   3rd Qu.:4.0        \n",
       " Max.   :1572.0   Max.   :20.000   Max.   :16.000   Max.   :5.0        \n",
       " NA's   :39974                                      NA's   :39820      \n",
       " buildingqualitytypeid calculatedbathnbr   decktypeid   \n",
       " Min.   : 1.000        Min.   : 1.000    Min.   :66     \n",
       " 1st Qu.: 4.000        1st Qu.: 2.000    1st Qu.:66     \n",
       " Median : 7.000        Median : 2.000    Median :66     \n",
       " Mean   : 5.805        Mean   : 2.288    Mean   :66     \n",
       " 3rd Qu.: 7.000        3rd Qu.: 3.000    3rd Qu.:66     \n",
       " Max.   :12.000        Max.   :20.000    Max.   :66     \n",
       " NA's   :11864         NA's   :1610      NA's   :39758  \n",
       " finishedfloor1squarefeet calculatedfinishedsquarefeet finishedsquarefeet12\n",
       " Min.   :   64            Min.   :    99               Min.   :   99       \n",
       " 1st Qu.: 1013            1st Qu.:  1205               1st Qu.: 1187       \n",
       " Median : 1287            Median :  1560               Median : 1525       \n",
       " Mean   : 1369            Mean   :  1809               Mean   : 1744       \n",
       " 3rd Qu.: 1605            3rd Qu.:  2114               3rd Qu.: 2048       \n",
       " Max.   :15998            Max.   :145002               Max.   :20433       \n",
       " NA's   :37096            NA's   :546                  NA's   :3655        \n",
       " finishedsquarefeet13 finishedsquarefeet15 finishedsquarefeet50\n",
       " Min.   : 470         Min.   :   148       Min.   :   64       \n",
       " 1st Qu.:1000         1st Qu.:  1680       1st Qu.: 1015       \n",
       " Median :1296         Median :  2143       Median : 1288       \n",
       " Mean   :1200         Mean   :  2667       Mean   : 1378       \n",
       " 3rd Qu.:1440         3rd Qu.:  2918       3rd Qu.: 1606       \n",
       " Max.   :2400         Max.   :145002       Max.   :15998       \n",
       " NA's   :39874        NA's   :37238        NA's   :37096       \n",
       " finishedsquarefeet6      fips       fireplacecnt    fullbathcnt    \n",
       " Min.   :  384       Min.   :6037   Min.   :1.00    Min.   : 1.000  \n",
       " 1st Qu.: 1149       1st Qu.:6037   1st Qu.:1.00    1st Qu.: 2.000  \n",
       " Median : 1890       Median :6037   Median :1.00    Median : 2.000  \n",
       " Mean   : 2278       Mean   :6047   Mean   :1.16    Mean   : 2.243  \n",
       " 3rd Qu.: 3066       3rd Qu.:6059   3rd Qu.:1.00    3rd Qu.: 3.000  \n",
       " Max.   :15341       Max.   :6111   Max.   :9.00    Max.   :20.000  \n",
       " NA's   :39776                      NA's   :36222   NA's   :1610    \n",
       "  garagecarcnt    garagetotalsqft  hashottuborspa heatingorsystemtypeid\n",
       " Min.   : 0.000   Min.   :   0.0       :39210     Min.   : 2.000       \n",
       " 1st Qu.: 2.000   1st Qu.: 324.0   true:  789     1st Qu.: 2.000       \n",
       " Median : 2.000   Median : 441.0                  Median : 2.000       \n",
       " Mean   : 1.822   Mean   : 384.1                  Mean   : 3.955       \n",
       " 3rd Qu.: 2.000   3rd Qu.: 494.0                  3rd Qu.: 7.000       \n",
       " Max.   :12.000   Max.   :3598.0                  Max.   :24.000       \n",
       " NA's   :30150    NA's   :30150                   NA's   :14003        \n",
       "    latitude          longitude          lotsizesquarefeet    poolcnt     \n",
       " Min.   :33339384   Min.   :-119448392   Min.   :    129   Min.   :1      \n",
       " 1st Qu.:33859220   1st Qu.:-118407176   1st Qu.:   5691   1st Qu.:1      \n",
       " Median :34029439   Median :-118200629   Median :   7002   Median :1      \n",
       " Mean   :34021505   Mean   :-118228695   Mean   :  23259   Mean   :1      \n",
       " 3rd Qu.:34169619   3rd Qu.:-117984600   3rd Qu.:   9993   3rd Qu.:1      \n",
       " Max.   :34786862   Max.   :-117557115   Max.   :6971010   Max.   :1      \n",
       "                                         NA's   :3031      NA's   :32993  \n",
       "  poolsizesum      pooltypeid10    pooltypeid2     pooltypeid7   \n",
       " Min.   :  64.0   Min.   :1       Min.   :1       Min.   :1      \n",
       " 1st Qu.: 448.0   1st Qu.:1       1st Qu.:1       1st Qu.:1      \n",
       " Median : 500.0   Median :1       Median :1       Median :1      \n",
       " Mean   : 531.2   Mean   :1       Mean   :1       Mean   :1      \n",
       " 3rd Qu.: 600.0   3rd Qu.:1       3rd Qu.:1       3rd Qu.:1      \n",
       " Max.   :2176.0   Max.   :1       Max.   :1       Max.   :1      \n",
       " NA's   :39640    NA's   :39540   NA's   :39669   NA's   :33323  \n",
       " propertycountylandusecode propertylandusetypeid propertyzoningdesc\n",
       " 0100   :16922             Min.   : 31.0                :11282     \n",
       " 122    : 5332             1st Qu.:261.0         LAR1   : 4062     \n",
       " 0101   : 3719             Median :261.0         LAR3   :  913     \n",
       " 010C   : 3123             Mean   :259.9         LARS   :  784     \n",
       " 1111   : 1823             3rd Qu.:261.0         LBR1N  :  779     \n",
       " 0200   : 1523             Max.   :275.0         LAR2   :  732     \n",
       " (Other): 7557                                   (Other):21447     \n",
       " rawcensustractandblock  regionidcity    regionidcounty regionidneighborhood\n",
       " Min.   :60371011       Min.   :  3491   Min.   :1286   Min.   :  6952      \n",
       " 1st Qu.:60373010       1st Qu.: 12447   1st Qu.:2061   1st Qu.: 46736      \n",
       " Median :60375502       Median : 24832   Median :3101   Median :118887      \n",
       " Mean   :60474742       Mean   : 35796   Mean   :2672   Mean   :191890      \n",
       " 3rd Qu.:60590218       3rd Qu.: 45602   3rd Qu.:3101   3rd Qu.:274705      \n",
       " Max.   :61110091       Max.   :396556   Max.   :3101   Max.   :764167      \n",
       "                        NA's   :625                     NA's   :23946       \n",
       "  regionidzip        roomcnt        storytypeid    threequarterbathnbr\n",
       " Min.   : 95982   Min.   : 0.000   Min.   :7       Min.   :1.00       \n",
       " 1st Qu.: 96163   1st Qu.: 0.000   1st Qu.:7       1st Qu.:1.00       \n",
       " Median : 96364   Median : 0.000   Median :7       Median :1.00       \n",
       " Mean   : 96535   Mean   : 1.259   Mean   :7       Mean   :1.01       \n",
       " 3rd Qu.: 96961   3rd Qu.: 0.000   3rd Qu.:7       3rd Qu.:1.00       \n",
       " Max.   :399675   Max.   :86.000   Max.   :7       Max.   :4.00       \n",
       " NA's   :29                        NA's   :39974   NA's   :36591      \n",
       " typeconstructiontypeid    unitcnt        yardbuildingsqft17 yardbuildingsqft26\n",
       " Min.   :6              Min.   :  1.000   Min.   :  24.0     Min.   :  14.0    \n",
       " 1st Qu.:6              1st Qu.:  1.000   1st Qu.: 180.0     1st Qu.:  96.0    \n",
       " Median :6              Median :  1.000   Median : 264.0     Median : 192.0    \n",
       " Mean   :6              Mean   :  1.171   Mean   : 318.9     Mean   : 272.5    \n",
       " 3rd Qu.:6              3rd Qu.:  1.000   3rd Qu.: 380.0     3rd Qu.: 360.0    \n",
       " Max.   :6              Max.   :127.000   Max.   :6407.0     Max.   :1280.0    \n",
       " NA's   :39926          NA's   :11339     NA's   :38810      NA's   :39966     \n",
       "   yearbuilt    numberofstories fireplaceflag structuretaxvaluedollarcnt\n",
       " Min.   :1862   Min.   :1.00        :39940    Min.   :       5          \n",
       " 1st Qu.:1949   1st Qu.:1.00    true:   59    1st Qu.:   74314          \n",
       " Median :1961   Median :1.00                  Median :  121444          \n",
       " Mean   :1963   Mean   :1.39                  Mean   :  169748          \n",
       " 3rd Qu.:1980   3rd Qu.:2.00                  3rd Qu.:  193796          \n",
       " Max.   :2015   Max.   :4.00                  Max.   :65407469          \n",
       " NA's   :611    NA's   :32149                 NA's   :658               \n",
       " taxvaluedollarcnt  assessmentyear landtaxvaluedollarcnt   taxamount      \n",
       " Min.   :       9   Min.   :2014   Min.   :       4      Min.   :      5  \n",
       " 1st Qu.:  174782   1st Qu.:2015   1st Qu.:   71021      1st Qu.:   2415  \n",
       " Median :  299194   Median :2015   Median :  162032      Median :   3915  \n",
       " Mean   :  413678   Mean   :2015   Mean   :  246634      Mean   :   5310  \n",
       " 3rd Qu.:  477595   3rd Qu.:2015   3rd Qu.:  299642      3rd Qu.:   6061  \n",
       " Max.   :66492479   Max.   :2015   Max.   :22101269      Max.   :1195296  \n",
       " NA's   :467                       NA's   :769           NA's   :264      \n",
       " taxdelinquencyflag taxdelinquencyyear censustractandblock\n",
       "  :39157            Min.   : 6.00      Min.   :6.037e+13  \n",
       " Y:  842            1st Qu.:14.00      1st Qu.:6.037e+13  \n",
       "                    Median :14.00      Median :6.038e+13  \n",
       "                    Mean   :13.95      Mean   :6.048e+13  \n",
       "                    3rd Qu.:15.00      3rd Qu.:6.059e+13  \n",
       "                    Max.   :15.00      Max.   :6.111e+13  \n",
       "                    NA's   :39157      NA's   :933        "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary(properties)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>39999</li>\n",
       "\t<li>58</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 39999\n",
       "\\item 58\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 39999\n",
       "2. 58\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 39999    58"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dim(properties)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(data.table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(dplyr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(ggplot2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(stringr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#library(DT)\n",
    "#library(tidyr)\n",
    "library(corrplot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#library(leaflet)\n",
    "library(lubridate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(data.table)\n",
    "library(dplyr)\n",
    "library(ggplot2)\n",
    "library(stringr)\n",
    "library(DT)\n",
    "library(tidyr)\n",
    "library(corrplot)\n",
    "library(leaflet)\n",
    "library(lubridate)\n",
    "\n",
    "properties<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/properties_2016.csv',header=T,sep=',')\n",
    "\n",
    "transactions<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/train_2016.csv',header=T,sep=',')\n",
    "\n",
    "summary(properties)\n",
    "\n",
    "properties <- properties %>% \n",
    "  mutate(taxdelinquencyflag = ifelse(taxdelinquencyflag=='Y',1,0),\n",
    "         fireplaceflag= ifelse(fireplaceflag=='Y',1,0),\n",
    "         hashottuborspa=ifelse(hashottuborspa=='Y',1,0)\n",
    "         )\n",
    "\n",
    "# Which particular month contibutes maximum number of transactions\n",
    "# distribution of transactions over the length of the time period we have\n",
    "\n",
    "Num_tr_by_M <- transactions %>%\n",
    "  mutate(year_month=make_date(year=year(transactiondate),\n",
    "                         month=month(transactiondate)))\n",
    "\n",
    "Num_tr_by_M %>% \n",
    "  group_by(year_month) %>% count() %>%\n",
    "  ggplot(aes(x=year_month,y=n)) +\n",
    "  geom_bar(stat = 'identity',fill='red')\n",
    "  \n",
    "# try to understand the distribution of the target variable\n",
    "transactions %>%\n",
    "  ggplot(aes(x=logerror)) +\n",
    "  geom_histogram(bins = 1000,fill='blue') +\n",
    "  theme_bw() + ylab('Count')\n",
    "\n",
    "#absolute value of the logerror\n",
    "transactions<-transactions %>%\n",
    "  mutate(abs_logerror=abs(logerror))\n",
    "\n",
    "transactions %>%\n",
    "  ggplot(aes(x=abs_logerror)) +\n",
    "  geom_histogram(bins = 500,fill='blue') +\n",
    "  theme_bw() + ylab('Count')\n",
    "\n",
    "#Is there any variation(trend/over time) in the absolute value of error?\n",
    "transactions %>%\n",
    "  mutate(year_month=make_date(year=year(transactiondate),\n",
    "                              month=month(transactiondate))) %>%\n",
    "  group_by(year_month) %>% \n",
    "  summarise(mean_abs_logerror=mean(abs_logerror)) %>%\n",
    "  ggplot(aes(x=year_month,y=mean_abs_logerror)) +\n",
    "  geom_line(size=1.2,fill='red') +\n",
    "    geom_point(size=4,color='blue')\n",
    "\n",
    "#Is there any variation(trend/over time) in the log error?\n",
    "transactions %>%\n",
    "  mutate(year_month=make_date(year=year(transactiondate),\n",
    "                              month=month(transactiondate))) %>%\n",
    "  group_by(year_month) %>% \n",
    "  summarise(mean_logerror=mean(logerror)) %>%\n",
    "  ggplot(aes(x=year_month,y=mean_logerror)) +\n",
    "  geom_line(size=1.2,fill='red') +\n",
    "  geom_point(size=4,color='red')\n",
    "\n",
    "# How many missing values are there for each feature ?\n",
    "# How many feauteres are there with zero missing values?\n",
    "missing_values <- properties %>%\n",
    "  summarise_all(funs(sum(is.na(.))/n()))\n",
    "\n",
    "missing_values <- gather(missing_values,key='feature',value = 'missing_percentage')\n",
    "missing_values %>%\n",
    "  ggplot(aes(x=reorder(feature,-missing_percentage),y=missing_percentage)) +\n",
    "  geom_bar(stat = 'identity',fill='orange') +\n",
    "  coord_flip()\n",
    "  \n",
    "# finding out relevant/good features where the missing % < 15%\n",
    "good_features <- filter(missing_values,missing_percentage<0.85)\n",
    "\n",
    "#understanding the relationship between good features and the target variable\n",
    "# join the two data frames\n",
    "vars <- good_features$feature %>% as.numeric()\n",
    "\n",
    "c_join <- transactions %>%\n",
    "  left_join(properties,by='parcelid')\n",
    "\n",
    "tab<- c_join %>% select(one_of(c(vars,'abs_logerror')))\n",
    "\n",
    "corrplot(cor(tab,use='complete.obs'),type = 'lower')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#Day 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(data.table)\n",
    "library(dplyr)\n",
    "library(ggplot2)\n",
    "library(stringr)\n",
    "library(DT)\n",
    "library(tidyr)\n",
    "library(corrplot)\n",
    "library(leaflet)\n",
    "library(lubridate)\n",
    "\n",
    "properties<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/properties_2016.csv',header=T,sep=',')\n",
    "\n",
    "transactions<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/train_2016.csv',header=T,sep=',')\n",
    "\n",
    "summary(properties)\n",
    "\n",
    "properties <- properties %>% \n",
    "  mutate(taxdelinquencyflag = ifelse(taxdelinquencyflag=='Y',1,0),\n",
    "         fireplaceflag= ifelse(fireplaceflag=='Y',1,0),\n",
    "         hashottuborspa=ifelse(hashottuborspa=='Y',1,0)\n",
    "         )\n",
    "\n",
    "# Which particular month contibutes maximum number of transactions\n",
    "# distribution of transactions over the length of the time period we have\n",
    "\n",
    "Num_tr_by_M <- transactions %>%\n",
    "  mutate(year_month=make_date(year=year(transactiondate),\n",
    "                         month=month(transactiondate)))\n",
    "\n",
    "Num_tr_by_M %>% \n",
    "  group_by(year_month) %>% count() %>%\n",
    "  ggplot(aes(x=year_month,y=n)) +\n",
    "  geom_bar(stat = 'identity',fill='red')\n",
    "  \n",
    "# try to understand the distribution of the target variable\n",
    "transactions %>%\n",
    "  ggplot(aes(x=logerror)) +\n",
    "  geom_histogram(bins = 1000,fill='blue') +\n",
    "  theme_bw() + ylab('Count')\n",
    "\n",
    "#absolute value of the logerror\n",
    "transactions<-transactions %>%\n",
    "  mutate(abs_logerror=abs(logerror))\n",
    "\n",
    "transactions %>%\n",
    "  ggplot(aes(x=abs_logerror)) +\n",
    "  geom_histogram(bins = 500,fill='blue') +\n",
    "  theme_bw() + ylab('Count')\n",
    "\n",
    "#Is there any variation(trend/over time) in the absolute value of error?\n",
    "transactions %>%\n",
    "  mutate(year_month=make_date(year=year(transactiondate),\n",
    "                              month=month(transactiondate))) %>%\n",
    "  group_by(year_month) %>% \n",
    "  summarise(mean_abs_logerror=mean(abs_logerror)) %>%\n",
    "  ggplot(aes(x=year_month,y=mean_abs_logerror)) +\n",
    "  geom_line(size=1.2,fill='red') +\n",
    "    geom_point(size=4,color='blue')\n",
    "\n",
    "#Is there any variation(trend/over time) in the log error?\n",
    "transactions %>%\n",
    "  mutate(year_month=make_date(year=year(transactiondate),\n",
    "                              month=month(transactiondate))) %>%\n",
    "  group_by(year_month) %>% \n",
    "  summarise(mean_logerror=mean(logerror)) %>%\n",
    "  ggplot(aes(x=year_month,y=mean_logerror)) +\n",
    "  geom_line(size=1.2,fill='red') +\n",
    "  geom_point(size=4,color='red')\n",
    "\n",
    "# How many missing values are there for each feature ?\n",
    "# How many feauteres are there with zero missing values?\n",
    "missing_values <- properties %>%\n",
    "  summarise_all(funs(sum(is.na(.))/n()))\n",
    "\n",
    "missing_values <- gather(missing_values,key='feature',value = 'missing_percentage')\n",
    "missing_values %>%\n",
    "  ggplot(aes(x=reorder(feature,-missing_percentage),y=missing_percentage)) +\n",
    "  geom_bar(stat = 'identity',fill='orange') +\n",
    "  coord_flip()\n",
    "  \n",
    "# finding out relevant/good features where the missing % < 15%\n",
    "good_features <- filter(missing_values,missing_percentage<0.85)\n",
    "\n",
    "#understanding the relationship between good features and the target variable\n",
    "# join the two data frames\n",
    "vars <- good_features$feature %>% as.numeric()\n",
    "\n",
    "c_join <- transactions %>%\n",
    "  left_join(properties,by='parcelid')\n",
    "\n",
    "tab<- c_join %>% select(one_of(c(vars,'abs_logerror')))\n",
    "\n",
    "corrplot(cor(tab,use='complete.obs'),type = 'lower')\n",
    "\n",
    "# understanding the shape of the distribution\n",
    "properties %>% ggplot(aes(x=yearbuilt)) + geom_line(stat = 'density',color='red')\n",
    "\n",
    "c_join %>% \n",
    "  group_by(yearbuilt) %>%\n",
    "  summarize(mean_abs_logerror=mean(abs(logerror)),n()) %>%\n",
    "  ggplot(aes(x=yearbuilt,y=mean_abs_logerror)) + \n",
    "  geom_point(color='red')\n",
    "\n",
    "###where the zestimate variable predicts well\n",
    "transactions <- transactions %>% \n",
    "  mutate(percentile = cut(abs_logerror,quantile(abs_logerror,\n",
    "                                                probs=c(0,0.1,0.25,0.75,0.9,1),names=F),\n",
    "                          include.lowest=T,labels=F))\n",
    "\n",
    "####Model Implementation #########\n",
    "prop<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/properties_2016.csv',header=T,sep=',')\n",
    "\n",
    "train<-read.csv('https://s3.amazonaws.com/hackerday.datascience/82/train_2016.csv',header=T,sep=',')\n",
    "\n",
    "# feature enginerring\n",
    "prop$hashottuborspa<-ifelse(prop$hashottuborspa=='true',1,0)\n",
    "prop$fireplaceflag<-ifelse(prop$fireplaceflag=='true',1,0)\n",
    "prop$taxdelinquencyflag<-ifelse(prop$taxdelinquencyflag=='Y',1,0)\n",
    "prop$propertycountylandusecode<-as.numeric(prop$propertycountylandusecode)\n",
    "prop$propertyzoningdesc<-as.numeric(prop$propertyzoningdesc)\n",
    "\n",
    "prop<-data.table(prop)\n",
    "train<-data.table(train)\n",
    "\n",
    "setkey(prop,parcelid)\n",
    "setkey(train,parcelid)\n",
    "\n",
    "training <-prop[train]\n",
    "\n",
    "target <-training$logerror\n",
    "\n",
    "dtrain <-training[,!c('logerror','parcelid','transactiondate'),with=F]\n",
    "\n",
    "feature_names <- names(dtrain)\n",
    "\n",
    "#XGBoost set up\n",
    "library(data.table)\n",
    "library(caret)\n",
    "library(xgboost)\n",
    "\n",
    "dtest <- xgb.DMatrix(data=as.matrix(prop[,..feature_names]))\n",
    "dtrain <- xgb.DMatrix(data=as.matrix(dtrain),label=target)\n",
    "\n",
    "# cross validation scheme to avoid model overfitting\n",
    "\n",
    "foldsCV <- createFolds(target,k=10,list = T,returnTrain = F)\n",
    "\n",
    "# hyperparameter tuning\n",
    "param <- list(\n",
    "  objective='reg:linear',\n",
    "  booster='gbtree',\n",
    "  eval_metric='mae',\n",
    "  eta=0.005,\n",
    "  max_depth=4,\n",
    "  min_child_weight=4,\n",
    "  subsample=0.5,\n",
    "  colsample_bytree=0.5,\n",
    "  gamma=0.01\n",
    ")\n",
    "\n",
    "# train a simple model\n",
    "# this function should be levergaed for hyperparameter optimization (gridExtra)\n",
    "\n",
    "xgb_mod <- xgb.train(data=dtrain,\n",
    "                     params=param,\n",
    "                     nrounds = 2500,\n",
    "                     print_every_n = 5)\n",
    "\n",
    "# view the model results\n",
    "# feature importance view\n",
    "importance_matrix <-xgb.importance(feature_names,model=xgb_mod)\n",
    "xgb.plot.importance(importance_matrix)\n",
    "\n",
    "#predict using the test dataset\n",
    "preds <- predict(xgb_mod,dtest)\n",
    "\n",
    "#xgb_mod$params$eval_metric\n",
    "\n",
    "results <- data.table(parcelid=prop$parcelid,\n",
    "                      '201610'=preds,\n",
    "                      '201611'=preds,\n",
    "                      '201612'=preds,\n",
    "                      '201710'=preds,\n",
    "                      '201711'=preds,\n",
    "                      '201712'=preds)\n",
    "head(results)\n",
    "\n",
    "# cross validation usage to get best hyper parameter\n",
    "xgb_cv <- xgb.cv(data=dtrain,\n",
    "                 params = param,\n",
    "                 nrounds = 2500,\n",
    "                 prediction = T,\n",
    "                 maximize = F,\n",
    "                 folds = foldsCV,\n",
    "                 )\n",
    "\n",
    "#to get the best results\n",
    "print(xgb_cv$evaluation_log[which.min(xgb_cv$evaluation_log$train_mae_mean)])\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:Anaconda3]",
   "language": "python",
   "name": "conda-env-Anaconda3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}