R-lang-lec_notes.txt

Start: Sun Sep 6, 2015

########################## R Lectures by Stuart Greenlee

require(MASS)

########### 5 - Pkgs
getwd()
setwd()

########### 6 - Assigning Vars
rm(var)
ls()
rm(list = ls() )

########### 7 - Basic data types
is.string, numeric, boolean
as.string, numeric, boolean

help or ?, ??

typeof(var) 

########### 8 - Workspace Ops
example

data()

cars93

datasets

########### 9 - How to access files on disk

Getting access to your files

########### 10 - Basic ops

*, /, ^ = **

rounding()

floor(), ceiling()

cos(), sin(), tan()

abs()

log(), log10(), exp()

factorial()

************ 11 - Assigning vecs

--- Declare vec

vec = c(#,#,#) etc.  -- c stands for combine

vec = 1:5 or 4:8  etc.

or 1.5:5.5, iterates by 1

--- Objs in vec

vec[1] is obj #1

vec[1:4] is obj 1 through 4

vec[-1]  all but obj 1
vec[-3]  all but obj 3
vec[-(1:3)] all but objs 1 to 3

--- Combining vecs together

vec = c(vec1,vec2, scalar, 1:5, -4:-3) --> all work

length(vec) returns the size

--- String vec, boolean vec

char_vec = c(string1, string2...)

boovec = c(T,T,F...)

--- Combining vecs of dif types

They combine into a vec of strings

--- Ops on vecs

vec*2 works

vec+10 adds 10 to each elt

vec^2 squares each obj

can do ops on vecs, *if* they are the same size


************ 12 - Sequences

seq(from = #, to = #, by = #), and the nums can be decimals and negative

can do boolean comparisons too, e.g. seq(5) == 1:5, and same with vectors

all( seq(5) == 1:5 ) --> rets true if all are true, only

rep(1,5) --> repeats 1 exactly 5 times

rep (c(1:3) , 5 ) --> will repeat the vec 1,2,3 now 5 times


************** 13 - Basic stat'l functs

sum(vec) --> sums it up

min,max,range  --> as expected, and can combine vecs in the args, too

range also has elts 1 and 2, give the min and max themselves

mean, median -> the avg

mad ---> mean something..

var = variance

sd = standev

Ok, now to e.g. test if 68% of a vec falls within one standev, you can do this:

lowerbound = mean(vec) - sd(vec)
upperbound = mean(vec) + sd(vec)

boolvec that says if an elt is within one standev:

boolvec = (vec > lowerbound)  & (vec < upperbound)

sum(boolvec) --> only sums the true vals of the boolvec, and should give ~68% for any dist

cor(vec1, vec2) --> corrln

cov --> covariance

table --> shows freq of vals inside of a vec



************** 14 - Matrices

assign as so: matrix(c(3,5,234,1),nrow=2,ncol=4)

t(mat) = transpose

dim(mat) = dimns of any mat

t(vec) = transpose of a vec

amat[r,c] is the r'th element of the row, c'th elt of the column

amat[1,] is all stuff in row 1
amat[,1] is all stuff in col 1

You can assign just nrows or ncols and it will divide # of elts by the
one you give, and then assign the other the result of this division.  Interesting.

newmat = amat(c(2,4),] --> this will take the 2nd and 4th rows of amat, and put them into newmat

diag(4) = diagonal mat of 1's along main diag

You can do the normal mult, divide, sqrt etc. functs as normal

Can do ops on 2 mats if they have same dimns

dim(amat)[1] --> is num rows, same as nrow funct
dim(amat)[2] --> is num cols, same as ncol funct

Doing amat == bmat will return a boolean in each location, doing
all(amat==bmat) will return a single bool of true for if each elt
matches.

---- Extra from web:  http://stackoverflow.com/questions/11995832/inverse-of-matrix-in-r
 http://statmethods.net/advstats/matrix.html

 - m1 %*% m2 is the matrix mult of these 2 mats

 - i = solve(m1) gives somehow the inverse of matrix m1

************** 15 - Matrix Ops

---> My first cxample script: see mat_ops_Exx_CityTemps.R

paste funct concats strings together

cbind, rbind -- adding cols or rows to a mat


c(matrix) will uncurl the mat into a linear array of just the elts,
then you can do length on it and check vs. dimns etc.



************** 16 - Basic Matrix Stats

min,max,range,sd
rowMeans
colMeans
apply(mat, 1 or 2, FUN = sd)
cor(mat) 
summary  ---- gives you basic stats on any mat or dataset
summary(t(mat))

************** 17 - Random nums

random uniform: runif(numRandoms, startval, endval) 
set.seed(int) -- to replicate same sequence

sample( set, numvals, replace=T or F) # set can be e.g. 1:10 or a mat
or a vec you've created, and numvals is how many vals to pick from
that set, replace says if i can draw the same value again

sum(sample ...) will sum it up

state.name # All the US states

s = sample(1:5,1000,replace = T,prob = c(0.1,0.2,0.3,0.1,0.0001)) ; table(s) # The prob is how often to draw that val, and the sum does *not* have to be 1, it renorms all probs to get the right vals

rnorm(numvals, mean, sd)  # Default: mean = 0, sd of 1 -- Gaussian dist

************** 18 - String functs

substr(str, start, end) # str can be a vec of strings too

paste(str1, str2,...,strN, sep = '') # pastes together -- sep defaults
to one blank space

# Can do paste to vec of strings too

grep(searchstr, strvec, value = T or F) # Will return which locations
of the strvec the searchstr occurs in; value is by default F, but if T
it will return the strings at those locations

strsplit(str, substr to split on)

sub( substr to search and replace for, substr to sub in, the orig str)  # First occurrence subst

gsub( substr to search and replace for, substr to sub in, the orig str) # Global subst

************** 19 - Times and dates

Sys.Date()
Sys.time()

seq(as.POSIXct("2015-08-15"), as.POSIXct("2015-08-20"), by = 'hour')

difftime


************** 20 - line plots

xg = rnorm(1000)
xu = runif(1000)
csum = cumsum(xg)

plot(xg)

plot(xu,type='CHAR', col = 'COLORNAME', ylim = c(MINVAL, MAXVAL)) # We
have l=line, p = point (default), h = histo, o = lines and points, s =
steps, COLORNAME = blue, red, green, cyan -- type must be in quotes

lines( newplotinfo ) will just add to the plot already made

# Can use cbind on vecs too, useful to get the limits, and rowMeans
  will work on this obj too

points( otherinfo ) 



************** 21 - Plotting args

lines( newplotinfo , ylim = range(data), lty = NUM, lwd = LINEWIDTH,
main = TITLE, xlab = X-AXIS LABEL, ylab = Y-AXIS LABEL ) # NUM can be
1, solid, 2, 3 etc. ; LINEWIDTH = 1 by default, can make smaller or
larger



************** 22 - Bar graphs and histos

barplot
hist
rchi
as.matrix


************** 23 - Scatterplots

head(DATASET) - gives the info in the first few lines
attach(DATASET) - tells R that this is the dataset we're about to use
plot(x=DATA1, y=DATA2)
abline(lm(DATA1~DATA2) ) # Does the regression fit, and plots it
abline(h = horiz maxline, v = vert maxline)
pairs(~Weight+Price+~MPG.city) # Will do scatter plots of all 3 of these vs. each other



dotchart # Can put labels on etc. -- this one is a bit unclear

orderCars = Cars93[order(MPG.city), ] # Ordergs by MPG, but by nothing elxe

cex = how large to make the labels


************** 24 - Prob plots

# These are not well-explained, but one gets the rough idea of how well 2 dists match
qqnorm
qqline 
qqplot
scaled
rchisq

************** 25 - Combining and saving plots

par(mfrow=c(2,2)) # Makes plots in 2x2 format

plot(x=DATA1 , y = DATA2)


R-ploggers website talks about par and margin function

save plots as pdf, or png etc.

layout(matrix(c(1,2,1,3), ncol = 2, nrow = 2, byrow=T))


************** 26 - Arrays

x = 1:9
 ax=array(x)
 attributes(ax)
$dim

 attributes(ax)$dim
 ax=array(x,dim=c(3,3))
 attributes(ax)$dim

ax=array(x,dim=c(3,2,2)) # This is 2 3x2 arrays

ax[1:2, 2, ] = 42 # Assign 42 into both rows 1 and 2, second col, both arrays

# Arrays can of course be any number of dimns


************** 27 - Lists

# Declare three vecs of dif types, lengths etc.
x = list(nums,strs, bools)

x[[2]][1] gives the first elt of vec 2 in the list etc.

x = list("numname"=nums,"strname"=strs, "bool-list"=bools)

Can get the vec by:

x$numname

attach(x) --> then you can use just "numname" vs. x$numname

numname will give that

detach(x) # Opposite of attach

unlist(x) # Creates vec of strings of each obj -- collapses it.  Will make strs by default if there are any other types of objs in the list.


newx = c(x, "sysdate" = as.character(Sys.Date() ) ) # Add another vec to the list



************** 28 - Data Frames

cities = c("city1", "city2", "city3")

pop = c(1000,2000,4000)

avghigh = c(55,65,75)
contiguousUS = c(T,T,F)

citydata = data.frame(cities,pop, avghigh,contiguousUS)


- Doing:

attributes(citydata)

  will give row and column names


Dataframes have stringsAsFactors defaulting to true, can set it to F if needed

- Factors are some kind of lists too, i think (?)

-  To get a basic overview of what's in the data:
  
str(citydata) will give what the types of the data are in citydata

summary(citydata) gives dif info, with avgs etc.


require(MASS)

class(Cars93)  -- tells us data type

str(Cars93) - gives us all the info on the various parts of the data

summary(Cars93$Cylinders) -- will give the freq table of the cars with that many cylinders

--- make a:
vec,
mat,
dataframe

-- Functs that convert between data types:

-as.char, as.numeric, as.logical -- from before

- also: as.list, as.data.frame

Can then unlist things too.



************** 29 - Reading in Data 

I made a file on disk called testfile.csv with just this:

trial, mass, velocity
a, 10, 13
b, 53, 52
a, 51, 88.2

-- readcsv funct:

t = read.csv('testfile.csv') -- will read it in and assume first line is header for column names by default

t = read.csv('testfile.csv', header=F) -- will force it not to do this, and call cols V1, V2 etc.

class(t$trial) - will give a factor
class(t$mass) - will give an integer
class(t$velocity) - will give a numeric, meaning float

-- If i instead read in the file as so:

t = read.csv('testfile.csv', stringsAsFactors = F)

  then

class(t$trial) - will give a char

-- Doing:

t = read.table('testfile.csv')  will read it in as char strs -- note this needs space between the dif elts!  Won't work with just commas.  Note that header defaults to F for this funct!!!

t = read.table('testfile.csv', header = T, sep = ',') will make it use the commas as sep vars, so it's flexible


************** 30 - Missing data elts

-- Made the weights.txt file:

wt.ht.gender
100.71.m
.66.f
220.68.
160.74.f
199..m

-- And read it in with:

w = read.table('weights.txt', header = T, sep = '.')

-- Then the output looked like:

   wt ht gender
1 100 71      m
2  NA 66      f
3 220 68       
4 160 74      f
5 199 NA      m

So missing numerics were filled in with NA, and strings with a space.

-- Now use the is.na funct on this:

is.na(w)
        wt    ht gender
[1,] FALSE FALSE  FALSE
[2,]  TRUE FALSE  FALSE
[3,] FALSE FALSE  FALSE
[4,] FALSE FALSE  FALSE
[5,] FALSE  TRUE  FALSE

Note the str isn't F because a space is a real str

Can also test by e.g. is.na(w[1,1])

And can assign NA in by w[1,2] = NA -- has to be capital

-- If you try e.g.

attach(w)

  then
  
mean(ht) -- you'll get an NA because there are NA's in there

  but you can do

mean(na.omit(ht)) -- and you'll get an answer

  doing

na.omit(ht)  -- will show you which vals it's using

  you can also use this syntax to do the same:
  
mean(ht, na.rm=T)

  Doing it on the full dataset:
  
na.omit(w) -- will cut out any row that has an NA anywhere

  Doing:

complete.cases(w)  -- will give a bool with T only for the rows that have no NA's

  If we want to make the dataset that has only full rows, do:
  
completedata = w[complete.cases(w),] -- note the end comma is needed because we have to specify we need the full row for all those cases

  Doing

any.(is.na(dataset))

  Will tell you if there is anything missing in any line of the dataset.
  
************** 31 - Missing data elts #2

This has some more advanced methods of filling in missing numbers, but
apparently i can't load the xts pkg via:

  require(xts)

so i can't do it.  If i could, i could use:

  na.locf

which fills with a const line from the last val.  Or i could use a more advanced:

   fill function

which goes from the last value to the next value by a sloped line.

There are also 2 functs:

  is.nan

  is.infinite

which can be useful for checking vals.


************** 32 - Ordering data

First load up some data:

  data("petrol")

Get info on what the cols are:

  ?petrol
  
Change col names to something more useful:

  names(petrol)=c('id','specgraph', 'vaporpress','volcrude', 'volgas','percentcrude')

Order the rows by one col:

  order(vaporpress)

Order the whole dataset by this way:

  newpetroldat = petrol[order(vaporpress),] -- note comma at end for saying order by row

Can also order by 2 or more cols, if the first index is the same, and can do it in increasing or decreasing order depending on flag:
  
  petrol[order(vaporpress,volgas, decreasing = T),]


If you want to reorder cols alphabetically:

  newdat = petrol[,order(names(petrol))]


************** 33 - Making subsets of data

- Take a dataset like cars = Cars93

 - Then:

   mt = cars[,c("Model","Type"]  gives just those 2 cols, and a new dataset assigned to that

Also:

  mt = cars[,c(2,5)]  will do the same because i know the col #'s

-----------> This lec is incomplete, audio stops 1/3 of the way through



   
************** 35 - Merge and match functions

Define some city as so:

cityinfo =data.frame(
   city=I(c('Chitown', 'Seatac')),
   state=I(c('IL','WA'))
   )


then some weather info:


wthrinfo = data.frame(
   city=I(c('Chitown', 'Seatac', 'SF')),
   avghi = c(90,60,50)                    ---------- Note the c is req'd here to make the vec
   )


the "I" means don't make this a factor, just a str, and you can use single quotes as above, or double.


Then use:

  indexes = match(cityinfo$city,wthrinfo$city)

to get a vector of the indices that match those cities, like [1,2]

Now if you do

  cityinfo$city[indexes]

It will show those cities that match, and similarly for:

   wthrinfo$city[indexes]

BUT if you do:

   wthrinfo[indexes]

It'll show all row -- i don't fully get this.

You can also use this syntax:

  cityinfo$city %in%    wthrinfo$city

To get a boolean vec to see which rows of the first are in the second -- it'll have the exact number of rows of the first arg, then whether that row exists in the second

Now doing a merge:

  mergeddat = merge(cityinfo,wthrinfo,by.x="city",by.y="city")

Will check see which of the col names of the x dataset (the first) and
the y (second) match, and then only keep those in the resulting
mergeddat dataset.

Because there are matching column names in this case you can also use just do:

    mergeddat = merge(cityinfo,wthrinfo)

And it will give you the same result.

If you try to do a merge on two datasets with no common names, it will
make it nrow1 * nrow2 long, since it won't know how to match on any
specific column.

Some indexing stuff, if wthrinfo is a 4x2 matrix:

  wthrinfo[,0:2]  --- gives all rows, and cols 0 to 2
  wthrinfo[,0:2]  --- gives rows 0 to 2, and all cols 

Adding a row:

  wthrinfo = rbind(c("Reno",100),wthrinfo)   --- Note you need the c() notation, otw it messes up
 
Will add a row with the new info to the table.


************** 36 -  Descriptive Stats

Use the built-in iris dataset, and use the

  summary -- basic states
  is.na   -- anything missing?
  head    -- first few lines
  tail    -- last few lines
  names -- col names
  dim   -- dimns
  
Functs to describe the basic dataset.

Now, use:

  isplit = split(iris,iris$Species)

Will give a list of 3 components based on the 3 species, and you can get each via:

  isplit$versicolor   etc. --- you can tab-complete after $ sign to see which ones
  
   
Now plot some stuff up:

  plot(iris$Petal.Length,iris$Petal.Width,col=ifelse(iris$Species=="setosa",'blue',ifelse(iris$Species=="versicolor",'green','red')))

Plots the 3 dif iris species separately in dif colors, depending on
their type.  Note i've mixed up single and double quotes there in that
ifelse, and it's all ok.

Now use corrln function.

  cor(iris[,0:4])

Can only look at corrs between num'l columns.

To do the 't-test':

  t.test(iris$Sepal.Length,iris$Petal.Length)

If t is high, and p is low, it means the avgs are well separated --
they don't see much else.


************** 37 - Using the apply funct

Using apply funct:

  apply(data, num, FUN = max) --- num = 1 for rows, 2 for cols.

FUN can also be: min, range, sd.

You can also use an apply funct across lists:

  lapply(isplit,summary)

And there is sapply, and other apply functs -- do this:

   ??base::apply

To get the several dif types of apply one can use.


************** 38 - Linear models

Some stuff with the cars dataset:

 attach(Cars93)

 lm(formula = Weight ~ MPG.city + EngineSize)

Then with the irises one:

Have to first set up the plot:

  plot(Petal.Length,Petal.Width)

Then fit it, and note how the vars are inverted from above:

  abline(lm(Petal.Width~Petal.Length))

Can get stats on the fit, which aren't totally obvious:

  summary(lm(Petal.Length~Petal.Width+0))

************** 39 - Doing some real linear regression (Extracting LinMod info)


summary(lm(Petal.Length~Petal.Width+0))

 plot(resid(fitmod))

Fit one thing vs. another:

    fitmod = lm(Weight~MPG.city)
    
Fit several things:

  fitmod = lm(Weight~MPG.city+EngineSize+Passengers)


Plot the fitted val of a var VS the orig value in the dataset:

  plot(Weight,fitmod$fitted.values,type='p',col='green')

Plot an x=y line:

  abline(c(1,1))

Syntax here is not obvious to me, as variations don't do as expected..

Now split dataset into a training and fitting one, starting as so:

  trainingsetIndices=sample(1:nrow(Cars93),size=round(nrow(Cars93)*0.50))

This will *randomly* pick 50% of the lines, not the first half.

Now define the training data set from these indices, then the one we want to fit on by the inverse of it:

  trainingCars =Cars93[trainingsetIndices,]
  fitcars = Cars93[-trainingsetIndices,]

Now make the fit on the data set:

  fitweights = predict(fitmod,newdata = fitcars)

Pull out the true orig wts using the indices:

  trueweights = Cars93$Weight[-trainingsetIndices]

Take the dif of the two:

  wterrs = fitweights - trueweights

Can show this as a funct of index:

  plot(wterrs)

Can then get the RMSE of the training set:

   rmseTrainingSet = sqrt(mean(fitmod$residuals^2))

And of the fitted set:

  rmseFitted = sqrt(mean(wterrs^2))

And as expected, the former is smaller than the latter:

 rmseTrainingSet
   175.7989
 rmseFitted
   283.0032

Phew!



************** 40 - PCA

prcomp
biplot

************** 41 - XTS = Extensible Time Series

Installed: xts

require(xts)

lag
x = xts(rnorm(20),order.by = Sys.Date()-20:1)

Timezones

diff

.... Really weird and unclear what they're even trying to do.

************** 42 - XTS and ACF objects


------------------ Another failed file, stops playing about a qrtr way through

************** 43 - More about time

Johnson and Johnson 1960 earnings

stl funct

s.window

------------------ Another failed file

************** 44 ------------------ Another failed file

************** 45,46,47 - not there


************** 48 - For loops ----> looks good

************** 49 - While loops ----> looks good

************** 50 - Appending loops ----> looks good

************** 51 - not there


************** 52 - Debug funct ----> looks good

************** 53 - Recursive funct ----> looks good

************** 54 - Saving dif file types ----> looks good

************** 55 - Additional resources ----> looks good


----------- Other useful cmds

---- Strings

nchar(str) will give you the number of chars in the string


hi Dad- i just had a mozzarella, basil, tomato hot sandwich here at Kaffa Cafe (old Paddy's), very delicious!  And a Oreo frappucino, too.  love,M