collaborative-filter.Rmd

---
title: "collaborative-filter"
author: "Charles Lang"
date: "1/31/2019"
output: html_document
---

In HUDK4051 there are six units, we will use your ratings of these units in terms of both interest and difficulty to produce individual suggestions about what unit to attempt next.

Start by uploading both the interest and difficulty csv files:
```{r}
interest <- read.csv("interest.csv")
diffuculty <- read.csv("difficulty.csv")
```

We will be using matrix operations in this assignment, so convert your data frames to matrices:
```{r, echo = FALSE}
#HINT: First you will need to remove the student ids as matrices can only contain one data type. You will then need to rename your row names with the student ids.
library(dplyr) 
I2 <- select(interest, 2:7)
I2 <- as.matrix(I2)
rownames(I2) <- interest$name
D2 <- select(difficulty, 2:7)
D2 <- as.matrix(D2)
rownames(D2) <- diffuculty$name
```

First, lets look at the interest data. We can generate a user-based similarity matrix based on cosine similarity using the ratings the class gave each unit. This matrix will represent the similarity of interests between students in the class.
```{r, echo = FALSE}
#First let's transpose the matrix so that multiplication occurs by students rather than units.

#Look at your data, missing values are coded with zero not NA. Investigate cosine similarity and decide whether your missing values should be coded with zeros or NAs. Explain your choice.

I2 <- t(I2)

#Then we can generate the cosine similarity values for each pair of students
#install.packages("lsa") #You will need to install the lsa package to access the cosine command.

library(lsa)

I.SIM <- cosine(I2) #This command generates the cosine similarity values as a new matrix. Click on I.SIM in the Global Environment pane to see what it looks like.

diag(I.SIM) <- NA #Since each student will be most similar to themselves we want to remove that information
```

Now, we can make a quick query to find out which students are most similar to you. 
```{r}
my.name <- "Ling Ai" #Input your name as it appears in the data set

head(rownames(I.SIM[order(I.SIM[my.name,], decreasing = TRUE),]), n = 2) #This code orders the column of the matrix corresponding to your UNI according to similarity and returns the top two UNI ids for the students who's interests are most similar to yours
```

This is a basic collaborative filter! You have used information about interest across the class to generate an individual suggestion. Email one of your top matches, you may find them to be a good person to work with or ask questions during the semester.

Now create a unit-based, rather than student-based similarity matrix for difficulty. Then use your similarity matrix to provide a suggested next unit to a student who is looking for the unit that is most similar in terms of difficulty to the "prediction" unit. 
```{r}
D.SIM <- cosine(D2)
diag(D.SIM) <- NA
head(rownames(D.SIM[order(D.SIM["pred.dif",], decreasing = TRUE),]), n = 1) 
```

Finally, educational settings have important differences to purely commercial settings such as film or product suggestions. In education we want people not to just follow their interests as they may simply choose things that are easy for them so they learn very little. To reduce this possibility with your collaborative filter create a composite measure from interest and difficulty, then construct a similarity matrix using this measure. (HINT: PCA). Once you have built the similarity matrix generate a suggestion for a student who has just completed the "prediction" unit.

```{r}
library(tidyr)
library(lsa)

I3 <- gather(interest,unit,interest, 2:7)
D3 <- gather(difficulty, name, difficulty)

C1 <- data.frame(I3$name, I3$unit, I3$interest, D3$difficulty)
names(C1) <- c("name", "unit", "interest", "difficulty")
C1 <- filter(C1, difficulty > 0)
C2 <- select(C1, "interest", "difficulty")

#Run PCA
pc <- prcomp(C2)
#Extract PC1 loadings as new measure and attach to stid & unit
C3 <- data.frame(C1$name, C1$unit, pc$x)
C4 <- select(C3, C1.name, C1.unit, PC1)
#Remove int from unit label
C4$C1.unit <- gsub(".int", "", C4$C1.unit)

#Recreate unit by student matrix
C5 <- spread(C4, C1.name, PC1)
row.names(C5) <- C5$C1.unit
C5$C1.unit <- NULL
C5 <- as.matrix(C5)
C5 <- ifelse(is.na(C5), 0, C5)
C5 <- t(C5)

#Generate cosine similarity matrix for units
C.SIM <- cosine(C5)
diag(C.SIM) <- NA

#Search for most similar unit to "neural"
head(rownames(C.SIM[order(C.SIM["neural",], decreasing = TRUE),]), n = 1)

```

Once you have completed your collaborative filter you can return to it each time you are choosing a new unit to complete.