layout:true <p class="footer"> <span xmlns:dct="http://purl.org/dc/terms/" property="dct:title">Algorithms</span> by <a xmlns:cc="http://creativecommons.org/ns#" href="http://www.datapolitan.com" property="cc:attributionName" rel="cc:attributionURL">Richard Dunks</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative-Commons-License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png" /></a> </p> --- class: center,middle  - - - #Algorithms: How I Learned to Make My Regressions Linear ##Richard Dunks, Instructor ###Follow along: http://bit.ly/algo2016-class5 --- #Do Now 5 + Query the training database you used in homework 4-1 and query the `noise_311` table + Perform a basic statistical analysis of the time a complaint is left open + Compare the results to what you found for DOT complaints + Submit this as Do Now 5 --- #Readings + "Brexiting Yourself In The Foot: Why Britain's Eurosceptic Regions Have Most To Lose From Eu Withdrawal" http://www.cer.org.uk/insights/brexiting-yourself-foot-why-britains-eurosceptic-regions-have-most-lose-eu-withdrawal (Extra credit if you can recreate their analysis) --- #Assignment 1 + Perform a basic statistical analysis of the time DOT 311 (table is called `dot_311`) complaints are open (subtract closed date from created date) + Connect to the database to get the data and do the analysis. Submit the code through Github and type up your results in your PR --- #Assignment 2 + Using the `2013_NYC_CD_MedianIncome_Recycle.xlsx` file, calculate the correlation between the recycling rate and the median income. Discuss your findings in your PR. --- #Assignment 3 + Using the `heights_weights_genders.csv`, analyze the difference between the height weight correlation in women and men. --- #Goals for today + Introduce simple and multiple linear regression + Discuss the application of regression to problems + Practice finding the regression weights in Python --- class:center, middle #First a quick review --- #Measures of Central Tendency -- + Quantitative data tends to cluster around some central value -- + Contrasts with the spread of data around that center (i.e. the variability in the data) -- + Mean is a more precise measure and more often used -- + Median is better when there are extreme outliers -- + Mode is used when the data is categorical (as opposed to numeric) --- #Measures of Variability -- + Describe the distribution of our data -- + Range (Maximum – Minimum) -- + Inter-quartile Range -- + Standard Deviation -- + Identification of outliers (1.5 x IQR) --- #Descriptive Statistics -- + Quantitatively describe the main features of a dataset -- + Help distinguish distributions and make them comparable --- #Exploratory Data Analysis -- + Goal -> Discover patterns in the data -- + Understand the context -- + Summarize fields -- + Use graphical representations of the data -- + Explore outliers --- #Coefficient of Correlation -- + Quantifies the amount of shared variability between variables -- + Ranges between -1 and +1 -- + Negative numbers are inversely proportional -- + Positive numbers are directly proportional -- + The closer to either -1 or +1, the greater the correlation --- #Coefficient of Correlation  http://pixshark.com/correlation-examples.htm --- class:center,middle #What we didn't talk about... --- #Cofficient of Determination -- + The percentage of variance in one variable shared with the other -- + More shared variability implies a stronger relationship -- + Calculate by squaring the correlation coefficient -- + Ex. the correlation of determination for our height and weight dataset is 0.77 --- #Linear Regression -- + Using the known relationship between continuous variables, we can predict unseen values -- + Assumes relationship is linear (but doesn’t need to be) --- #Linear Regression -- + Draw a line that minimizes the distance between each point -- + “Line of best fit” -> minimizes the sum of squared residuals [](http://nbviewer.jupyter.org/github/justmarkham/DAT4/blob/master/notebooks/08_linear_regression.ipynb) --- #Linear Regression -- + Characteristics of the line defines the relationship -- + Slope -> relationship between independent and dependent variable (how Y increases per unit of X) -- + Intercept -> expected mean value of Y at X=0 -- + Values along the line are the predicted values for any given value X --- #Linear Regression  [Source](http://lms.caribbeanoer.org/courses/23/modules/items/722) Open Algebra Study Guide by [John Redden](http://www.openalgebra.com/) is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-nc-sa/3.0/deed.en_US). --- #Linear Regression   --- #Linear Regression + The sum of squared residuals (or sum of squared errors) is the best measure of how good a regression line is -- + Ideally, we want the smallest SSR possible --- class:center,middle #And how does it do the fit? ---  Andrew Ng Machine Learning Class: http://cs229.stanford.edu/notes/cs229-notes1.pdf --- #For the rest of us -- + The fit function calculates the coefficients through a relatively simple calculation that approximates the values very closely -- + This makes the calculation relatively quick, even for larger datasets --- #Prediction -- + Using the relationship between variables, we can predict values based on the relationship -- + Can estimate the magnitude as well as the general trend -- + More data points, the better the prediction -- <br> <br>  --- #Prediction [](http://ci.columbia.edu/ci/premba_test/c0331/s7/s7_6.html) --- class:middle  --- class:center,middle #10 Min Break  Source: http://xkcd.com/552/ --- #Linear Regression -- + What we've done so far has been called simple linear regression -- + It involved one explanatory variable and one response variable -- + It's possible to have multiple explanatory variables --- #Multiple Regression -- + Instead of one feature we have several features -- + Each feature has their own coefficient --  --- #Multiple Regression [](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/R/R5_Correlation-Regression/R5_Correlation-Regression4.html) Licensed under [Creative Commons Attribution-NonCommercial 3.0 Unported](http://creativecommons.org/licenses/by-nc/3.0) --- class:center,middle #Multiple Regression Example ##Open `Multiple_Linear_Regression.ipynb` --- #Overfitting/Underfitting [](http://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html) [Source](http://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html). © 2010 - 2014, scikit-learn developers ([BSD License](https://opensource.org/licenses/BSD-2-Clause)) --- #Overfitting Example  --- #Overfitting Example  --- class:center,middle  --- #Data Modeling -- + We're writing a formula to approximate reality -- + The formula can be complex or simple depending on how much error we're willing to tolerate -- + Reality is rarely so simple -- + But you'd be surprised how far this can get us ---  -- ###Obviously... --- class:center,middle #Any questions?  --- #Resources + [Statistical Sampling and Regression: Simple Linear Regression](http://ci.columbia.edu/ci/premba_test/c0331/s7/s7_6.html) + [Introduction to Linear Regression](http://nbviewer.jupyter.org/github/justmarkham/DAT4/blob/master/notebooks/08_linear_regression.ipynb) + [An Introduction to Statistical Learning with R](http://www-bcf.usc.edu/~gareth/ISL/) --- #Reading/Watching + "[Not Even Scientists Can Easily Explain P-values](http://fivethirtyeight.com/features/not-even-scientists-can-easily-explain-p-values/)" + "[Last Week Tonight with John Oliver: Scientific Studies](https://youtu.be/0Rnq1NpHdmw)" --- #Assignment 1 + Use the data from `heights_weights_genders.csv` to create a simple predictor that takes in a person's height and guesses their weight based on a model using all the data, regardless of gender. To do this, find the parameters (`lm.params`) and use those in your function (i.e. don't generate a model each time) --- #Assignment 2 + Create two models for the relationship between height and weight based on gender + Modify the code in Assignment 1 to ask for a person's gender as well as their height to produce an estimate of a person's weight using the models you created + Find the weights and use those in your function (i.e. don't generate a model each time) --- #Assignment 3 + Using the data from the `2013_NYC_CD_MedianIncome_Recycle.xlsx` file, create a predictor using the weights from the model. This time, use the built in attributes in your model rather than hard-coding them into your algorithm --- #Assignment 4 + Using data from [this FiveThirtyEight post](http://53eig.ht/1e2aV6U), write code to calculate the correlation of the responses from the poll. + Respond to the story in your PR. Is this a good example of data journalism? Why or why not? --- class:center,middle #Thank You!