R as it is – Win-Vector Blog

September 22, 2017September 23, 2017John MountOpinion, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, TutorialsBig Data, enterprise, part-time R user, R, R and big data, R as it is2 Comments

My advice on dplyr::mutate()

There are substantial differences between ad-hoc analyses (be they: machine learning research, data science contests, or other demonstrations) and production worthy systems. Roughly: ad-hoc analyses have to be correct only at the moment they are run (and often once they are correct, that is the last time they are run; obviously the idea of reproducible research is an attempt to raise this standard). Production systems have to be durable: they have to remain correct as models, data, packages, users, and environments change over time.

Demonstration systems need merely glow in bright light among friends; production systems must be correct, even alone in the dark.

“Character is what you are in the dark.”

John Whorfin quoting Dwight L. Moody.

I have found: to deliver production worthy data science and predictive analytic systems, one has to develop per-team and per-project field tested recommendations and best practices. This is necessary even when, or especially when, these procedures differ from official doctrine.

What I want to do is share a single small piece of Win-Vector LLC‘s current guidance on using the R package dplyr. Continue reading My advice on dplyr::mutate()

September 3, 2017September 4, 2017John MountOpinion, Programming, Statisticsdplyr, R, R and big data, R as it is, replyr1 Comment

It is Needlessly Difficult to Count Rows Using dplyr

Question: how hard is it to count rows using the R package dplyr?
Answer: surprisingly difficult.

When trying to count rows using dplyr or dplyr controlled data-structures (remote tbls such as Sparklyr or dbplyr structures) one is sailing between Scylla and Charybdis. The task being to avoid dplyr corner-cases and irregularities (a few of which I attempt to document in this "dplyr inferno").

August 19, 2017August 19, 2017John MountOpinion, Programming, Rants, Statisticspart-time R user, R, R as it is2 Comments

Is dplyr Easily Comprehensible?

dplyr is one of the most popular R packages. It is powerful and important. But is it in fact easily comprehensible? Continue reading Is dplyr Easily Comprehensible?

July 10, 2017July 10, 2017John MountProgramming, Statisticsmagrittr, pipe, R, R as it is, tidyverse4 Comments

What is magrittr’s future in the tidyverse?

For many R users the magrittr pipe is a popular way to arrange computation and famously part of the tidyverse.

The tidyverse itself is a rapidly evolving centrally controlled package collection. The tidyverse authors publicly appear to be interested in re-basing the tidyverse in terms of their new rlang/tidyeval package. So it is natural to wonder: what is the future of magrittr (a pre-rlang/tidyeval package) in the tidyverse? Continue reading What is magrittr’s future in the tidyverse?

June 26, 2017John MountOpinion, Programming, StatisticsR, R as it is, wrapr, wrapr for sweet R5 Comments

Please Consider Using wrapr::let() for Replacement Tasks

From dplyr issue 2916.

The following appears to work.

suppressPackageStartupMessages(library("dplyr"))

COL <- "homeworld"
starwars %>%
  group_by(.data[[COL]]) %>%
  head(n=1)

## # A tibble: 1 x 14
## # Groups:   COL [1]
##             name height  mass hair_color skin_color eye_color birth_year
##            <chr>  <int> <dbl>      <chr>      <chr>     <chr>      <dbl>
## 1 Luke Skywalker    172    77      blond       fair      blue         19
## # ... with 7 more variables: gender <chr>, homeworld <chr>, species <chr>,
## #   films <list>, vehicles <list>, starships <list>, COL <chr>

Though notice it reports the grouping is by "COL", not by "homeworld". Also the data set now has 14 columns, not the original 13 from the starwars data set.

Continue reading Please Consider Using wrapr::let() for Replacement Tasks

June 16, 2017July 22, 2017John MountCoding, data science, Opinion, Programming, Statistics, Tutorialsdplyr, function composition, functional programming, non-standard evaluation, NSE, R, R as it is, referential transparency, replyr, SE, wrapr10 Comments

Non-Standard Evaluation and Function Composition in R

In this article we will discuss composing standard-evaluation interfaces (SE: parametric, referentially transparent, or “looks only at values”) and composing non-standard-evaluation interfaces (NSE) in R.

In R the package tidyeval/rlang is a tool for building domain specific languages intended to allow easier composition of NSE interfaces.

To use it you must know some of its structure and notation. Here are some details paraphrased from the major tidyeval/rlang client, the package dplyr: vignette('programming', package = 'dplyr')).

":=" is needed to make left-hand-side re-mapping possible (adding yet another "more than one assignment type operator running around" notation issue).
"!!" substitution requires parenthesis to safely bind (so the notation is actually "(!! )", not "!!").
Left-hand-sides of expressions are names or strings, while right-hand-sides are quosures/expressions.

Continue reading Non-Standard Evaluation and Function Composition in R

June 15, 2017John MountOpinion, Rants, Statistics, Tutorialslinear regression, R, R as it is1 Comment

An easy way to accidentally inflate reported R-squared in linear regression models

Here is an absolutely horrible way to confuse yourself and get an inflated reported R-squared on a simple linear regression model in R.

We have written about this before, but we found a new twist on the problem (interactions with categorical variable encoding) which we would like to call out here. Continue reading An easy way to accidentally inflate reported R-squared in linear regression models

June 7, 2017John MountCoding, Opinion, Programming, Statisticsbase-R, R, R as it is2 Comments

More on safe substitution in R

Let’s worry a bit about substitution in R. Substitution is very powerful, which means it can be both used and mis-used. However, that does not mean every use is unsafe or a mistake.

Continue reading More on safe substitution in R

June 5, 2017June 7, 2017John MountOpinion, Programming, Statisticsbase-R, R, R and big data, R as it is, spark, sparklyr

There is usually more than one way in R

Python has a fairly famous design principle (from “PEP 20 — The Zen of Python”):

There should be one– and preferably only one –obvious way to do it.

Frankly in R (especially once you add many packages) there is usually more than one way. As an example we will talk about the common R functions: str(), head(), and the tibble package‘s glimpse(). Continue reading There is usually more than one way in R

June 4, 2017June 7, 2017John Mountdata science, Opinion, Statisticsbase-R, R, R as it is, summary

R summary() got better!

Here is a really nice feature found in the current 3.4.0 version of R: summary() has become a lot more reasonable.

summary(15555)

#    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#   15555   15555   15555   15555   15555   15555

Please read on for some background. Continue reading R summary() got better!