August 4, 2017August 6, 2017John MountOpinion, Programming, Statisticsdplyr, part-time R user, R, rlang, seplyr, wrapr9 Comments

Let’s Have Some Sympathy For The Part-time R User

When I started writing about methods for better "parametric programming" interfaces for dplyr for R dplyr users in December of 2016 I encountered three divisions in the audience:

dplyr users who had such a need, and wanted such extensions.
dplyr users who did not have such a need ("we always know the column names").
dplyr users who found the then-current fairly complex "underscore" and lazyeval system sufficient for the task.

Needing name substitution is a problem an advanced full-time R user can solve on their own. However a part-time R would greatly benefit from a simple, reliable, readable, documented, and comprehensible packaged solution. Continue reading Let’s Have Some Sympathy For The Part-time R User

July 22, 2017July 22, 2017John MountCoding, data science, Opinion, Programming, Statistics, Tutorialsdplyr, non-standard evaluation, R, seplyr, standard evaluation13 Comments

Tutorial: Using seplyr to Program Over dplyr

seplyr is an R package that makes it easy to program over dplyr 0.7.*.

To illustrate this we will work an example.

Continue reading Tutorial: Using seplyr to Program Over dplyr

July 19, 2017July 19, 2017John MountAdministrativia, Exciting Techniques, Statistics, Tutorialsdplyr, R, seplyr1 Comment

seplyr update

The development version of my new R package seplyr is performing in practical applications with dplyr 0.7.* much better than even I (the seplyr package author) expected.

I think I have hit a very good set of trade-offs, and I have now spent significant time creating documentation and examples.

I wish there had been such a package weeks ago, and that I had started using this approach in my own client work at that time. If you are already a dplyr user I strongly suggest trying seplyr in your own analysis projects.

Please see here for details.

July 15, 2017July 17, 2017John Mountdata science, Opinion, Programming, Statistics, Tutorialsdplyr, R, rlang, seplyr, tidy eval12 Comments

dplyr 0.7 Made Simpler

I have been writing a lot (too much) on the R topics dplyr/rlang/tidyeval lately. The reason is: major changes were recently announced. If you are going to use dplyr well and correctly going forward you may need to understand some of the new issues (if you don’t use dplyr you can safely skip all of this). I am trying to work out (publicly) how to best incorporate the new methods into:

real world analyses,
reusable packages,
and teaching materials.

I think some of the apparent discomfort on my part comes from my feeling that dplyr never really gave standard evaluation (SE) a fair chance. In my opinion: dplyr is based strongly on non-standard evaluation (NSE, originally through lazyeval and now through rlang/tidyeval) more by the taste and choice than by actual analyst benefit or need. dplyr isn’t my package, so it isn’t my choice to make; but I can still have an informed opinion, which I will discuss below.

Continue reading dplyr 0.7 Made Simpler

July 12, 2017July 15, 2017John Mountdata science, Statistics, Tutorialsdplyr, R, rlang, tidy eval10 Comments

Better Grouped Summaries in dplyr

For R dplyr users one of the promises of the new rlang/tidyeval system is an improved ability to program over dplyr itself. In particular to add new verbs that encapsulate previously compound steps into better self-documenting atomic steps.

Let’s take a look at this capability.

Continue reading Better Grouped Summaries in dplyr

July 6, 2017July 8, 2017John Mountdata science, Opinion, Statisticsdplyr, R, R and big data, replyr, spark, sparklyr2 Comments

Working With R and Big Data: Use Replyr

In our latest R and Big Data article we discuss replyr.

Why `replyr`

replyr stands for REmote PLYing of big data for R.

Why should R users try replyr? Because it lets you take a number of common working patterns and apply them to remote data (such as databases or Spark).

replyr allows users to work with Spark or database data similar to how they work with local data.frames. Some key capability gaps remedied by replyr include:

Summarizing data: replyr_summary().
Combining tables: replyr_union_all().
Binding tables by row: replyr_bind_rows().
Using the split/apply/combine pattern (dplyr::do()): replyr_split(), replyr::gapply().
Pivot/anti-pivot (gather/spread): replyr_moveValuesToRows()/ replyr_moveValuesToColumns().
Handle tracking.
A join controller.

You may have already learned to decompose your local data processing into steps including the above, so retaining such capabilities makes working with Spark and sparklyr much easier. Some of the above capabilities will likely come to the tidyverse, but the above implementations are build purely on top of dplyr and are the ones already being vetted and debugged at production scale (I think these will be ironed out and reliable sooner).

Continue reading Working With R and Big Data: Use Replyr

June 28, 2017July 1, 2017John MountOpinion, Programming, Statisticsdplyr, R, rlang, tidy eval, wrapr, wrapr for sweet R2 Comments

Using wrapr::let() with tidyeval

While going over some of the discussion related to my last post I came up with a really neat way to use wrapr::let() and rlang/tidyeval together.

Please read on to see the situation and example. Continue reading Using wrapr::let() with tidyeval

June 16, 2017July 22, 2017John MountCoding, data science, Opinion, Programming, Statistics, Tutorialsdplyr, function composition, functional programming, non-standard evaluation, NSE, R, R as it is, referential transparency, replyr, SE, wrapr10 Comments

Non-Standard Evaluation and Function Composition in R

In this article we will discuss composing standard-evaluation interfaces (SE: parametric, referentially transparent, or “looks only at values”) and composing non-standard-evaluation interfaces (NSE) in R.

In R the package tidyeval/rlang is a tool for building domain specific languages intended to allow easier composition of NSE interfaces.

To use it you must know some of its structure and notation. Here are some details paraphrased from the major tidyeval/rlang client, the package dplyr: vignette('programming', package = 'dplyr')).

":=" is needed to make left-hand-side re-mapping possible (adding yet another "more than one assignment type operator running around" notation issue).
"!!" substitution requires parenthesis to safely bind (so the notation is actually "(!! )", not "!!").
Left-hand-sides of expressions are names or strings, while right-hand-sides are quosures/expressions.

Continue reading Non-Standard Evaluation and Function Composition in R

May 6, 2017June 2, 2017John MountOpinion, Statistics, Tutorialsdplyr, pivot tables, R, tidyr2 Comments

dplyr in Context

Introduction

Beginning R users often come to the false impression that the popular packages dplyr and tidyr are both all of R and sui generis inventions (in that they might be unprecedented and there might no other reasonable way to get the same effects in R). These packages and their conventions are high-value, but they are results of evolution and implement a style of programming that has been available in R for some time. They evolved in a context, and did not burst on the scene fully armored with spear in hand.

Zeus giving birth to Athena, Rudolph Tegner

March 25, 2017John MountOpinion, Statistics, Tutorialsbizarro, Bizarro glyph, bizarro pipe, dot debugging, dplyr, extra eager, lazy eval, R

Debugging Pipelines in R with Bizarro Pipe and Eager Assignment

This is a note on debugging magrittr pipelines in R using Bizarro Pipe and eager assignment.

Why replyr

Introduction

Why `replyr`