seplyr is an R package that makes it easy to program over dplyr 0.7.*.
To illustrate this we will work an example.
Continue reading Tutorial: Using seplyr to Program Over dplyr
Month: July 2017
seplyr update
The development version of my new R package seplyr is performing in practical applications with dplyr 0.7.* much better than even I (the seplyr package author) expected.
I think I have hit a very good set of trade-offs, and I have now spent significant time creating documentation and examples.
I wish there had been such a package weeks ago, and that I had started using this approach in my own client work at that time. If you are already a dplyr user I strongly suggest trying seplyr in your own analysis projects.
dplyr 0.7 Made Simpler
I have been writing a lot (too much) on the R topics dplyr/rlang/tidyeval lately. The reason is: major changes were recently announced. If you are going to use dplyr well and correctly going forward you may need to understand some of the new issues (if you don’t use dplyr you can safely skip all of this). I am trying to work out (publicly) how to best incorporate the new methods into:
- real world analyses,
- reusable packages,
- and teaching materials.
I think some of the apparent discomfort on my part comes from my feeling that dplyr never really gave standard evaluation (SE) a fair chance. In my opinion: dplyr is based strongly on non-standard evaluation (NSE, originally through lazyeval and now through rlang/tidyeval) more by the taste and choice than by actual analyst benefit or need. dplyr isn’t my package, so it isn’t my choice to make; but I can still have an informed opinion, which I will discuss below.
Better Grouped Summaries in dplyr
For R dplyr users one of the promises of the new rlang/tidyeval system is an improved ability to program over dplyr itself. In particular to add new verbs that encapsulate previously compound steps into better self-documenting atomic steps.
Let’s take a look at this capability.
What is magrittr’s future in the tidyverse?
For many R users the magrittr pipe is a popular way to arrange computation and famously part of the tidyverse.
The tidyverse itself is a rapidly evolving centrally controlled package collection. The tidyverse authors publicly appear to be interested in re-basing the tidyverse in terms of their new rlang/tidyeval package. So it is natural to wonder: what is the future of magrittr (a pre-rlang/tidyeval package) in the tidyverse? Continue reading What is magrittr’s future in the tidyverse?
In praise of syntactic sugar
There has been some talk of adding native pipe notation to R (for example here, here, and here). And even a tidyeval/rlang pipe here.
I think a critical aspect of such an extension would be to treat such a notation as syntactic sugar and not insist such a pipe match magrittr semantics, or worse yet give a platform for authors to insert their own preferred ad-hoc semantics. Continue reading In praise of syntactic sugar
Working With R and Big Data: Use Replyr
In our latest R and Big Data article we discuss replyr.
Why replyr
replyr stands for REmote PLYing of big data for R.
Why should R users try replyr? Because it lets you take a number of common working patterns and apply them to remote data (such as databases or Spark).
replyr allows users to work with Spark or database data similar to how they work with local data.frames. Some key capability gaps remedied by replyr include:
- Summarizing data:
replyr_summary(). - Combining tables:
replyr_union_all(). - Binding tables by row:
replyr_bind_rows(). - Using the split/apply/combine pattern (
dplyr::do()):replyr_split(),replyr::gapply(). - Pivot/anti-pivot (
gather/spread):replyr_moveValuesToRows()/replyr_moveValuesToColumns(). - Handle tracking.
- A join controller.
You may have already learned to decompose your local data processing into steps including the above, so retaining such capabilities makes working with Spark and sparklyr much easier. Some of the above capabilities will likely come to the tidyverse, but the above implementations are build purely on top of dplyr and are the ones already being vetted and debugged at production scale (I think these will be ironed out and reliable sooner).
Join Dependency Sorting
In our latest installment of “R and big data” let’s again discuss the task of left joining many tables from a data warehouse using R and a system called "a join controller" (last discussed here).
One of the great advantages to specifying complicated sequences of operations in data (rather than in code) is: it is often easier to transform and extend data. Explicit rich data beats vague convention and complicated code.