This note is a comment on some of the timings shared in the dplyr-0.8.0 pre-release announcement.
The original published timings were as follows:
With performance metrics: measurements are marketing. So let’s dig in the above a bit.
Tag: dplyr
A Better Example of the Confused By The Environment Issue
Our interference from then environment issue was a bit subtle. But there are variations that can be a bit more insidious.
Please consider the following.
Continue reading A Better Example of the Confused By The Environment Issue
A Subtle Flaw in Some Popular R NSE Interfaces
It is no great secret: I like value oriented interfaces that preserve referential transparency. It is the side of the public debate I take in R programming.
"One of the most useful properties of expressions is that called by Quine referential transparency. In essence this means that if we wish to find the value of an expression which contains a sub-expression, the only thing we need to know about the sub-expression is its value."
Please read on for discussion of a subtle bug shared by a few popular non-standard evaluation interfaces.
Continue reading A Subtle Flaw in Some Popular R NSE Interfaces
Timing Column Indexing in R
I’ve ended up (almost accidentally) collecting a number of different solutions to the “use a column to choose values from other columns in R” problem.
Please read on for a brief benchmark comparing these methods/solutions.
Using a Column as a Column Index
We recently saw a great recurring R question: “how do you use one column to choose a different value for each row?” That is: how do you use a column as an index? Please read on for some idiomatic base R, data.table, and dplyr solutions.
Parameterizing with bquote
One thing that is sure to get lost in my long note on macros in R is just how concise and powerful macros are. The problem is macros are concise, but they do a lot for you. So you get bogged down when you explain the joke.
Let’s try to be concise.
Timings of a Grouped Rank Filter Task
Introduction
This note shares an experiment comparing the performance of a number of data processing systems available in R. Our notional or example problem is finding the top ranking item per group (group defined by three string columns, and order defined by a single numeric column). This is a common and often needed task.
data.table is Really Good at Sorting
The data.table R package is really good at sorting. Below is a comparison of it versus dplyr for a range of problem sizes.
Collecting Expressions in R
Not a full R article, but a quick note demonstrating by example the advantage of being able to collect many expressions and pack them into a single extend_se() node.
Speed up your R Work
Introduction
In this note we will show how to speed up work in R by partitioning data and process-level parallelization. We will show the technique with three different R packages: rqdatatable, data.table, and dplyr. The methods shown will also work with base-R and other packages.
For each of the above packages we speed up work by using wrapr::execute_parallel which in turn uses wrapr::partition_tables to partition un-related data.frame rows and then distributes them to different processors to be executed. rqdatatable::ex_data_table_parallel conveniently bundles all of these steps together when working with rquery pipelines.
The partitioning is specified by the user preparing a grouping column that tells the system which sets of rows must be kept together in a correct calculation. We are going to try to demonstrate everything with simple code examples, and minimal discussion.