The data.table R package is really good at sorting. Below is a comparison of it versus dplyr for a range of problem sizes.
Tag: R
Meta-packages, nails in CRAN’s coffin
Derek Jones recently discussed a possible future for the R ecosystem in “StatsModels: the first nail in R’s coffin”.
This got me thinking on the future of CRAN (which I consider vital to R, and vital in distributing our work) in the era of super-popular meta-packages. Meta-packages are convenient, but they have a profoundly negative impact on the packages they exclude.
For example: tidyverse advertises a popular R universe where the vital package data.table never existed.
And now tidymodels is shaping up to be a popular universe where our own package vtreat never existed, except possibly as a footnote to embed.
Users currently (with some luck) discover packages like ours and then (because they trust CRAN) feel able to try them. With popular walled gardens that becomes much less likely. It is one thing for a standard package to duplicate another package (it is actually hard to avoid, and how work legitimately competes), it is quite another for a big-brand meta-package to pre-pick winners (and losers).
All I can say is: please give vtreat a chance and a try. It is a package for preparing messy real-world data for predictive modeling. In addition to re-coding high cardinality categorical variables (into what we call effect-codes after Cohen, or impact-codes), it deals with missing values, can be parallelized, can be run on databases, and has years of production experience baked in.
Some places to start with vtreat:
Collecting Expressions in R
Not a full R article, but a quick note demonstrating by example the advantage of being able to collect many expressions and pack them into a single extend_se() node.
How to use rquery with Apache Spark on Databricks
A big thank you to Databricks for working with us and sharing:
rquery on Databricks is a great data science tool.
John Mount speaking on rquery and rqdatatable
rquery and rqdatatable are new R packages for data wrangling; either at scale (in databases, or big data systems such as Apache Spark), or in-memory. The packages speed up both execution (through optimizations) and development (though a good mental model and up-front error checking) for data wrangling tasks.
Win-Vector LLC‘s John Mount will be speaking on the rquery and rqdatatable packages at the The East Bay R Language Beginners Group Tuesday, August 7, 2018 (Oakland, CA).
Continue reading John Mount speaking on rquery and rqdatatable
Speed up your R Work
Introduction
In this note we will show how to speed up work in R by partitioning data and process-level parallelization. We will show the technique with three different R packages: rqdatatable, data.table, and dplyr. The methods shown will also work with base-R and other packages.
For each of the above packages we speed up work by using wrapr::execute_parallel which in turn uses wrapr::partition_tables to partition un-related data.frame rows and then distributes them to different processors to be executed. rqdatatable::ex_data_table_parallel conveniently bundles all of these steps together when working with rquery pipelines.
The partitioning is specified by the user preparing a grouping column that tells the system which sets of rows must be kept together in a correct calculation. We are going to try to demonstrate everything with simple code examples, and minimal discussion.
seplyr 0.5.8 Now Available on CRAN
We are pleased to announce that seplyr version 0.5.8 is now available on CRAN.
seplyr is an R package that provides a thin wrapper around elements of the dplyr package and (now with version 0.5.8) the tidyr package. The intent is to give the part time R user the ability to easily program over functions from the popular dplyr and tidyr packages. Our assumption is always that a data scientist most often comes to R to work with data, not to tinker with the programming language itself.
Big News: vtreat 1.2.0 is Available on CRAN, and it is now Big Data Capable
We here at Win-Vector LLC have some really big news we would please like the R-community’s help sharing.
vtreat version 1.2.0 is now available on CRAN, and this version of vtreat can now implement its data cleaning and preparation steps on databases and big data systems such as Apache Spark.
vtreat is a very complete and rigorous tool for preparing messy real world data for supervised machine-learning tasks. It implements a technique we call “safe y-aware processing” using cross-validation or stacking techniques. It is very easy to use: you show it some data and it designs a data transform for you.
Thanks to the rquery package, this data preparation transform can now be directly applied to databases, or big data systems such as PostgreSQL, Amazon RedShift, Apache Spark, or Google BigQuery. Or, thanks to the data.table and rqdatatable packages, even fast large in-memory transforms are possible.
We have some basic examples of the new vtreat capabilities here and here.
R Tip: Be Wary of “…”
R Tip: be wary of “...“.
The following code example contains an easy error in using the R function unique().
vec1 <- c("a", "b", "c")
vec2 <- c("c", "d")
unique(vec1, vec2)
# [1] "a" "b" "c"
Notice none of the novel values from vec2 are present in the result. Our mistake was: we (improperly) tried to use unique() with multiple value arguments, as one would use union(). Also notice no error or warning was signaled. We used unique() incorrectly and nothing pointed this out to us. What compounded our error was R‘s “...” function signature feature.
In this note I will talk a bit about how to defend against this kind of mistake. I am going to apply the principle that a design that makes committing mistakes more difficult (or even impossible) is a good thing, and not a sign of carelessness, laziness, or weakness. I am well aware that every time I admit to making a mistake (I have indeed made the above mistake) those who claim to never make mistakes have a laugh at my expense. Honestly I feel the reason I see more mistakes is I check a lot more.
wrapr 1.5.0 available on CRAN
The R package wrapr 1.5.0 is now available on CRAN.
wrapr includes a lot of tools for writing better R code:
let()(let block)%.>%(dot arrow pipe)build_frame()/draw_frame()(data.framebuilders and formatters )qc()(quoting concatenate):=(named map builder)%?%(coalesce) NEW!%.|%(reduce/expand args) NEW!uniques()(safeunique()replacement) NEW!partition_tables()/execute_parallel()NEW!DebugFnW()(function debug wrappers)λ()(anonymous function builder)
I’ll be writing articles on a number of the new capabilities. For now I just leave you with the nifty operator coalesce notation.