John Mount, Nina Zumel; Win-Vector LLC 2019-04-27
In this note we will use five real life examples to demonstrate data layout transforms using the
R package. The examples for this note are all demo-examples from tidyr:demo/ (current when we shared this note on 2019-04-27, removed 2019-04-28), and are mostly based on questions posted to StackOverflow. They represent a good cross-section of data layout problems, so they are a good set of examples or exercises to work through.
Continue reading Data Layout Exercises
cdata implements an operator notation for data transform.
The idea is simple, yet powerful.
Continue reading Operator Notation for Data Transforms
We’ve been getting some good uptake on our piping in
R article announcement.
The article is necessarily a bit technical. But one of its key points comes from the observation that piping into names is a special opportunity to give general objects the following personality quiz: “If you were an
R function, what function would you be?”
Continue reading “If You Were an R Function, What Function Would You Be?”
R users have been enjoying the benefits of
SQL query generators for quite some time, most notably using the
dbplyr package. I would like to talk about some features of our own
rquery query generator, concentrating on derived result re-use.
Continue reading Query Generation in R
R package and training materials we emphasize the record-oriented thinking and how to design a transform control table. We now have an additional exciting new feature: control table keys.
The user can now control which columns of a
cdata control table are the keys, including now using composite keys (that is keys that are spread across more than one column). This is easiest to demonstrate with an example.
Continue reading cdata Control Table Keys
Composing functions and sequencing operations are core programming concepts.
Some notable realizations of sequencing or pipelining operations include:
The idea is: many important calculations can be considered as a sequence of transforms applied to a data set. Each step may be a function taking many arguments. It is often the case that only one of each function’s arguments is primary, and the rest are parameters. For data science applications this is particularly common, so having convenient pipeline notation can be a plus. An example of a non-trivial data processing pipeline can be found here.
In this note we will discuss the advanced
R pipeline operator "dot arrow pipe" and an
S4 class (
wrapr::UnaryFn) that makes working with pipeline notation much more powerful and much easier.
Continue reading Function Objects and Pipelines in R
One of the design goals of the
R package is that very powerful and arbitrary record transforms should be convenient and take only one or two steps. In fact it is the goal to take just about any record shape to any other in two steps: first convert to row-records, then re-block the data into arbitrary record shapes (please see here and here for the concepts).
But as with all general ideas, it is much easier to see what we mean by the above with a concrete example.
Continue reading Fully General Record Transforms with cdata
RcppDynProg is a new
R package that implements simple, but powerful, table-based dynamic programming. This package can be used to optimally solve the minimum cost partition into intervals problem (described below) and is useful in building piecewise estimates of functions (shown in this note).
Continue reading Introducing RcppDynProg
vtreat‘s purpose is to produce pure numeric
data.frames that are ready for supervised predictive modeling (predicting a value from other values). By ready we mean: a purely numeric data frame with no missing values and a reasonable number of columns (missing-values re-encoded with indicators, and high-degree categorical re-encode by effects codes or impact codes).
In this note we will discuss a small aspect of the
vtreat package: variable screening.
Continue reading vtreat Variable Importance
R are popular, the most popular one being
magrittr as used by
This note will discuss the advanced re-usable piping systems:
rqdatatable operator trees and
wrapr function object pipelines. In each case we have a set of objects designed to extract extra power from the
wrapr dot-arrow pipe
Continue reading Reusable Pipelines in R