I need a few volunteers to please “test pilot” the development version of the `R`

package `cdata`

, please.

Jacqueline Cochran: at the time of her death, no other pilot held more speed, distance, or altitude records in aviation history than Cochran.

Skip to content
# Category: Exciting Techniques

Posted on Categories Coding, data science, Exciting Techniques, Programming, Statistics, TutorialsLeave a comment on Wanted: ## Wanted:

Jacqueline Cochran: at the time of her death, no other pilot held more speed, distance, or altitude records in aviation history than Cochran.

Posted on Categories Exciting Techniques, Programming, Statistics, Tutorials4 Comments on Supercharge your R code with wrapr## Supercharge your R code with wrapr

Posted on Categories Administrativia, data science, Exciting Techniques, Practical Data Science, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, TutorialsLeave a comment on Data Reshaping with cdata## Data Reshaping with cdata

Posted on Categories Administrativia, Exciting Techniques, Practical Data Science, Pragmatic Data Science, Pragmatic Machine Learning, StatisticsLeave a comment on Big cdata News## Big cdata News

Posted on Categories Exciting Techniques, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, Tutorials2 Comments on Plotting Deep Learning Model Performance Trajectories## Plotting Deep Learning Model Performance Trajectories

Posted on Categories data science, Exciting Techniques, Pragmatic Data Science, Pragmatic Machine Learning, Programming, Statistics, TutorialsLeave a comment on How to Greatly Speed Up Your Spark Queries## How to Greatly Speed Up Your Spark Queries

Posted on Categories Coding, data science, Exciting Techniques, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, Tutorials1 Comment on Win-Vector LLC announces new “big data in R” tools## Win-Vector LLC announces new “big data in R” tools

Continue reading Win-Vector LLC announces new “big data in R” tools
Posted on Categories Exciting Techniques, Practical Data Science, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, Tutorials2 Comments on Partial Pooling for Lower Variance Variable Encoding## Partial Pooling for Lower Variance Variable Encoding

Posted on Categories data science, Exciting Techniques, Practical Data Science, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, Tutorials## Custom Level Coding in vtreat

Posted on Categories Exciting Techniques, Programming, Statistics, Tutorials1 Comment on Neat New seplyr Feature: String Interpolation## Neat New seplyr Feature: String Interpolation

`cdata`

Test Pilots`cdata`

Test PilotsI need a few volunteers to please “test pilot” the development version of the `R`

package `cdata`

, please.

Jacqueline Cochran: at the time of her death, no other pilot held more speed, distance, or altitude records in aviation history than Cochran.

I’ve just shared a short webcast on data reshaping in R using the cdata package.

(link)

We also have two really nifty articles on the theory and methods:

Please give it a try!

This is the material I recently presented at the January 2017 BARUG Meetup.

I have some big news about our R package cdata. We have greatly improved the calling interface and Nina Zumel has just written the definitive introduction to cdata.

cdata is our general coordinatized data tool. It is what powers the deep learning performance graph (here demonstrated with R and Keras) that I announced a while ago.

However, cdata is *much* more than that.

I am excited to share a new deep learning model performance trajectory graph.

Here is an example produced based on Keras in R using ggplot2:

Continue reading Plotting Deep Learning Model Performance Trajectories

For some time we have been teaching `R`

users "when working with wide tables on Spark or on databases: narrow to the columns you really want to work with early in your analysis."

The idea behind the advice is: working with fewer columns makes for quicker queries.

The issue arises because wide tables (200 to 1000 columns) are quite common in big-data analytics projects. Often these are "denormalized marts" that are used to drive many different projects. For any one project only a small subset of the columns may be relevant in a calculation.

Win-Vector LLC is proud to introduce two important new tool families (with documentation) in the `0.5.0`

version of `seplyr`

(also now available on CRAN):

`partition_mutate_se()`

/`partition_mutate_qt()`

: these are query planners/optimizers that work over`dplyr::mutate()`

assignments. When using big-data systems through R (such as PostgreSQL or Apache Spark) these planners can make your code faster and sequence steps to avoid*critical*issues (the complementary problems of too long in-mutate dependence chains, of too many mutate steps, and incidental bugs; all explained in the linked tutorials).`if_else_device()`

: provides a`dplyr::mutate()`

based simulation of per-row conditional blocks (including conditional assignment). This allows powerful imperative code (such as often seen in porting from SAS) to be directly and legibly translated into performant`dplyr::mutate()`

data flow code that works on Spark (via Sparklyr) and databases.

Image by Jeff Kubina from Columbia, Maryland – [1], CC BY-SA 2.0, Link

In a previous article, we showed the use of partial pooling, or hierarchical/multilevel models, for level coding high-cardinality categorical variables in `vtreat`

. In this article, we will discuss a little more about the how and why of partial pooling in `R`

.

We will use the `lme4`

package to fit the hierarchical models. The acronym “lme” stands for “linear mixed-effects” models: models that combine so-called “fixed effects” and “random effects” in a single (generalized) linear model. The `lme4`

documentation uses the random/fixed effects terminology, but we are going to follow Gelman and Hill, and avoid the use of the terms “fixed” and “random” effects.

The varying coefficients [corresponding to the levels of a categorical variable] in a multilevel model are sometimes called

random effects, a term that refers to the randomness in the probability model for the group-level coefficients….

The term

fixed effectsis used in contrast to random effects – but not in a consistent way! … Because of the conflicting definitions and advice, we will avoid the terms “fixed” and “random” entirely, and focus on the description of the model itself…

– Gelman and Hill 2007, Chapter 11.4

We will also restrict ourselves to the case that `vtreat`

considers: partially pooled estimates of conditional group expectations, with no other predictors considered.

Continue reading Partial Pooling for Lower Variance Variable Encoding

One of the services that the `R`

package `vtreat`

provides is *level coding* (what we sometimes call *impact coding*): converting the levels of a categorical variable to a meaningful and concise single numeric variable, rather than coding them as indicator variables (AKA "one-hot encoding"). Level coding can be computationally and statistically preferable to one-hot encoding for variables that have an extremely large number of possible levels.

By default, `vtreat`

level codes to the difference between the conditional means and the grand mean (`catN`

variables) when the outcome is numeric, and to the difference between the conditional log-likelihood and global log-likelihood of the target class (`catB`

variables) when the outcome is categorical. These aren’t the only possible level codings. For example, the `ranger`

package can encode categorical variables as ordinals, sorted by the conditional expectations/means. While this is not a completely faithful encoding for all possible models (it is not completely faithful for linear or logistic regression, for example), it is often invertible for tree-based methods, and has the advantage of keeping the original levels distinct, which impact coding may not. That is, two levels with the same conditional expectation would be conflated by `vtreat`

‘s coding. This often isn’t a problem — but sometimes, it may be.

So the data scientist may want to use a level coding different from what `vtreat`

defaults to. In this article, we will demonstrate how to implement custom level encoders in `vtreat`

. We assume you are familiar with the basics of `vtreat`

: the types of derived variables, how to create and apply a treatment plan, etc.

The `R`

package `seplyr`

has a neat new feature: the function `seplyr::expand_expr()`

which implements what we call “the string algebra” or string expression interpolation. The function takes an expression of mixed terms, including: variables referring to names, quoted strings, and general expression terms. It then “de-quotes” all of the variables referring to quoted strings and “dereferences” variables thought to be referring to names. The entire expression is then returned as a single string.

This provides a powerful way to easily work complicated expressions into the `seplyr`

data manipulation methods. Continue reading Neat New seplyr Feature: String Interpolation