A note to
dplyr with database users: you may benefit from inspecting/re-factoring your code to eliminate value re-use inside
dplyr::mutate() statements. Continue reading Please inspect your dplyr+database code
Our article "Let’s Have Some Sympathy For The Part-time R User" includes two points:
- Sometimes you have to write parameterized or re-usable code.
- The methods for doing this should be easy and legible.
The first point feels abstract, until you find yourself wanting to re-use code on new projects. As for the second point: I feel the
wrapr package is the easiest, safest, most consistent, and most legible way to achieve maintainable code re-use in
In this article we will show how
wrapr makes code-rewriting even easier with its new
let x=x automation.
Continue reading Let X=X in R
I am pleased to announce that
vtreat version 0.6.0 is now available to
R users on CRAN.
vtreat is an excellent way to prepare data for machine learning, statistical inference, and predictive analytic projects. If you are an
R user we strongly suggest you incorporate
vtreat into your projects. Continue reading Upcoming data preparation and modeling article series
My favorite advice on debugging is from Professor Norman Matloff:
Finding your bug is a process of confirming the many things that you believe are true – until you find one that is not true.
Continue reading On debugging
There are substantial differences between ad-hoc analyses (be they: machine learning research, data science contests, or other demonstrations) and production worthy systems. Roughly: ad-hoc analyses have to be correct only at the moment they are run (and often once they are correct, that is the last time they are run; obviously the idea of reproducible research is an attempt to raise this standard). Production systems have to be durable: they have to remain correct as models, data, packages, users, and environments change over time.
Demonstration systems need merely glow in bright light among friends; production systems must be correct, even alone in the dark.
“Character is what you are in the dark.”
John Whorfin quoting Dwight L. Moody.
I have found: to deliver production worthy data science and predictive analytic systems, one has to develop per-team and per-project field tested recommendations and best practices. This is necessary even when, or especially when, these procedures differ from official doctrine.
What I want to do is share a single small piece of Win-Vector LLC‘s current guidance on using the
dplyr. Continue reading My advice on dplyr::mutate()
Authors: John Mount and Nina Zumel.
p-value is a valid frequentist statistical concept that is much abused and mis-used in practice. In this article I would like to call out a few features of
p-values that can cause problems in evaluating summaries.
Keep in mind:
p-values are useful and routinely taught correctly in statistics, but very often mis-remembered or abused in practice.
Continue reading Remember: p-values Are Not Effect Sizes
- Question: how hard is it to count rows using the
- Answer: surprisingly difficult.
When trying to count rows using
dplyr controlled data-structures (remote
tbls such as
dbplyr structures) one is sailing between Scylla and Charybdis. The task being to avoid
dplyr corner-cases and irregularities (a few of which I attempt to document in this "
Continue reading It is Needlessly Difficult to Count Rows Using dplyr
Recently I noticed that the
sparklyr had the following odd behavior:
#>  '0.7.2.9000'
#>  '0.6.2'
#>  '126.96.36.19900'
sc <- spark_connect(master = 'local')
#> * Using Spark: 2.1.0
d <- dplyr::copy_to(sc, data.frame(x = 1:2))
#>  NA
#>  NA
#>  NA
This means user code or user analyses that depend on one of
nrow() possibly breaks.
nrow() used to return something other than
NA, so older work may not be reproducible.
In fact: where I actually noticed this was deep in debugging a client project (not in a trivial example, such as above).
Tron: fights for the users.
In my opinion: this choice is going to be a great source of surprises, unexpected behavior, and bugs going forward for both
dbplyr users. Continue reading Why to use the replyr R package
dplyr is one of the most popular
R packages. It is powerful and important. But is it in fact easily comprehensible? Continue reading Is dplyr Easily Comprehensible?
Somebody nice reached out and gave us this wonderful feedback on our new Supervised Learning in R: Regression (paid) video course.
Thanks for a wonderful course on DataCamp on
Random forest. I was struggling with
Xgboost earlier and
Vtreat has made my life easy now :).
Continue reading Thank You For The Very Nice Comment