Posted on Categories data science, Pragmatic Data Science, Programming, Statistics, TutorialsTags , , 1 Comment on Arbitrary Data Transforms Using cdata

Arbitrary Data Transforms Using cdata

We have been writing a lot on higher-order data transforms lately:

Cdata

What I want to do now is "write a bit more, so I finally feel I have been concise."

Continue reading Arbitrary Data Transforms Using cdata

Posted on Categories Programming, Statistics, TutorialsTags , , , , , , 2 Comments on RStudio Keyboard Shortcuts for Pipes

RStudio Keyboard Shortcuts for Pipes

I have just released some simple RStudio add-ins that are great for creating keyboard shortcuts when working with pipes in R.

You can install the add-ins from here (which also includes both installation instructions and use instructions/examples).

RStudio Logo Blue Gradient

Wraprs BizarroPipe Logo

Posted on Categories Pragmatic Data Science, Pragmatic Machine Learning, Programming, Statistics, TutorialsTags , , , , , , , Leave a comment on Data Wrangling at Scale

Data Wrangling at Scale

Just wrote a new R article: “Data Wrangling at Scale” (using Dirk Eddelbuettel’s tint template).

Fd

Please check it out.

Posted on Categories Administrativia, Statistics, TutorialsTags , , , , , 8 Comments on Update on coordinatized or fluid data

Update on coordinatized or fluid data

We have just released a major update of the cdata R package to CRAN.

Cdata

If you work with R and data, now is the time to check out the cdata package. Continue reading Update on coordinatized or fluid data

Posted on Categories Coding, Opinion, Statistics, TutorialsTags , , , , Leave a comment on Let X=X in R

Let X=X in R

Our article "Let’s Have Some Sympathy For The Part-time R User" includes two points:

  • Sometimes you have to write parameterized or re-usable code.
  • The methods for doing this should be easy and legible.

The first point feels abstract, until you find yourself wanting to re-use code on new projects. As for the second point: I feel the wrapr package is the easiest, safest, most consistent, and most legible way to achieve maintainable code re-use in R.

In this article we will show how wrapr makes code-rewriting even easier with its new let x=x automation.


411gJqs4qlL

Let X=X

Continue reading Let X=X in R

Posted on Categories Coding, data science, Pragmatic Data Science, Programming, Statistics, TutorialsTags , , 1 Comment on Big Data Transforms

Big Data Transforms

As part of our consulting practice Win-Vector LLC has been helping a few clients stand-up advanced analytics and machine learning stacks using R and substantial data stores (such as relational database variants such as PostgreSQL or big data systems such as Spark).


IMG 6061 3

Often we come to a point where we or a partner realize: "the design would be a whole lot easier if we could phrase it in terms of higher order data operators."

Continue reading Big Data Transforms

Posted on Categories Administrativia, data science, StatisticsTags , , , 1 Comment on Some Announcements

Some Announcements

Some Announcements:

  • Dr. Nina Zumel will be presenting “Myths of Data Science: Things you Should and Should Not Believe”,
    Sunday, October 29, 2017
    10:00 AM to 12:30 PM at the She Talks Data Meetup (Bay Area).
  • ODSC West 2017 is soon. It is our favorite conference and we will be giving both a workshop and a talk.
    • Thursday Nov 2 2017,
      2:00 PM,
      Room T2,
      “Modeling big data with R, Sparklyr, and Apache Spark”,
      Workshop/Training intermediate, 4 hours,
      by Dr. John Mount (link).

    • Friday Nov 3 2017,
      4:15 PM,
      Room TR2
      “Myths of Data Science: Things you Should and Should Not Believe”,
      Data Science lecture beginner/intermediate, 45 minutes,
      by Dr. Nina Zumel (link, length, abstract, and title to be corrected).

    • We really hope you can make these talks.

  • On the “R for big data” front we have some big news: the replyr package now implements pivot/un-pivot (or what tidyr calls spread/gather) for big data (databases and Sparklyr). This data shaping ability adds a lot of user power. We call the theory “coordinatized data” and the work practice “fluid data”.
Posted on Categories Administrativia, Opinion, Practical Data Science, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, TutorialsTags , , , , , , 3 Comments on Upcoming data preparation and modeling article series

Upcoming data preparation and modeling article series

I am pleased to announce that vtreat version 0.6.0 is now available to R users on CRAN.


Vtreat

vtreat is an excellent way to prepare data for machine learning, statistical inference, and predictive analytic projects. If you are an R user we strongly suggest you incorporate vtreat into your projects. Continue reading Upcoming data preparation and modeling article series

Posted on Categories Opinion, Programming, TutorialsTags , Leave a comment on On debugging

On debugging

My favorite advice on debugging is from Professor Norman Matloff:

Finding your bug is a process of confirming the many things that you believe are true – until you find one that is not true.


LeafInsect

Continue reading On debugging

Posted on Categories Opinion, Pragmatic Data Science, Pragmatic Machine Learning, Statistics, TutorialsTags , , , , , 2 Comments on My advice on dplyr::mutate()

My advice on dplyr::mutate()

There are substantial differences between ad-hoc analyses (be they: machine learning research, data science contests, or other demonstrations) and production worthy systems. Roughly: ad-hoc analyses have to be correct only at the moment they are run (and often once they are correct, that is the last time they are run; obviously the idea of reproducible research is an attempt to raise this standard). Production systems have to be durable: they have to remain correct as models, data, packages, users, and environments change over time.

Demonstration systems need merely glow in bright light among friends; production systems must be correct, even alone in the dark.


Vlcsnap 00887

“Character is what you are in the dark.”

John Whorfin quoting Dwight L. Moody.

I have found: to deliver production worthy data science and predictive analytic systems, one has to develop per-team and per-project field tested recommendations and best practices. This is necessary even when, or especially when, these procedures differ from official doctrine.

What I want to do is share a single small piece of Win-Vector LLC‘s current guidance on using the R package dplyr. Continue reading My advice on dplyr::mutate()