1) Jacques Hadamard in “An Essay on the Psychology of Invention in the Mathematical Field” called out the importance of non-voluntary intuitive creative leaps that occur in rest periods between intervals of intense work and preparation.

2) It has been noted again and again that what actually makes people happy (versus what they anticipate would make them happy) are activities and experiences with rising challenges (for example see Daniel Gilbert’s “Stumbling on Happiness”).

3) It is folklore that a number of the greatest computer scientists are also fairly accomplished musicians.

And here is the punch-line: take up a skill building hobby (in my case I am trying to learn how to draw). You definitely enjoy it, but some part of your subconscious also resents being made to work (learning is work, don’t confuse that with repetition). To defend itself your subconscious then starts throwing out more and better technical ideas during periods of repose. Jot these down (without trying to work on them). The effect is even stronger than Hadamard’s effect (where your brain is solving problems for you to end an effort) as it is closer to the classic trick of making progress on one task by procrastinating on another task.

This is similar to the “left brain/right brain” ideas of the 1970s (it assumes the existence of a subconscious) but assumes far less unverified structure of a subconscious. And here is where the “10,000 hours to mastery effect” (Malcolm Gladwell, “Outliers: The Story of Success”) works in your favor- you can use the same source of deliberate practice (remember you have to be learning not puttering around) for a long time.

I think if you are in good health and have enough energy you can pull this trick off at will.

No related posts.

But researchers in other fields are not always so good about making copies of their papers easily available, and older classic papers (say, for example, Bradley Efron’s 1979 Annals of Statistics paper on the Jackknife) are often still worth reading, but are not always easy to find. Where to go?

This is a list of some resources that I’ve discovered over the years. The list isn’t comprehensive, by any means, but I offer them here because maybe you will find them helpful, too. The list, and my opinions, are biased towards research in the mathematical and computer sciences, but many of these resources are potentially useful for any research area, including the humanities.

JSTOR

JSTOR is a digital archive of over one thousand scholarly journals, covering topics in the humanities, social and physical sciences and mathematics. I love JSTOR. It is an incredibly useful resource, containing the full contents of every issue of every journal in their collection up to within 3-5 years of the present time (it’s a moving wall). The collection is full-text searchable. I use JSTOR to find classic papers in Math, Statistics, and Computer Science, as well as more recent papers that have been published in journals that are otherwise not available to me.

Access to JSTOR is available to members of participating institutions, mostly universities, but also many public libraries. I have access to JSTOR free with my San Francisco Public Library card, via the SFPL website. (I believe that any resident of California is eligible for a SFPL library card with proof of California residency; good news if you are in California and your local library doesn’t subscribe).

As a side note, San Francisco Public Library subscribes to several quite useful digital research services, including FirstSearch, the OED, Encyclopedia Brittanica, and Morningstar. Some of these other services also provide access to selected full-text articles. SFPL also participates in ILL (Interlibrary Loan) and Link+, a similar cross-library loan service. All good reasons to support your local library!

ArXiv

ArXiv is a pre-print server hosted by Cornell, serving pre-prints of papers in Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance and Statistics. Many important researchers use ArXiv to get around the fact that major journal publishers insist on holding the copyright to articles published in their journals. “Pre-prints” haven’t yet been published, and hence the authors are free to distribute them freely. Fields Medalist Terence Tao regularly distributes his about-to-be published work through ArXiv.

On the other hand, ArXiv has very open submission policies, so you should be more careful of the papers you find here than you would be with a refereed or curated source, such as JSTOR or PubMed Central (which we will discuss later). ArXiv has, unfortunately, more than its fair share of what Augustus de Morgan used to politely call “paradoxers“. The “Journal Reference” field of the article summaries will generally give you an indication of whether or not the paper is legitimate, in the sense of having been peer-reviewed; but note, for instance, this paper on a polynomial-time algorithm for Traveling Salesman (the Traveling Salesman problem is provably NP-complete, so a result of this magnitude would win the Clay Millennium Prize, if true).

Another side note: I’ve linked to the Amazon page on de Morgan’s Budget of Paradoxes because that was the first synopsis I found. The copyright on the book has expired, so if you are actually interested in reading it (it’s fairly funny, in places), you can find the full version on Google Books or Project Gutenberg.

CiteSeerX

CiteSeer was the original search engine and archive for online technical papers; it got me through graduate school, and my first post-PhD position at SRI. I don’t believe that the original CiteSeer system is still active, but its successor, CiteSeerX, is being developed and hosted at Penn State. It concentrates on computer science literature, as did the original. CiteSeerX builds its corpus by webcrawling, so again, the papers it finds are not necessarily refereed. Like its predecessor, CiteSeerX search results include the paper’s abstract, a BibTex citation, a list of the paper’s references, a pointer to the paper’s original location, and (usually) an archived version of the paper, in case the original link has gone dead. Good stuff.

AccessMyLibrary

AccessMyLibrary is a service that pools the periodical resources of several libraries across the United States. Any article in a periodical held by a participating library is available for free download to anyone who holds a library card in any other participating library. I find this service less useful than JSTOR: the holdings are generally newspapers and popular magazines, although there are some journals represented, as well as law and business reviews. The download format strips all of the original formatting from articles, which makes them rather ugly and a bit harder to read. I think you lose the figures, too. Still, it’s free if you have a library card, and it’s a good place to search for an article if you can’t find it anywhere else.

Questia

Questia is a for-pay service that claims to have “the world’s largest online collection of books and journal articles in the humanities and social sciences, plus magazine and newspaper articles”. Their collection is full-text searchable and, as they say, “you can read every title cover to cover”. Good luck doing so, though — articles and book chapters are not downloadable. Instead, you have to read them through Questia’s online interface, which is pretty clunky. On the plus side, they allow you to build your own “bookshelves” to collect books and articles that are relevant to you by topic or project. You can bookmark key sections, and highlight key passages. I used Questia when I was involved in research projects with psychology and organizational science aspects. I could get hold of articles or textbooks that I wanted to look at faster than through Interlibrary Loan, and more conveniently than going down to Stanford. The subscription fee at the time was cheaper than a membership to the APA or buying the articles piecemeal from Elsevier, or whoever.

Currently, Questia’s subscription fee is $19.95/month for full library access; you can also subscribe to specific collections (such as Psychology, Literature, or Philosophy) for $9.95 per collection per month.

Mendeley

Another way to find useful literature is to connect with other people out there who share your interests. Mendeley is a tool that allows you to organize your collection of research papers, share it with colleagues, and to peruse the collections of other researchers with similar interests. I haven’t used it myself; but a friend of ours who is an active and influential AI researcher recommends it. It’s certainly worth a mention.

PubMed Central

PubMed Central is a free digital archive of biomedical and life sciences journal literature, sponsored and managed by the NIH. We don’t do life science research here at Win-Vector, but I’m mentioning PubMed because of this awesome policy by the NIH:

NSF and DoD should institute similar policies, too.

Google Books, Google Scholar

Yes, they’re out there. Personally, I find them less useful than JSTOR or a subscription to (say) IEEE Explore. Google Scholar generally returns the abstracts of articles at sites that don’t provide open access to the full-text article, such as the website of the journal that published the article, or the website of a restricted research archive, like the ACM. This is useful, in that it tells you that the article exists, but it’s rather frustrating, too. I don’t find Google Scholar to be significantly more helpful than doing a general Google search on the same keywords. On the other hand, some people swear by Google Scholar, so obviously your mileage may vary.

Google Books has a very annoying habit of returning hits on your search terms, then not giving you read access to the page in question. Useless. If you happen to be doing research in an area where older books in the public domain are still of interest (for instance, my amateur interest in folklore and mythology), then Google Books can be quite helpful; of course, this situation is generally not true in technical research.

Offline: Your Local University Library

Here in the Bay Area, we are fortunate because the Stanford Library System has generous visitor access policies. The visitors’ policy statement is here; briefly, non-Stanford visitors are allowed 7 courtesy visits per year, with no borrowing privileges. For more visits, you can purchase an access card. I used the Stanford Libraries when my company was down in Mountain View, and I’m grateful for their openness. I don’t think many universities are as generous as Stanford is, but if you are near a university campus, it doesn’t hurt to check. For instance, the University of San Francisco will sell access cards to their library, with or without borrowing privileges, to non-affiliated visitors (it ain’t cheap), and allows practicing California attorneys access to their Law Library. San Francisco State has a Friends of the Library program, whereby non-affiliated visitors have access and borrowing privileges to the CSUSF library collection for $45/year.

And there you have it. Research away!

Related posts:

1. YAYGDA (Yet Another Yahoo Google Deal Article)
2. I know, I am the one being a jerk
3. Must Have Software

As more of our life becomes digital (work, finances, passwords, pictures, contacts,dairies,videos and email) we must be more diligent in backing up our data. If your hard drive fails at work you might lose some spreadsheets (and you might not lose anything if your IT department is on their toes) if you computer fails at home you lose your wedding album. Your hard disk will fail and try to take all of your data (life) with it- it is a matter of when not a matter of if. You want this to be an inconvenience, not a disaster. Become expert at backing up and take the time to help others.

First some definitions. Everything stored on your computer is called “data” and it is most commonly stored on a single “hard drive.” The act of making an extra copy of your data is traditionally called “backing up.” The act of trying to get access to your extra copy of your data is traditionally called “restoring.” The whole point of backing up is to be able to restore. If you can’t get your data back it really doesn’t matter what steps you took. Backing up with no ability to restore is just cargo cult behavior.

If you have a professional service available they will likely do a better job than you can (this is one reason that larger businesses have professional IT staffs). However, at home you are likely on your own.

This is an opinion piece and I am advocating backing up everything (whole drives) locally. If you do not back up everything you will need to choose what to back up and what to skip- and you will make mistakes and lose things. If you do not have a local back up, you might not be able to restore (back up service goes out of business, internet connections are still too slow to be practical). At the very least you should have a local back up; a remote back up is a good second step. Remote back up services are a good idea for important data and there are some high quality ones, but a few have gone out of business (Xdrive) so do not want one to be your only chance of salvation.

Let us first address a technical issue- what sort of set-up are you backing up? The three most common situations are: Windows, OSX and other Unix (Linux/BSD, yes I know OSX is a Unix). Each of these have different appropriate tools:

• Windows:

  For Windows Home type operating systems you are unlikely to have access to Microsoft’s back up tools (which is a real shame, the tools are more useful at home than at a business). So you need to install something.

  I have not researched the Windows world extensively, so I can not give advice. I can, however relate my experiences and current policies.

  I now avoid EMC Retrospect (often comes free with USB drives) at all costs. My experience has been that EMC Retrospect is hard to use to restore your data (the whole point of backing up). For me it often refused to run (due to licensing issues) and it was very sensitive to the exact version of the Microsoft.Net framework that was installed on my Windows system. Two separate times an update in the Microsoft.Net system rendered EMC Retrospect unusable (and broke nothing else).

  I have happily purchased Acronis True Image three times now (twice for myself and once for a friend). Their website is a bit confusing (you must be careful to get the retail product, not the many thousands of dollar enterprise product). The software seems to be very good. It can back up, restore and can even read data from an “image” (which means you can get to your data with out even restoring).

• OSX:

  An embarrassment of riches:

  The free options include following Jamie Zawinski’s wonderful advice (which I am shamelessly stealing from here) , using the free copy of SuperDuper! (which is very good and a complete back up solution even in the free version) or Time Machine (the back up utility included in the current Mac operating system: Leopard).

  One huge advantage of modern Macs is if you have formatted your drive correctly you can boot off a USB drive. So if you use the above instructions you can plug your back in and use it to run (delaying your need to open up your machine or attempt a restore until later). This is also important in rehearing your restore procedures.

  Finally, if you have the cash there is the somewhat over-priced (but wonderful) Time Capsule. You can live without Time Capsule, but it is part of my “dream set up” (described below).

• Unix:

  Follow any sort of advice on how to script back ups (such as Jamie Zawinski’s) and you should be protected. Rsync is a great tool.

More important than the back up tools is having a precise back up goal and a matching back up plan. I use my own goal and plan as an example and you can use it as a basis for safer or more risky plans (depending on your resources and needs).

My goal is to: (with very high probability) not lose more than a week of my life. The plan to achieve this is a full local back up every week and the willingness to buy some new equipment if I have to do a restore. A failure could delay my work for a day or so, but not put me out of business. For my business it does not make sense to ensure “no down time”- this is an unreasonably expensive thing to try to achieve (and the inappropriateness of this goal is one reason many people have no back ups at all). My worst case “restore” plan is to drive to a store and buy the cheapest temporary computer. A more likely case is I just need to use one of my extra drives to do my restore (very cheap). I would then restore the back up onto a fresh drive (or the temporary computer) and work from there until I could repair or replace my major system.

My back up plan has several “eyes open” weaknesses. I only back up every week, so I could lose a week’s of data if my disk dies right before a back up. Also, to restore my data could take a day and $500 (trip to store to buy a temporary computer and hours to restore drive contents). Knowing these weaknesses are the point of the back up plan: I am trading hoping that my drive doesn’t blow up and take all of my data away for hoping my drive doesn’t blow up and cost me a day of work and few hundred dollars. That is I am trading the Sword of Damocles for worrying about something like stubbing my toe. Drive failures while inevitable are not frequent. if I put a quarter in a jar every day I don’t have a drive failure I would more than likely have the $500 needed to perform an emergency restore saved up long before I have a drive failure. By not purchasing excess extra equipment (computers) before the failure I save money by maybe not having to purchase it all or at least purchasing cheaper and better equipment at the time of failure (instead of now).

Now to describe my implementation of my plan. First I purchased the following
things:

• Time Capsule (optional):

  

• Thermaltake External Hard Drive SATA Dock ($40 : Newegg):

  

• Two 1TB drives ($90/each Newegg, these
  are the cheaper “internal” drives that go into desktop computers or into the Themaltake dock. If you don’t like the ugly you could skip the Themaltake dock and buy USB drives instead.):

  

So for a little over $220 I am in business. Every week I could take one of the drives out of its envelope, stick it in the Thermaltake dock and use one of the tools described above to create a complete back up. What I actually do is even better. Any time I want I ask my computer to use Time Machine to back up to the Time Capsule (typically takes about 20 minutes) and then once a week I stick a drive in the Themaltake dock and let the Time Capsule copy itself onto the drive (so both me and my computer are completely uninvolved in the 8 hours this step can take). For offsite back ups (to defend against things like fire) I can take one of the drives to a safe place off site (locker, safe deposit box). I recommend physical protection (locks, fire safes) to protect your drives (not encryption, there is a good chance you will get something wrong with encryption and not be able to restore).

Using Time Machine gives me the benefit of having multiple back ups so I can look at earlier versions of files and the speed of only needing to perform incremental back ups (only what has changed needs to be copied). Another way to get the advantage of having extra versions of all of your files is to put most of your files under management of a “source control system” like Bazaar. Systems like this (free, runs on Windows, OSX and Unix) let you keep all versions of all of your files (answers things like “what did I have in the file before I deleted it last week?”) and are incredibly useful (you will wonder how you lived without them).

Finally I end with some “defensive thinking” required to succeed with back ups. I have not said why I purchased two extra drives. This is so I can rotate which extra drive I back up onto. Drives most often fail when being used- so it is very plausible that my main machine could die while backing up. If the main machine dies while backing up then not only is its data lost but the back up is also useless (as the main machine was interrupted while trying to write it out). This is not quite ironic because while it is contrary to what you would want it is not unexpected. To be safe from a failure during the back up procedure you must have a second drive that is not being used. Only after the first back up is known to have succeeded can you then back up onto the other drive.

You must rehearse and think through all of your back up steps. If you are lucky you will find flaws in your plan during rehearsals instead of when you go to restore. For example tape back up procedures are notorious for writing out years incremental back ups that don’t work during a restore attempt. Use a system that allows safe rehearsals (such as trying to boot from a bootable back up or inspect a file from an Acronis image or Time Machine archive). Plans that only allow restores are not safely rehearsable (if the rehearsal fails you damage something on your primary machine). Also: if you are really trying to restore you are not likely to be in a good mood, iron out potential kinks with rehearsals not during a panic.

No plan is perfect- we can not cheaply eliminate all risk. In this case what we can do is eliminate exposure to likely scenarios. Data loss can still happen, but it does not have to be not inevitable.

Update 7-8-2011: Survived a double failure. MacBook Pro logic board died. Dock burned up trying to duplicate TimeCapsule contents. In the end MacBook Pro was repaired with no cost or data-loss (yey Apple care and Boo defective Nvidia chips) and TimeCapsule was able to back up to another USB drive. So came out of a double failure with two usable copies of the data.

For Linux machines adding a second drive devoted to Rsnapshot right in the case (like TimeCapsule inside the box, still need external backup rotation of course).

Related posts:

1. Public Service Article: JSTOR and other Useful Research Archives
2. YAYGDA (Yet Another Yahoo Google Deal Article)
3. Microsoft Store Again