Sign up and participate for free at http://training.oreilly.com/googleapps/. The course is going to be highly interactive, and O’Reilly is offering a pretty delicious bundle of free content to participants.

The course is being produced by creativeLIVE in partnership with O’Reilly Media, and we’re going to cover the full range of capabilites of Google Apps that are of interest to businesses of all sizes.

I’ve been putting the course together over the past couple of months with my friend, editor, and co-host Brian Sawyer, along with author, visionary, and Head Firsty extraordinaire Beth Robson. It’s going to be a blast… hope you can join us!

I’d like to add a few points about why this and other Kickstarter projects are such good fundraising opportunities and how these successes look next to classic nonprofit fundraising techniques.

Kickstarter makes giving personal

Craig doesn’t say this in his blog entry, but there can be no doubt that a large part of the success of the project stems from the credibility he and his collaborator Ashley Rawlings have. If you know their work you know they’re not going to drop the ball on this project.

So people funding the Kickstarter project are motivated by more than seeing a book re-released. They want to fund Craig’s and Ashley’s re-release of a book. And many, I suspect, just want to fund Craig and Ashley doing good work of any sort. Who wouldn’t want to bet on a winner?

This personal dimension of fundraising may be the most important factor in the selection of deserving projects. It becomes more complicated as projects and organizations scale up, but it worked elegantly in this project.

People love restricted giving

In fundraising parlance the restricted/nonrestricted gift distinction corresponds to whether gift is for a special project or whether it can be used for whatever project or program the recipient sees fit. For better or worse, all else being equal, people prefer to give restricted gifts for projects. Doing so gives them a more concrete sense that their dollars are going directly to fund what they want to see happen.

Unsurprisingly, nonprofits really like unrestricted gifts. It would be hard to pay for an organization that did more that a couple of things using restricted dollars exclusively. And nonprofits that provide an ongoing service are engaged in a process rather than a project. For these groups it’s reasonable to say, “We do x, y, and z. You know that someone needs to do these things, so give us money that we can use at our discretion to keep these processes running.”

It’s exciting to be involved in a special project with a concrete outcome. So I’d predict that the Pre/Post will have an easier time raising restricted funds for specific projects than unrestricted funds for the more general pursuit of quality publishing.

Kickstarter demands honesty from solicitors

If a Kickstarter project does not secure pledges equal to or greater than the target amount, then no donor pays.

This gives the seeker of funds an incentive to set her target honestly. If it really costs you $1000 to do a project, but you think you can raise $3000, so you set your target to $2000, and you raise $1900, then you receive $0. There you were, on track to receive double the cost of your project, and now you have nothing.

On the flip side, you’d be nuts to set your target to lower than the anticipated actual cost of the project. You’d be lying to your friends, and if the campaign was a success you’d have to figure out how to make up the difference.

Craig’s campaign revenues exceeded his target, and he suggests that a takeaway from this experience is that one shouldn’t be shy about setting one’s target high. I’d say to set it at the high end of your range of likely costs. Go higher and risk losing everything. You can always start a new project if the goodwill you discover in your fundraising exceeds what you’d expected.

In nonprofit capital campaigns, fundraisers go to great lengths to set accurate targets for timing and funds raised before they go public about their initiative. One reason is that they begin collecting cash from donors before the end of the campaign, unlike what you have in Kickstarter, and it’d be painful and embarrassing to have to return money.

Doing so would show that you weren’t competent enough to research your campaign adequately before beginning. On the other hand, a Kickstarter campaign goes public before anyone puts up any money. So perhaps the more money you want to raise, the more certainty you need before you go public, the less likely that Kickstarter would be a suitable platform.

There’s lots of great “Ask” data in Kickstarter

Asking for the right amount of money and setting the right giving tiers is a big deal in fundraising, and I’ve spent quite a lot of time designing tiers for different organizations and creating models to assign gift-maximizing “Ask” amounts to donors in light of their donation history (and myriad other factors).

Kickstarter projects provide a great resource to show where people tend to end up along an ordinal spectrum of ask amounts. They’re worth researching and pondering, no matter what sort of fundraising you’re doing.

Final thoughts

Craig Mod’s project is really cool and illustrates a style of giving that lots of people find attractive. I don’t see a Kickstarter model supplanting most classic forms of fundraising yet, but you’d be nuts not to experiment with it.

The data that best shows what people want to support would be a record of what they actually support, and experimental fundraising projects on Kickstarter are a great way to create that data.

Analytics is a big field with huge potential for nonprofits, and in my presentation I go high-level and discuss a few preliminary ideas about how analytics should fit within the development enterprise (there may be a book in there some day). I hope you enjoy it!

How this book is different: The Excel part

I wrote this book with a specific idea about what it means to be good at using Excel. People who know Excel know formulas. And by “knowing formulas,” I mean that these users know a bunch of different functions, they know how to make those functions work with each other, and they know how to find new functions when the functions they know aren’t sufficient to solve their problems.

If you want to get good at Excel, you need to master formulas.

And if you want to have fun with Excel, formulas are the way to do it, too. It’s enormously gratifying to write just a line or two of code that makes your data sing, or to make data do or say something that your colleagues never thought possible. It’s also gratifying to take someone else’s spreadsheet — something consisting of an elaborate soup of formulas — and figure out how it works.

You can use Excel without being a formula master, but the power of Excel is in its formulas, and I’m so excited about Head First Excel because it will take total formula newbies and help them develop into masters. The book assumes you know nothing, but by the end you’re doing seriously heavy stuff with spreadsheets. Here’s the chapter list:

1. Introduction to formulas
2. Visual design
3. References
4. Change your point of view
5. Data types
6. Dates and times
7. Finding functions
8. Formula auditing
9. Charts
10. What if analysis
11. Text functions
12. Pivot tables
13. Booleans
14. Segmentation

Plenty of the chapters aren’t explicitly about formulas, but every one of them has you practicing this crucial topic at some level. And what surprised me so much about other books on Excel is that they don’t ask you to practice formulas. You’d think that something so fundamental to getting good at the program would receive more treatment.

How this book is different: The Head First part

The style of Head First books, as you can see from the free sample chapters for each book, is colloquial and visual. But don’t imagine for a second that this means the content is simplistic. The style makes the books very highly optimized for learning. You pick up concrete skills in a fun but inexorable way as your progress through the book.

We who write and publish these books believe that the difficulty people experience learning technical topics comes not from the difficulty of the topics of themselves but rather from poor presentation by the teachers. Head First Excel is a 448-page book that’s designed to be read straight through without the reader ever getting bored or frustrated.

I started off as a fan, reading Head First books back when I read the first edition of Head First Java as a theology student (long story). During the past several years O’Reilly has published a whole mess of Head First books, and they’re all fantastic. Last year I wrote Head First Data Analysis, so Excel is my second Head First book.

I hope you enjoy Head First Excel. Being able to use this software’s powerful features is one of the most important skills you can have in business, and frankly I don’t think I’ve made an important decision in my life or career without firing up Excel first.

Several weeks back I was thinking about how the Bible is taught at seminaries, and on a lark I decided to check out the website of a big school 200 miles from my apartment in Texas: Dallas Theological Seminary. I wondered, who are the people who teach Bible here? As I looked through their bios, something surprised me. By my count, 26 of their 29 Bible professors received degrees from Dallas Theological Seminary.

Interpretations of this statistic will vary. I don’t know anyone who works at DTS, but I suspect that their Bible faculty would say that they their professors were chosen because they are the best scholars in the field, and that nearly all of them have DTS degrees because DTS is the best school in the field. Someone with views like my own, on the other hand, would suspect that the statistic reveals an extreme form of insularity, with older professors all having come up together and having taught the current crop of younger professors, creating a strong network of mutual reinforcement for a static set of ideas.

I might be wrong. Anyhow, the DTS website piqued my interest, so then I looked at the websites for every single seminary in United States and recorded the names of all their Bible professors as well as where those professors got their various degrees. I compiled the data, and here it is:

Seminary Data April 2010

I haven’t had a chance to analyze the data, except to rough up an adjacency matrix that I plotted in Rgraphviz (see the image above). But this particular visualization doesn’t say much to me (aside from “Rgraphviz == awesome!”), and I have lots of fiddling to do with the data before I can draw any conclusions.

To the extent that this stuff interests you, from the perspective of either analyzing the data or checking it for accuracy, I’m eager for input and advice. I’m sure I made some mistakes when I drew up the data, which took a long, long time, and I’m sure that there are many out there better at analyzing this sort of data than I am.

What I’d really like to know is whether and how seminaries cluster together in terms of their Bible professors. I got a M.A.R. degree at Yale Divinity School (where my Bible grades were awful, BTW), and I imagine that Dallas Theological Seminary-trained scholars would have a hard time getting a gig at Yale, and vice versa. If this is true, the data won’t say why — the reasons could be geographical, or ideological, or denominational, or of course theological — but I’m interested in seeing how the schools group. Can we see, for example, that people trained at “liberal” schools tend to teach at “liberal” schools, and the same for “conservatives”?

(Note that many of the numerous caveats about this data and its potential usefulness can be found in the document linked above.)

Your thoughts?

My two cents: What everyone wants to do is kick ass, so here’s how to do it — philosophically — in three simple steps.

1. State the philosopher’s position more clearly than he does

Philosophy is usually slow, careful, explicit argumentation, so if you want to argue with a philosopher you need to go down the long road of thinking through his position fully. It’s unphilosophical to argue with an oversimplification or parody of his position. Philosophers aren’t supposed to beat up straw men.

By stating the philosopher’s positions better than he does, you signal that your own argument engages his argument on the correct terms. You want the person you’re arguing with to agree that you can state his position with precision and sympathy.

The “more clearly than he does” part is a way of kicking his ass even before you start your refutation. You can do it nicely, of course.

2. Refute his position

He started from premises and, through a series of honest cognitive moves, reasoned his way to a conclusion. If you have a really good reason to believe that some element in this progression is wrong, and that removing that element causes the position to collapse like jenga blocks, here is where you state that reason.

Ideally, you’d find a bunch of contestable areas in the argument you just articulated. If one doesn’t stick, you’ve got another. Oh, and you should also state (and refute) plausible counterarguments to your own position before the other guy has a chance.

3. Clean up

At this point most likely he’ll either take exception to your argument or double back to refine his own in a way that undermines your argument.

But say you really do succeed in refuting his argument, even in his mind. He’ll do one of two things: he’ll either restate his argument in a progressively louder way (à la cable news), in which case you might want to consider kicking worthier asses in the future, or he’ll express his gratitude for getting set straight, in which case he’s a mensch and you owe him a beer or coffee to restore harmony in the universe.

Easy, right?

I said that these steps were simple, because they are. But they’re not at all easy. I’ll be honest with you: I’ve only done what I describe above a handful of times. I argue with people all the time, but the formal, careful, slow business of philosophical refutation eats up quite a lot of time and energy.

People who do philosophy are smart, and thorough. So if you have a subject that’s philosophically interesting to you, and if you want to do real philosophy with it, you need to be ready to put in some long hours. Philosophy, done right, hurts.

Philosophy hurts because of its demands on your attention

Doing philosophy makes brutal demands on your attention. Most of the reading is really, truly boring. From the writer’s perspective, it’s hard to be both thorough and engaging. Making it through most classic and contemporary philosophy books is almost an athletic event. It’s just so hard to stay focused, I don’t care how smart you are.

Most of what you read nowadays has been optimized for quick consumption. You’re probably not going to spend more than 90 seconds reading this, for example, so I have to think about how to break up the content so that you get the key points. Philosophers will not do this for you. Philosophy is a long, slow, painful slog.

Here’s how philosopher John Searle describes it.

[Doing philosophy is] murder, absolutely. I compare it … if you really want to know how to do it, you get up in the morning, there’s a large brick wall and you run your head against that brick wall. And you keep doing that every day until eventually you make a hole in the wall. That’s what it feels like.

Maybe you’ve read some philosophy that is dazzling and fun to read. Those qualities are more literary than philosophical. If you go back and read the same stuff as philosophy, you’re going to be back to the slog.

And lest we imagine that this practice is some sort of Western intellectual machismo, here’s the Dalai Lama on a related subject:

In Dharamsala, India, one of the Tibetans practicing concentrated meditation told me that the cultivation of one-pointed concentration was worse than imprisonment in a Chinese jail! (How to Practice, p. 127)

Formation of attention can be formation of character

Which leads us to the question. Why would you subject yourself to all this?

You might do it for a number of reasons. For example, you might have a burning passion for a topic in philosophy. If you care enough, you’ll fight your way through the dry reading (fighting against the authors, really) to get the insights it contains.

Or you could view the powers of attention that sustained study of philosophy requires as an end in itself. The faculty of attention is something you can exercise like a muscle and develop to high levels of function. Here’s William James on attention:

In some persons [attention] is highly focalized and concentrated, and the focal ideas predominate in determining association. In others we must suppose the margin to be brighter, and to be filled with something like meteoric showers of images, which strike into it at random, displacing the focal ideas, and carrying association in their own direction. Persons of the latter type find their attention wandering every minute, and must bring it back by a voluntary pull. The others sink into a subject of meditation deeply, and, when interrupted, are ‘lost’ for a moment before they come back to the outer world.

The possession of such a steady faculty of attention is unquestionably a great boon. Those who have it can work more rapidly, and with less nervous wear and tear.

James goes on to say that he doubts that attention can be cultivated in the way that I’m arguing it can.

And not only can it be cultivated, it must be cultivated, because it’s critical that people have the ability to apply profound faculties of attention to real problems. Strong analytic skills applied to worthy problems can improve society. Who do you want to see in the voting booth? The person who can think through issues thoughtfully and thoroughly, or the person who thinks in sound bites and talking points?

I’m not saying that reading philosophy will make you a better person. But you could use it that way if you were motivated to.

People won’t care that you’ve studied philosophy

Take note that the mere fact that you’ve studied philosophy is unlikely to impress people who have not. (In fact, I’ve studied philosophy, and others’ study of philosophy doesn’t impress me either unless I have reason to think that they are good at it.)

Unless they themselves studied philosophy, or unless they have had favorable experiences with philosophy majors, HR types do not value philosophy degrees.

I had a summer job before grad school where my boss, who was trying to determine my wage, asked me whether I had a degree in her company’s field, and I told her my degree was in the field because philosophy applied to everything. She didn’t see that I was half-joking and thought I was insulting her. Things went downhill from there.

This reality is why Scott Berkun shrewdly picked up a CS degree in addition to his philosophy coursework. Still, computer science, like a lot of applied-looking liberal arts majors like statistics and economics, can be done in a philosophical way, since all the big thinkers knew their philosophy. And you should study these fields philosophically, because doing so will make it less likely that you become a slave to their assumptions.

So while philosophy may not itself give you a body of knowledge that people will pay you for, it will cultivate a style of thinking that will always serve you well. I went into philosophy like a lot of teenagers, looking for answers to “meaning of life” questions, and I came out with some halfway decent answers to those questions but also with a profound and unexpected appreciation for intellectual discipline.

Now that I think about it, though, I probably shouldn’t have just told you how to kick my ass.

What is surprising to me is how uncommon it is that individual donors realize that finding out the debt of any nonprofit is as easy as clicking a few web links, and while the information about debt won’t by itself tell you whether to donate to a nonprofit, that information plus a few questions to board members can tell you whether to exclude the nonprofit from your donation choices very quickly.

And, if you like the idea of thinking of a nonprofit donation as being metaphorical to a for-profit investment, looking at this data can be a lot of fun. So let’s look at the data mentioned in the NYT article and then talk about its implications.

Take a look at the 990

The public document that shows a nonprofit’s balance sheet (among many other things) is called the 990, and my preferred site for finding 990s is GuideStar. So head over to GuideStar, do the required registration, and pull up the page for the Harlem School of the Arts. They’ve got three years of 990s, and if you want more you could always just ask the nonprofit.

There’s lots of interesting material in these 990s, but we’re thinking about debt, so take a peek at the liabilities on p.4 of the 2005 return.

(Click to enlarge)

The first column is the liabilities at the beginning of the statement period, and the second column is liabilities at the end. So from 7-1-05 to 6-30-06, a mortgage of over $300,000 disappears, and there’s an increase in “Other liabilities” of nearly $300,000. The Statement 7 mentioned other Other liabilities breaks that it into two not very illuminating categories.

So these changes merit some explanation, but before we go there let’s see what happened during fiscal year 2006 to liabilities.

Hmmm… not very encouraging. A $700,000 increase in “Other Liabilities.” That sounds like such a miscellaneous category, but evidently it’s not. And the 2007:

Well, the rate of increase of “Other liabilities” seems to be declining, but we’re still adding liabilities. How would the board explain this?

“Wait!” you might say. “They may have added a whole lot to their liabilities, but what about their assets? We might be more inclined to dismiss over $1 million in Other liabilities if their assets have increased by $1 billion!” Fair enough. Let’s skip back to p.1 and look at their net assets. Here’s 2005:

An operating deficit, but they still have assets. 2006:

That deficit’s getting bigger. 2007:

Uh-oh.

The nonprofit may not volunteer this information to you

Let’s imagine how a similar but fictional assessment might play out in the decision to donate.

You’re out to lunch with reps from a nonprofit with big debts, even though you’ve not yet discussed them. The director of development called a few months ago to ask you to increase your annual gift from $100 to $1000, and you sent in $5000. He queried your name in a database and found out that you’re (ahem) doing well. Then he went to the CEO with this information, who passed it along to the board.

Two board members know you, and everyone decided to make a move, so now you’re dining with all four of them. They want you to donate $100,000, which they privately see as the next step to an even bigger donation.

So what about these big liabilities? Before you met with these people, you did well to check the 990s, because it’s by no means assured that they’ll tell you about the liabilities. They’re thinking about them — boy, are they thinking about them — and their grantmakers know about them, because grantmakers are in the business of evaluating this information and acting on it quickly.

You disclose that you know about the debt. They say that times are tough, which is true, and that tough times for the economy mean tough times for nonprofits, which is also true. How sympathetic are you feeling at this point?

It can be ok to donate to a nonprofit in debt, IF…

There are certainly good uses of debt (I’m about to buy some coffee using debt, and I feel great about it, e.g.). But just to push this example to its extreme let’s imagine that the debt was brought about for bad reasons. Someone embezzled money, someone bought something stupid, someone made a mistake that required $1 million to clean up… whatever.

The four people you’re dining with — or rather, the two board members, because they’re the ones who really count — can take one of two approaches to responding to the debt, and their decision in this regard will determine whether the debt should disqualify the nonprofit from your donation consideration.

In the worst case, they’ll pretend the debt doesn’t exist. They’ll be able to get away with this as long as their cash position enables them to make payments and as long as their bank doesn’t call the loan. If the debt is collateralized, that’ll be a help. You might see board members pretending in this way if the debt was incurred by the predecessors of current board members and the current board doesn’t psychologically “own” the problem. Why should they bend over backwards to erase the debt? The debt is someone else’s fault!

In the best case, the board members will own the debt psychologically regardless of who incurred it or why. You want to look across the table and in your gut assessment see people who can discuss the nonprofit’s situation honestly, who (you believe) will stop at nothing to get rid of the debt, and who have made major personal commitments (relative to their own wealth) to pay down the debt. If the key players have the motivation and credibility to get the nonprofit back on the right track, then the debt alone shouldn’t disqualify the nonprofit from your giving decision.

If you think they’d sooner shutter the nonprofit than take the tough steps to fix it, then you certainly shouldn’t donate.

For board members: debt depresses fundraising receipts

I’ve seen plenty of board member take the worst case approach. If pressed on the subject of the debt, their response is typically that it’ll go away if management can create incremental operating revenue. And this can be true if the size of the debt is small. I hope, for example, that I can make the money this month that will enable me to erase the debt I’ve incurred for this exquisite cup of coffee.

Boards who take this quixotic approach will tend to go through CEOs pretty quickly. Either the CEO will fail to achieve the revenue gains required and get fired, or the CEO will courageously press the board to fulfill its responsibilities and get fired. Board members like those at the Harlem School of the Arts should know that they actually look better if they admit failure and take steps to rectify it than if they try to scapegoat the people below them after everything has fallen apart.

But perhaps the most important thing board members should know is this: just as they are not bending over backwards to disclose to donors that their gifts service debt, donors are not bending over backwards to disclose to them that their paltry contributions are a function of the nonprofit’s debt.

Want to increase revenue? Get rid of your debt and ride the wave of incremental revenue created by the increased confidence of sophisticated donors who pay attention.

If it’s six figure debt, ok, maybe an enterprising CEO can make it go away without your help. Heck, you might even be able to pass 5-figure liability down to a director of development. But if you’re seven figures in the hole, it’s exceedingly unlikely that your CEO will be successful at solving your debt problem without a credible public commitment from you.

For board members: it’s all on you

My belief is that, if a nonprofit incurs a debt while you’re a board member, you’re responsible for ensuring that it’s paid off. I believe that this obligation follows you even if you leave the board.

I also believe that, if a nonprofit incurs debt before your time, the debt becomes your responsibility once you join the board. To join the board of a nonprofit is to commit publicly to ensuring its financial well-being. If you can’t commit to paying off the debt out of your own pocket and out of funds that you raise, you shouldn’t sign up.

Ok, enough moralizing…

Here are the slides, which include web links, and here is the data. Stay on the lookout for a video of the presentation itself, which should be up soon.

(These slides correct my description of tabular data, which I’d called “2×2″ when I meant “2D”.) Thank you for the comments!

To this person I’d strongly recommend that you extend your Excel repertoire beyond list-making, highlighting, and the occasional SUM formula. My experience showing people how to use Excel suggests that for every Excel power user there are a hundred people using Excel in a minimal way. Excel has plenty of firepower, and you paid for it, so why not learn it?

You should also learn R (more specifically, the S language). It’s not the easiest language to pick up, but once you get a hang of the basic data structures, the data analytic floodgates open. Since I started using R over a year ago, data analysis has become a much more exciting, “bwa-ha-ha!”-inducing experience than it ever was before. The power of R is astonishing. I haven’t been this excited about doing data work since I printed out SuperCalc bar charts on my Epson 9-pin dot matrix printer. If you haven’t already, you should learn R.

(I know what you may be thinking. “Guess what? Michael’s answer to life’s problems is… to buy what Michael’s selling! Oh boy.” Well, touché.)

Ok, so let’s look at the specific question. Here are a few suggestions.

When to use Excel

When you have something that needs a nice presentation. Most people use Excel as a page layout program for quantitative or list-based data. Seriously, as a page layout program, like InDesign or something.

This is probably not the primary usage that the authors of VisiCalc had in mind, but it’s a big need that people have, and the accretion of new formatting features in Excel 2007 and 2010 shows that the good folks at Microsoft recognize that their job is to give people what they want. Excel is a fast and straightforward tool for the presentation of tabular information. It would be the first tool I’d use if I wanted to present a summary of data, and I’ve even dropped graphics created in R into an Excel spreadsheet (which I then ripped to a PDF) when I wanted to create a nice presentation.

Now, people can use Excel for data presentation either well or poorly. I know that the Head First-approved Non-Designer’s Design Book was a big help for me when it comes to recognizing when I’m creating something hideous in Excel. I recommend it.

When you have quick and dirty number crunching to do. With Excel, loading data and writing formulas is quick and easy. With R, there’s generally some configuration overhead you have to endure in order to start crunching numbers. If you need to do a small handful of descriptive stats on your data, or you need to look something up, run a quick sort/filter, or even a pivot table, Excel is the tool.

Some people never need to go beyond this sort of data work. They probably don’t need to learn R, even though I’m inclined to say that everyone needs to learn R.

When I want to eyeball data quickly and maybe run a few basic formulas, I’m happy to fire up Excel instead of R.

When to use R

When you have to explore data. At the start of an analytic project, it’s a good idea to create a bunch of graphical visualizations of your data to get a sense of what’s inside it. In terms of its graphical capabilities, R exists in a whole separate dimension from Excel. This was perhaps the most shocking part to me about using R for the first time: I really thought I had a handle on data analysis even though I’d restricted my software to Excel, but boy was I wrong. The visualizations you can create in R are much more sophisticated and much more nuanced. And, philosophically, you can tell that the visualization tools in R were created by people more interested in good thinking about data than about beautiful presentation. (The result, ironically, is a much more beautiful presentation, IMHO.)

Here’s how I’d put the difference to someone who’s familiar with Excel but not yet with R. The graphics creation options that Excel gives you are all based in the graphical user interface. This is what makes Excel relatively easy to use—all your options are laid out before you with nice buttons and fill-in-the-blank boxes. But in order to create a graphical interface that’s easy to use, the creators of Excel had to make a bunch of decisions about what sorts of graphics you are and are not likely to want. With too many choices, the graphical interface becomes cumbersome and frustrating, so to achieve simplicity they had to eliminate options.

And this isn’t a gripe or anything. I can’t say I’d have done a better job designing Excel’s charting graphical interface. I cut my teeth on it.

These limitations become a problem when you want to inspect data visually in a bunch of different ways in order to explore it. R, through a combination of its well-designed base graphics package, the exceptionally well-designed lattice graphics package, and the jaw-droppingly well-designed ggplot2 graphics package, offers a breathtaking array of visualization options that you access through the command line or scripts. It has power that you just can’t get using a graphical interface to generate your charts.

When you need to be really clear about how you change your data. Setting aside the cool bells and whistles of R, this particular angle has had the most practical significance for me. A lot of the work that I do as a consultant involves direct marketing data. I set up the data either for analysis or for print/web production, and this means that I have to mutate it quite a bit from its original form. It’s rare that I can take client data in the state they store it and use it directly without manipulation. I say “rare,” but it’s really “never.”

The way I’ve always handled the cleaning of data in Excel is to create a bunch of intermediary formulaic columns alongside my raw data columns. I work and rework these intermediaries until I finally have columns containing data that’s been “cleaned” for whatever purpose I have in mind. Then I copy and Paste Special > Values the clean data to a new sheet. That way, I have an audit of what I did to the data, in case I screwed something up, which never happens [COUGH!].

With R all the data mutation I do is now saved in little text files called scripts. I’ll have my raw data in a CSV or something and create a script that loads the data, mutates it in any way I want, then spits it out into another CSV and/or R object. The advantage of using R in this way rather than Excel is that I can be a lot more concise in terms of code and descriptive in terms of commenting about what I do. Another advantage is that I can use regular expressions in R, which one cannot use in Excel. R is worth learning so you can clean data more elegantly.

I think of the raw data > script > clean data workflow in R as similar to the RAW image > metadata manipulation > JPG image workflow in Adobe Lightroom, for those of you familiar with Lightroom. It’s nice.

When you need serious statistical capabilities. Excel has a bad reputation for statistics. Historically there have been a variety of situations where Excel has demonstrated numerical and programmatic errors that produce flat wrong answers. I haven’t seen a statistician’s review of Excel for version 2007 or later, though, so I can’t really pass judgment about Excel’s current incarnations. Let me know if you have a link to something along those lines.

Excel 2007 was, generally speaking, a big improvement over previous versions. I’m a big fan of the SUMIFS and IFERROR functions, and I really appreciate the “Big Grid” and multiprocessor support. And all indications are that Excel 2010 will be another big improvement—check out the Excel team’s very interesting blog. My impression is that the Microsoft folks are sensitive to historical problems and have taken steps to fix them. If so, this is worth applauding.

But I’ll make a last point that is really a restatement of the point I made about the overdetermination of options you get with a graphical interface. This phenomenon applies not just to graphics functions, but to statistical functions generally. The statistical functions you get in R are much more flexible, numerous, and reliable. By a very long shot. This is in large part because R uses a full-blown scripting language rather than a GUI.

Conclusion

I’m quite sure I haven’t hit everything, but these distinctions between the two programs are where I’d start for someone with familiarity with Excel and no knowledge of R. Become a formula whiz in Excel, and learn R!