Informed Skeptics

The current issue of Harvard Business Review contains an article by Shvetank Shah, Andrew Horne, and Jaime Capella, entitled, "Good Data Won't Guarantee Good Decisions" (23-25). The authors argue that the era of Big Data is leading to a false sense of confidence in decision-making. "Investments in analytics can be useless, even harmful, unless employees can incorporate that data into complex decision making. Our research offers a succinct warning to managers. At this very moment, there's an odds-on chance that someone in your organization is making a poor decision on the basis of information that was enormously expensive to collect" (23).

So, who is making decisions in a given organization, and what is their ability "to find and analyze relevant information"? The authors evaluted 5,000 employees at 22 global companies and sorted them into three groups:

• unquestioning empiricists -- those who trust analysis over judgment
• visceral decision makers -- those who go with their gut, exclusively
• informed skeptics -- those who balance judgment and analysis, possess strong analytic skills, and listen to others' opinions but are willing to dissent

As they point out, the informed skeptics are the folks that companies (schools??!!) should be cultivating. However, as they discovered in their evaluation, only 38% of employees and 50% of senior managers fall into this group.

Interesting (to me, at any rate) also are the four problems they identify that "prevent organizations from realizing better returns on their investment in Big Data":

1. Analytic skills are concentrated in too few employees -- in other words, "too many companies [schools??!!] are stuck in the 'expert phase'; they have a handful of highly skilled analtyics professionals but have not begun to train everyone else to make use [of analytics].
2. IT needs to spend more time on the "I" and less time on the "T" -- so many departments have "diverse data demands or may need to use data in ways they can't clearly articulate. Meeting these challenges requires anthropological skills and behavioral understanding--traits that are often in short supply in IT departments."
3. Reliable information exists, but it's hard to locate. The analogy: a library with no card catalog (print or digital), or perhaps with books that lack covers. What is more, "the rise in social media [...] has made it even harder to manage analytic content. Fewer than 44% of employees say they know where to find the information they need for their day-to-day work."
4. Business executives don't manage information as well as they manage talent, capital, and brand. In short, they "consider themselves too inexpert to get deeply involved in how data is shared across the organization. Managers need to wake up to the fact that their data investments are providing limited returns because their organization is under-invested in understanding the information."

The obvious solution, of course, is to develop more informed skeptics. How might one do that, especially in independent schools? First, though, I would ask this: what data are we using? What data ought we to be using? Then we can deal with the question of how to develop informed skeptics. Regardless of the answer(s), though, we can move forward with one certainty: we do need to train our faculties and administrators to increase their data literacy, with school leadership showing the way. That data then needs to be brought into decision making, so that folks can see how data can inform decisions.

We all need to understand the factors and calculations behind the numbers, and we must learn to think critically about their accuracy, sample sizes, biases, and, of course, quality. (Statistics, anyone??) Coaching could play an important role here, as could project-based learning; we say that we want to see our students do these things, but why shouldn't we model it for them?