Moneyball: What’s the Real Problem?
Ken Medlock in Moneyball, courtesy of Columbia, a Sony Pictures Entertainment company.
I recently read an article on Medium, entitled, “Nike’s $25B blunder shows us the limits of “data-driven,” by Pavel Samsonov. It’s a brilliant treatment of the lazy (yes, I used this term specifically) use of data. But even more telling, was a clear and succinct response to the article by a Medium member, “Anopheles.” I quote…
There is a very famous example from WW2 of protecting bombers from enemy fighters and anti aircraft [sic] fire.
When the bombers returned, they mapped out all the bullet holes. So, what do you think the conclusion is? To protect those areas, right?
Wrong. They tried that and it doesn’t work.
The correct answer is to protect the areas without bullet holes. Because the planes which returned with bullet holes meant those areas are not the critical. They can be shot and still get home. Those planes were not shot in critical areas.
This is the counterintuitive thinking which the majority of people get wrong. In this example it’s called Survivorship Bias. https://en.wikipedia.org/wiki/Survivorship_bias
I thought this quite fascinating, and it made me think of the film Moneyball and the fourth thing I learned about organizational excellence from the film.
Moneyball
In Moneyball, a cinematic depiction of the book of the same name by author Michael Lewis, at 02:05 minutes into the film, the Oakland A’s recruiting staff are trying to parse together replacements for their three top players, who left for bigger contracts. Brad Pitt’s character, Billy Beane, is portrayed as clearly impatient and frustrated with the status quo that is being evidenced in the discussion. He later says, “You guys are just talking… like this is business as usual.”
The head recruiter - Grady Fuson, played by Ken Medlock - responds that they are trying to solve the problem. To which Billy Beane says they are not even looking at the problem. He then asks several of the recruiters in the room what the problem is. Beane gets a few varied responses of the same idea, which is that the problem is replacing the three team players lost, with at least three players that can be developed to equivalent competence of those now gone.
Unhappy with the three responses Beane gives a guttural sound like a buzzer and says, “There are rich teams and there are poor teams. Then there’s 50 feet of crap. Then there’s us. It’s an unfair game. And now we’ve been gutted, organ donors for the rich. Boston’s taken our kidneys, the Yankees have taken our heart, and you guys sitting around talking the same old good body nonsense like we’re selling jeans, like we’re looking for Fabio. We’ve got to think differently. We are the last dog at the bowl. You see what happens to the runt of the litter? He dies.”
What follows is a dialogue by Grady Fuson, stating the recruiters are well aware of the problem. Beane then picks some of the players discussed as replacement candidates and asks if they are up to par with the players they lost. All agree that they are not, to which Beane responds, and I think this is brilliant,
If we try to play like the Yankees in here, we will lose to the Yankees out there.
Through this whole scene, Beane is trying to get his recruiters to understand, “What’s the real problem?” And his statement quoted just above, is the definition of the real problem they faced.
Business Application
In business, particularly as it relates to organizational excellence, we have to be willing to ask, “What’s the real problem?” Jim Collins, in his book, Good To Great, talks about knowing and facing the hard reality. Mike Simons, VP Manufacturing Operations at Bullfrog Spas, has the whole organization looking for the “ugly.” If you don’t know what is ugly in or organization, you can’t remove or improve it. I did a lot of work in South America last year and they taught me about watermelon metrics. This is where green is shown and red avoided in the metrics reviews because green is perceived as being good, like a whole watermelon might look good. But we have to be willing to get to the fruit, the red, because that is where we can really enjoy the watermelon, or in real terms, organizational excellence. There is a common lean saying, “there’s gold in the red,” as it implies the elimination, or at least the reduction of waste. But you have to be willing to ask, “What’s the real problem?” to get there.
The Doctor, The Priest and The Demographer
Earlier this year at the 36th Annual Shingo Conference, I was privileged to hear Tim Costello speak about data, and really understanding what the data is telling you; understanding “What’s the Real Problem.” In his presentation, “Lessons From: [the] Doctor [,the] Priest [and the] Demographer” he outlines the learnings from London’s Broad Street Cholera Outbreak. London was in transition from an agrarian society to the new industrial age. There was a mass migration to the city for better paying jobs, to the tune of 28,000 residents per square mile. This sets the stage for the perplexing London cholera outbreak centered around Broad Street in the Soho district of Londan.
Conventional Wisdom
The conventional wisdom of the medical profession of the day claimed it was “bad air” or miasma. Physician John Snow, an early founder of epidemiology and modern germ theory, was highly skeptical of claims of miasma by the medical establishment. John Snow documented the locations of the outbreak and with the help of a priest named Henry Whitehead, together they began to interview residents located in outbreak clusters, with Snow creating a dot map of cholera incidents, particularly fatalities from the disease. As a note, this is one of the very first documented uses of data visualization, as an aid to scientific methods.
John Snow postulated the cause as the Broad Street water pump, given the dot map indicated the pump was in the center of the data. Given this evidence, he was laughed to scorn by his medical peers, who persisted with the conventional wisdom miasma was the cause of the outbreak. There was no physical evidence this was the case, but rather the collective thought of the medical community at the time. They argued that it could not be the pump because of the outlier clusters on the dot map.
Sometime later, Henry Whitehead began interviewing families of the deceased and found some interesting information with respect to those outliers. A well-known and favored bakery had someone go to the Broad Street pump to collect water daily for use in the bread that was made and sold. Because the water was not from the bakery locale, and considered precious, those employed at the bakery drank water from the pump closest their homes. No one working in the bakery died of cholera, but many of its customers did.
Looking Beyond the Conventional Wisdom
In another closer look at outliers, a woman who had once lived near the broad street pump deemed it the best water in the city. Her son would walk the distance to the Broad Street pump out of love and respect for his mother. Rather than drink the water from the Broad Street pump, to minimize his trips, he too, would drink only from the local well. His mother passed from Cholera, but he did not.
There are several curious stories of outlier cholera clusters, that challenged the conventional wisdom. In 1855, John Snow published his findings in a scientific journal and again, was laughed to scorn.
19th century epidemiologist William Farr, a more aged contemporary of John Snow mused that it was not only the Broad Street pump as a source of cholera, but also water pumps in densely populated areas. Farr produced his Bills of Mortality outlining cholera fatality around these pumps. He too was pilloried, and never found acceptance of his findings until after his passing shortly thereafter.
In 1866, another major cholera outbreak took place and at this point, public opinion started challenging the conventional wisdom that it was all due to miasma. The work of John Snow, Henry Whitehead and William Far was finally accepted. Further investigation revealed that these wells in densely populated areas all had some sort of fecal contamination, causing the vibrio cholera bacteria to thrive and contaminate the water supply.
No one of the established medical profession of the day, was humble enough to ask, when the data was shown, “What’s the real problem?”
Freeze Dry Cakes & Stainless Steel
Years ago, while working in the pharmaceutical industry, I was trained in formal root cause analysis (RCA), following the pattern used by the FAA (United States Federal Aviation Administration). Once I was assigned as lead investigator of a particularly perplexing issue involving a small gray disk on top of a freeze dry cake.
To provide context, twelve of my fifteen years in the pharmaceutical business was supporting parenteral manufacturing operations, or the manufacture of injectable drugs. The specific drugs I supported were primarily oncolytic, or drugs for cancer. Some of these drugs are unstable - i.e., short shelf life - in solution, so they are freeze dried. A small amount of a sugar such as lactose, is introduced to the drug solution and when the drug is freeze dried, all the water in the solution is gone and what is left is a fragile crystalline matrix of the sugar with the drug product attached. The result is called a freeze dry cake, which is bright and stark white in appearance. This is reconstituted at the point of use for chemotherapy.
Conventional Wisdom & Biases
We had some freeze-dried drug product that appeared with a small gray disk sitting in the center, atop the freeze dry cake. Everyone was very perplexed. After being assigned as lead investigator for the RCA, I was able to gather a team of some rather brilliant, people, scientists, engineers, etc. We had a kickoff meeting for the investigation and the site quality head attended. After initial comments to the team and stakeholders, followed by some Q&A, the site quality head asked if he could deliver remarks to the investigative team. I yielded and his remarks to the team were, “Go after the stopper maker. I know it’s them because we have had nothing but trouble with them.” I was a bit surprised as was everyone else in the room. I responded, “We will let the investigation take us where it takes us. If it is the stopper vendor, then that is where the investigation takes us. But we will remain true to the investigative process.” In other, we wanted to know, “What’s the real problem?” My response was supported by the investigative team but was not well received by the site quality head.
Seeking to know What’s the Real Problem
After some intense efforts at discovery, we produced mounds of data. The equipment used to fill and stopper the vials was new and completely scrutinized. We learned the stainless-steel particulates were small enough they could not break the surface tension of the solution in the vial. We learned that (316L) stainless steel can actually be polarized if it is subject to constant mechanical action (rubbing), even by stainless steel of the same exact type; that the polarization of these tiny particulates caused them to join together in an agglomeration creating a disk, not visible to the naked eye atop the solution in the vial, but clearly visible atop the white freeze dry cake after lyophilization. We learned that these tiny, nearly microscopic bits of stainless steel somehow found themselves in the filling suite atmosphere and were eventually pushed down into the vials via the hyper clean filtered air drafting over the vial filling process. But “What’s the real problem?”
We learned the operations team filling the vials and adding the stoppers had problems with the new filling and stoppering equipment, resulting in a number of maintenance calls. We could not however, determine what the cause was. We decided that all of our mountains of data educated us on ancillary facts to the problem, but we still needed more information, so we went to the gemba and interviewed the maintenance folks. During one particular interview, we learned a well meaning maintenance person, observed that the stoppers were not running well because they were getting snagged on a guide plate the stoppers traveled through. The maintenance person created a shim to prevent the stoppers catching and getting stuck. Unfortunately, the shim - also made of stainless steel - wound up rubbing against a lateral drive disk for the stoppers, also made of the same grade stainless steel. This was what was causing the tiny particulates that eventually became the gray disk atop the freeze dry cake.
Knowing the Real Problem Begets Real Solutions
We further learned that when the equipment was installed no one thought to identify the zero-base setting for the equipment before starting a production run. As a result, the operators were making adjustments on top of the adjustments by the operators before them. This happened for several sequential production runs, which led the maintenance person to come up with the shim idea. We had identified the real problem, but the bias of the site quality head had been to focus on the stopper supplier. Had we gone down that path, we would have come to some sort of pseudo-root cause, further estranged an already tenuous supplier relationship, and we would have continued seeing gray disks atop freeze dry cakes.
Because we focused on learning, “What’s the real problem?” we wound up identifying the cause and setting up zero-base settings that all the new equipment had to be set at before beginning a production run. When we understood, “What’s the real problem?” we were able to have a meaningful investigation that got to true root cause.
As a side bar, there were things identified during the investigation that led to another investigation I was asked to be lead investigator for, in collaboration with the stopper supplier, and the bag maker for the stoppers, to identify the cause of particulates that were associated with the stoppers. A different story and discussion altogether.
Conclusion
Don’t be satisfied with what others may think is the problem. It will have huge impact on the quality of your solution. I once did work for a small pharmaceutical company with a novel delivery method for a common drug product. They had constant repeat deviations. When I started looking at the repeat deviations, I realized that they did not know, “What’s the real problem?” Make knowing this a focus of what you do, and your outcomes will be the best possible and you will be contributing significantly to the excellence of your organization.
I very much appreciate you and the time you have taken to read this article. You can find more articles like this from me at https://www.legup.solutions/blog.
If you have thoughts on this or other topics regarding yours or your organization’s journey of excellence, feel free to continue the conversation on my Secret Sauce slack feed.
Originally published at https://www.legup.solutions on 20 AUG 2024