So, your points are well-taken, of course, and you would think a coach who pursued a win-maximizing strategy would be duly rewarded. However, there are several things that must also be considered.
The first is that coaches are humans, not automatons. The coaches are making dozens of high-stakes decisions during the course of a football game, and decision fatigue is a well-characterized phenomenon (
https://en.wikipedia.org/wiki/Decision_fatigue). At the end of a night of game-changing decisions large and small, coaches will default away to easily available heuristics, one of which is another common human cognitive bias: preserving future choice. A coach deciding between a win-maximizing choice to go for it on fourth and short on their own side of the field late in the game while trailing, and a punt that allows them to believe that if they force the other team to go three and out they can get the ball back with a minute left, will usually elect not to pursue the option that could result in the game being effectively over. The widespread prevalence of these poor decisions (at one point in time, ESPN's Tuesday Morning Quarterback had a nearly weekly segment about stupid punting decisions) suggests that this isn't isolated to a few particularly bad coaches, and, in fact, it is the rare coach who breaks from the pack. Add to that the fact that historically, coaches who took the conservative choice were rarely criticized for it, while those who took the aggressive choice and failed were, and for almost all of the history of football, the "wrong" choice was rewarded. I think that has started to change with the widespread use of analytics, but old habits die hard. I do expect to see an increase in the "more correct" strategy being employed going forward, as the next generation of coaches who is more facile with analytics comes to the fore.
I can't remember which coach it was, but I remember reading an article that one NFL coach had embraced analytics and had near real-time analysis provided by an analytics expert to supplement his decision making at key points in the game. He said he didn't always default to what the computer said, but he said that it helped him a lot in his decision-making, especially in situations where his gut instinct and the computer diverged, by forcing him to question his assumptions. I would love to see Stanford adopt it, but for all the positives I think of Shaw, this appears to be anathema to him. You have to take the occasional bad punting decision with all the good he brings to the program.
BC