My favorite point that he makes is that men feel powerless at work, on the roads, really everywhere these days. So it makes sense that they'd look for power in other places, even in a fantasy role. Hence, the rabid "fanatic" that many of these men become when watching their favorite sports team.
Most guys have demanding bosses and nagging wives, with the majority of these men falling short in life. But when their team wins, it allows them to "win in life" if even just for one day.
I admit myself to playing fantasy baseball, although I play for money. It gives me the sense of placing myself in the GM's position of a team, something I never will be able to actually experience myself. I can buy low and sell high on players (much like stocks) and have the same feeling as being a successful investor. This hobby is something that I can't give up. Even for me, sports has become too essential here in the US to completely ignore it.
I do think though that removing yourself from a fanbase is liberating. Even though I still watch football to appreciate the skills of certain players and keep tabs on my "stocks", I am no longer on the emotional rollercoaster during games. This at least allows me to appreciate skills, without actually idolizing anyone or any team.