Hard problems and the Temple of Complexity
Some might interpret my ranting against enterprisey astronauts as denying the existence of truly hard problems in the world. This is the farthest from the truth. There are seriously hard problems. A hard problem to me is one whose solution might fall into one of the following categories:
- P or NP-complete
- Analysis of huge (terabytes) of data in real-time
- Stream prediction
- Provable systems
- Trusted systems (see provable)
Collecting customer data, sales information and offering a few pretty pie charts is not a “hard problem.” It has a lot of pieces, and it might be complex if you let it become that, but it’s not a hard problem. There are no hidden Markov models predicting customer behavior. There is no link analysis to determine latent connection. There’s just some tables, some presentation and a tiny “wafer thin” controller layer.
Hard problems require leaps of intellectual capacity that 99% of the population is incapable of—including myself. Hard problems challenge us as human beings. We devalue the truly difficult by pretending, for the benefit of our own egos, that what we do is hard. So why do we do this?
I call it the Temple of Complexity: a religion so ingrained in the IT industry that it has slipped from a conscious behavior to a subconscious assumption. We are programmed by the industry—and society to a lesser extent—to proclaim our littlest accomplishments as comparable to the great works of the world. I witness this every day when people sigh and tell me how much work something is going to take. It’s not a lot of work, but our nature says we want to believe what we do is “real work.”
There is a saying1 that I use repeatedly with my coworkers: “With infinite flexibility comes infinite complexity.” The complexity of a “solution” becomes asymptotic as you approach total flexibility. Learning where to stop on that curve before it becomes unmanageable is the art that requires self-observation, a keen eye and a perspective that questions the conventional wisdom.
I am continually amazed at the obsession with the complex that I witness in my day-to-day experiences. People will drag an entire project into the weeds to examine, in excruciating detail, a feature that perhaps 1% of our customer base might use, some time, if they discover it and don’t get fed up with the rest of the product first. The question is, to me, can they accomplish their job—their real business goal—without that feature? If so, then it isn’t important and should be removed from the system until we reach a point where it enables some new capability hither-to unavailable in the system.
How many features are in Word? Windows? MacOS X even? Do you even know? Do the programmers even know? The argument that “we need to give people options” is the tacit admission that “we have no idea what people need.” Rather than step back and examine the situation—or better yet get something out there so you can get real feedback—people just pile on the feature set, pile on the complexity, and watch their timelines slip into the future.
Let me, for a moment, look at a specific category of software that someone brought up as “enterprise problems,” and discuss the complete and total systemic failure of it as a solution to the underlying business problem: Customer Relationship Management—CRM.
I have worked with 3 major packages (all of which equally inadequate and so unworthy of mention), which represent I would suspect 80% of the “enterprise CRM” market. I have worked with some of them in multiple deployments at Fortune 500 companies. In each case, without exception, they have been: over budget (sometimes by a factor of 10), geologically behind schedule, and when measured against actual benefits delivered to the company, woefully underperforming. Tends of millions are spent, and little value is delivered. Thousands of features are delivered, a tiny handful are actually used.
Why is this? Why do companies spend millions of dollars on CRM “solutions” that solve nothing except how a consulting firm is going to hit their profit goals? Their drivers are inverted from their natural order. For example, the sales cycle component is focused around the reporting needs of management, and the sales individual is saddled with a burdensome process that doesn’t make them more money, and money is what drives sales people. So you get people who fake the numbers, put them in 2 hours before their manager is going to use them, and use a swag at best to manage thing. Why should they do anything different? They get zero value from the system. That’s enterprisey at its most pure form.
Complexity isn’t power. Clarity and simplicity are power. Power is having the right feature for the right person, not 100 features. Knowing what that one feature is requires a discipline that is anathema to what we are taught. It requires saying “no” to yourself, your coworkers and to others. It requires the removal of the facade of complexity and difficulty that we hide behind, both to make ourselves feel better and to absolve ourselves of the risk of failure. Only when we remove that pretension can we begin to understand the real problem.
1 I stole this from Spiderman, so shoot me.
This entry was posted at 11:14 pm on 14 April 2006 and is filed under Technology. You can follow any responses to this entry through the post-specific RSS 2.0 feed.
Let’s tag this one k-rad k00l.
The easist problems are:—> ones that have been solved for already—> easy to write down on one piece of paper—> correcting past mistakes—> utility based in nature
The hardest problems:—> the most difficult to put into word—> subject to the most highly unique aspects of the customer —> attempt to solve for people power (aka “wetware”
There are a staggeringly few number of “problems” in the IT world that haven’t been solved repeatedly—some even tens of thousands of times. This is why I argue that the laughable notion that they are “hard problems” is born more by a desire to feel important than by any real innate difficulty in the problem itself.
There are hard problems, but we drain our resources and intellectual capital into many places where we do nothing truly hard. Often things become hard because we attempt to make everyone happy. Such things, bred of strange psychosis in the human animal, are disaster to accomplishing anything.
I agree on all counts and rather enjoyed this article, but you slipped a really embarrassing little mistake in there:
P or NP complete
There is no such thing as “P complete.� There is only “NP-complete,� which is the name of a special class of isomorphic NP problems. I was going to break it down a little more here, but then I noticed you would be better served to just look up the Wikipedia articles on NP-complete and complexity theory.
Another dimension of complexity is the changing nature of requirements. Being asked to build something is generally not “hard”. But when they change their mind six months in, then nine months, and so on, you have a hard puzzle on your hands. Keep building on the original (and now inadequate) foundation? Redo the entire foundation? Redo the foundation with even additional flexibility in expectation of further requirements changes? Redo part and add a mapping layer??
Both comments and pings are currently closed.
Funny how Ruby or rails can’t address them..
Welcome to THE Real World….
Mr. Know-no-thing™ sy
What are those tags? Is’nt it just BALONEY?
Why can’t I tag your blogs….
Who’s tagging your blogs????? Why?