29. Black Markets Are Not Efficient – The Software Architect Elevator

Chapter 29. Black Markets Are Not Efficient

But They Reveal How Things Actually Get Done

I got anythin’ you need, bro

A common complaint about large organizations is that they are slow and mired in processes that are designed to exert control (Chapter 27) as opposed to supporting people in getting their work done quickly. For example, I used to be allowed to make technical decisions involving tens of millions of dollars, but I had to obtain management approval to purchase a $200 plane ticket. By the time I got the approval, often the fare had increased.

Most organizations consider such processes as crucial to keeping the organization running smoothly. “What would happen if everyone did what they wanted?” is the common justification. Most organizations never dare to find out, not because they fear chaos and mayhem, but because they fear that everything will be fine, and the people creating and administering the processes will no longer be needed.

Black Markets to the Rescue

Ironically, beneath the covers of law and order, such organizations are intrinsically aware that their processes hinder progress. That’s why these organizations tolerate a “black market” where things get done quickly and informally without following the self-imposed rules. Such black markets often take the innocuous form of needing to “know who to talk to” to get something done quickly. You need a server urgently? Instead of following the standard process, you call your buddy who can “pull a few strings.” Setting up an official “priority order” process, usually for a higher price, is fine. Bypassing the process to get special favors for those who are well connected is a black market.

Note

If the answer to “how long does it take to get a server?” is “it depends on who’s asking,” then you have a black market.

Another type of black market can originate from “high up.” While it’s not uncommon to offer different service levels, including “VIP support,” providing senior executives with support that ignores the very process- or security-related constraints imposed by the executives in the first place is a black market. Such a black market appears, for example, in the form of executives sporting sexy mobile devices that are deemed too insecure for employees, notwithstanding the fact that executives’ devices often contain the most sensitive data.

Black Markets Are Rarely Efficient

What these examples have in common is that they are based on unwritten rules and undocumented, or sometimes secret, relationships. That’s why black markets are rarely efficient, as you can see from countries where black markets constitute a major portion of the economy: black markets are difficult to control and deprive the government of much-needed tax income. They also tend to circumvent balanced allocation of resources: those with access to the black market will be able to obtain goods or favors that others cannot. Black markets therefore stifle economic development because they don’t provide broad and equal access to resources. This is true for countries as much as large enterprises.

Note

Black markets stifle innovation because they don’t provide equal access to resources. The digital world democratizes access, which is exactly the opposite.

In organizations, black markets often contribute to slow chaos (Chapter 31), in which the outside of the organization appears to be disciplined and structured, but the reality is completely different. They also make it difficult for new members of the organization to gain traction because they lack connections into the black market, presenting one way systems resist change (Chapter 10).

Black markets also cause inefficiency by forcing employees to learn the black-market system. Knowing how to work the black market is undocumented organizational knowledge that’s unique to the organization. The time it takes employees to learn the black market doesn’t benefit the organization and presents a real but rarely measured cost. Once acquired, the knowledge doesn’t benefit the employee either, because it has no market value outside of the organization. Ironically, this effect can contribute to large organizations tolerating black markets: it aids employee retention because much of their knowledge consists of undocumented processes, special vocabulary, and black-market structures, which ties them to the organization.

Worse yet, black markets break necessary feedback cycles: if procuring a server is too slow to compete in the digital world, the organization must resolve the issue and speed up that process. Circumventing it in a black-market fashion gives management a false sense of security, which often goes along with fabricated heroism: “I knew we could get it done in two days.” Amazon can get it done in a few minutes for a hundred thousand customers. The digital transformation is driven by democratization; that is, giving everyone rapid access to resources. That’s exactly the opposite of what a black market does.

You Cannot Outsource a Black Market

Another very costly limitation of black markets is that they cannot be outsourced. Large organizations tend to outsource commodity processes like human resources or IT operations, exactly the areas that are subject to black market economies. Specialized outsourcing providers have better economies of scale and lower cost structures, partly because they follow officially established processes. Because services are now performed by a third-party provider, and processes are contractually defined, the unofficial black market bypass no longer works. Essentially, the business has subjected itself to a work-to-rule slowdown. Organizations that rely on an internal black market, therefore, will experience a huge loss in productivity when they outsource part of their service portfolio.

Beating the Black Market

How do you avoid running the organization via a black market? More control and governance could be one approach: just like the DEA cracks down on the black market for drugs, you could identify and shut down the black-market traders. However, one must recall that the IT organization’s black market isn’t engaged in trading illegal substances. Rather, people circumvent processes that don’t allow them to get their work done. Knowing that overambitious control processes caused the black market in the first place makes more control and governance an unlikely solution. Still, some organizations will be tempted to do so, which is a perfect example of doing exactly the opposite of what has the desired effect (Chapter 10).

Note

You can’t eliminate black markets with more control and governance. After all, those are the very mechanisms that caused the black market in the first place.

The only way to avoid a black market is to build an efficient “white market,” one that doesn’t hinder progress but enables it. An efficient white market reduces people’s desire to maintain an alternate black-market system, which does take some effort after all. Trying to shut down the black market without offering a functioning white market is likely to result in resistance and substantial reduction in productivity.

Self-service systems are a great tool to starve black markets because they remove the human connection and friction by giving everyone equal access, thus democratizing the process. If you can order IT infrastructure through a self-explanatory tool that provides fast provisioning times, there’s much less motivation to do it “through the back door.” Automating undocumented processes is cumbersome, though, and often unwelcome because it can highlight the slow chaos (Chapter 31).

Feedback and Transparency

Black markets generally originate as a response to cumbersome processes, which result from process designers prioritizing reporting and control: inserting a checkpoint or quality gate at every step provides accurate progress tracking and valuable metrics. However, it makes people using the process jump through an endless sequence of hurdles to get anything done. That’s the reason I have never seen a single user-friendly HR or expense reporting system. Forcing the people designing processes to use them for their own daily work can highlight the amount of friction the processes create and thus provide a valuable feedback loop (Chapter 27). This means no more VIP support but support that’s good enough for everyone to use. Wouldn’t everyone like to be treated like a VIP? Similarly, HR teams should apply for their own job postings to experience the process firsthand.

Note

When recruiting, I routinely apply for my own job openings so I can detect any hurdles in the process.

Transparency is a good antidote to black markets. Black markets are inherently nontransparent, providing benefit to only a small subset of people. When users gain full transparency of the official processes, such as ordering a server, they might be less inclined to want to order one from the black market, which does carry some overhead and uncertainty. For example, will a black market server be supported or perhaps reallocated during the next inventory sweep? Therefore, full transparency should be embedded into an organization’s systems as a main principle.

Replacing a black market with an efficient, democratic white market also makes control less of an illusion (Chapter 27): if users employ official, documented, and automated processes, the organization can observe actual behavior and exert governance; for example, by requiring approvals or issuing usage quotas. No such mechanisms exist for black markets.

The main hurdle to drying up black markets is that improving processes has a measurable up-front cost while the cost of the black market is usually not measured. This gap leads to the cost of no change (Chapter 33) being perceived as being low, which in turn reduces the incentive to change.