Summary: Remember to step back and look at the bigger picture of user experience and context to learn see what it takes to make a truly great product.

No project is only what it seems to be on its face. No matter how detailed the visible stuff like a project charter or business requirements might seem, there’s always an invisible narrative churning in the background. That’s because every project is nested in a broader organizational context made of people. And people are complicated.

Organizations are not just “organized” but “organic”; they’re living, breathing systems, messy in ways that the clean-lined structure of an org chart often obscures. This may sound obvious, but it’s so easy to forget. Unfortunately, forgetting to pay attention to the invisible background narrative is what causes many projects to fail. That’s why I’ve come to think of every project as having a second, “meta-project” going on in the background—one where these invisible challenges are dealt with head-on.

Planning and designing a user experience solution based only on the official, visible narrative can result in a brittle, overly optimistic (and overly “optimized”) outcome that doesn’t have the resilience to survive the buffeting and stresses of how people and human systems actually work. So what does one do about it?

Understand the Reality, Not the Fiction

Traditional IT requirements gathering brings Subject Matter Experts into conference rooms and interviews them on their responsibilities and workflows. It documents these “inputs” and looks for points of pain and inefficiencies, not unlike looking for bugs in source code or a poorly routed wire in an electrical circuit. That would be great, except that people aren’t code or circuitry.

A better approach is to do contextual inquiry, otherwise known as field studies, or ethnographic research. This style of research (found in methods such as Contextual Design) draws on tools from the social sciences to understand how people actually think and act in the context of their work, as well as how they operate within a larger human system rather than only in individual, official “tasks.”

In “The Design of Everyday Things,” Donald Norman extols the virtues of such “applied ethnography,” urging design teams to “Watch [users] commute, at parties, at mealtime, and with friends at the local bar. Follow them into the shower if necessary, because it is essential to understand the real situations that they encounter, not some pure isolated experience.”

Norman may sound like he’s joking about how far to follow people, but he’s not. John Seely Brown, former Chief Scientist at Xerox, often relates a story about how the tech company pioneered using these methods to figure out why its copier technicians weren’t following procedure by using the official manual; ethnographic inquiry discovered they were actually learning more from each other in conversations away from the workplace, like at a diner during lunch. Xerox wisely didn’t quash the community of practice, but invented new ways for them to learn from one another while in the office or on-site with customers.

Look Past the “UI” to the Real Story

We have a natural tendency to talk about abstract, underlying issues in terms of concrete solutions and features. This is why projects are often so impatient to see wireframes and prototypes: it’s hard to talk about what you’re making without literal representations of its appearance.

Getting poor usage numbers for your mobile app? “Let’s add badges for gamification!” But why? Maybe what you need is better content, or a less confusing taxonomy, or a more learnable interface that people enjoy using, rather than slathering game mechanics onto a broken structure.

There are many complex, systemic dynamics that contribute to things like user adoption. It’s crucial to figure out what those are and bring them out of the shadows, then address them rather than only focusing on the surface point-of-pain.

An organizational narrative drives the need for any project, and that narrative has to do with things like corporate culture, interdepartmental politics, and even the way a company structures worker incentives. I can’t count the projects I’ve seen scheduled and forced to deliver just to meet a sponsor’s annual review deadline. And almost every time I’ve seen a structural chasm between one part of a user experience and another, there’s some political divide behind it: fragmented information architectures are often like X-ray films exposing the cultural cracks in an organization.

Remember the Context of the Project

Some of the most important conversations you can have are with the senior stakeholders are about how you can help them succeed not just with the product, but with the context around it.

Often, stakeholders don’t want to get into that stuff, assuming it’s beside the point. But it’s actually the entire point; it’s just hidden under the surface. Whether you’re an external or internal designer, find out how you can help your client contact succeed; let them know you’re aware there are always non-technical pressures and constraints around any project, and that you’re willing and able to help them overcome those invisible complexities. This can be a tricky conversation to initiate, and may need to wait until some basic trust has been established, but it can make or break the relationship and the project.

And by the way, this works from the other way around too: a project sponsor can do her project a world of good by finding opportunities to involve savvy design partners in the meta-project. Why fret over the invisible stuff alone, of there are people willing to help you deal with it?

Understand the “Why” and “Who”

As projects progress, it’s easy to get sucked into focusing on the “meta-project” of “do whatever it takes to make senior stakeholders happy!” or “Produce documents to meet project-plan deadlines!” The problem, though, is that designing for internal acceptance is not the same work as designing a viable, usable product. The product itself can get lost in the drama of deciding on requirements, prioritizing backlogs, and managing the expectations of multiple interested parties. (Personas especially have a tendency to fall prey to this misdirected effort.)

Agile methods are supposed to help us avoid this problem, but bad habits are hard to break. Teams end up focusing on the dog-and-pony show of the “demo” rather than the long game delivering the right product.

As Bill Buxton explains in “Sketching User Experiences,” it’s a myth to think we can “adopt a process that will take us a straight path from intention to implementation.” Design is inefficient by nature; it still has to work toward a goal, but we need space and time to try things that often fail and end up on the cutting room floor.

This can be especially challenging in straight-to-prototype design work, where the organization has a hard time telling the difference between a clickable/touchable concept and deployable code. Sometimes it’s necessary to bake these two separate pieces of work into the project plan, to protect the messy, throw-away-our-mistakes work of prototyping. It can also help to not call it a prototype at all, but something like a “functional sketch.” (It’s amazing what changing the semantics of the work can do.)

Design How We Design

Sometimes it can be tempting to ignore these “meta” concerns as irrelevant. Like a novelist who wants to create out of inspiration rather than what an editor or the marketplace wants, design can be a bit precious about its “vision.” But making a viable product means paying attention to the context of how the product is being envisioned, planned, and made.

Maybe a better way of framing this is to think of the meta-project as a design challenge in itself. How can we not only make something great for end-users, but make a great project that navigates the tangled wilderness and stormy weather of the enterprise as well? It takes extra attention and effort, but it almost always pays for itself in the end.