Navigating working with other teams
Generating value upwards
Two weeks ago, the post I wrote about perf and showing value as a data scientist got a surprising number of people reading and sharing it (also, wow, thanks!). But amidst the conversations that popped up around the article, I got this nice piece of feedback:
And Koen is very correct, I spent much of the post writing from the perspective of a low/mid-ish Individual Contributor (IC) data scientist showing value upwards to various people. That post alone had taken over 2,600 words, so I have no regrets about ending it where I did, but it’s true that there’s EVEN MORE to talk about, especially looking upwards and even more broadly across an organization.
So, while the topic is still fresh in my brain, let’s get some more of it out of the way by discussing the complex dance that’s involved in showing value to other teams on a broad basis. One problem I’ve had while writing this is it’s all very squishy, with lots of “it depends”. Hopefully I can illustrate enough to give you some ideas.
I’m reminded of this old video about the science behind persuasion. I can see many of the principles detailed there reflected in some of the stuff I’m writing here.
Still, to keep with the theme of showing value, I’ll say that showing and creating value higher up gets more intangible. The primary way work is done at the higher levels is not in direct deliverables. It’s in making decisions, allocating resources, prioritizing work, setting policies, and planning for the future, all of those are largely intangible. Sure, you can say “lead the team to launch X” but you didn’t do the launching, and the leading part is intangible.
One thing to note here is that I’m most definitely NOT an expert on this topic. I have my set of strategies and techniques that I have experience using, but there are TONS of other ways that I don’t even know about.
Ground rules: everyone’s on the same team
Product development work generally has three large stakeholder groups that broadly encompass the important skills needed to make something:
Engineering, the people responsible for building and launching product features
Product, the people representing the business side, and also brings market needs, product vision and strategic vision
UX, the people representing the needs of users through design, research, and writing
Data usually falls somewhere vaguely between UX and Product groups as a support function, depending on the exact needs of the org
Not all places have all the same teams split out this way (for example, many UX functions can wind up split between Product and Eng teams), but the functions and concerns always exist.
No matter the organizational setup, each of these groups usually have their own priorities. This means that much of the work of showing value at higher levels as a data scientist involves influencing and convincing other teams to be on board with things that you want to have done. Just like how there are instances where other teams will need things from you.
On the topic of “influencing others”, it’s very easy to write things that sounds adversarial, since a lot of things I’ll cover involve “convincing team X to do Y.” So it’s important to remember that everyone is working towards the same goal: to build something successful. There’s plenty of room for disagreement about what should be done, without any need to be negative about anything.
Side note: While office politics can definitely be a complicating factor depending on your organization, I’m going to avoid discussing that topic because I’ve never worked in an environment that has a tough, complicated, Game-of-Thrones-esque political situation.
Leveraging the priorities of other teams
Everyone has things they want to do.
Engineering has features they want to build, tech debt they want to pay off. Product has a line of angry customers with complaints and deals they want to close with new features. UX has a list of horrible experiences that they want to make less painful to the user.
Usually, all the things that each team wants to do needs help from others. Product will need UX research and design to vet their product ideas. Eng needs product to provide prioritization and guidance on what to build, product needs Eng to build a functioning and sustainable product. The interdependencies are endless. There’s simply not going to be enough work-hours in the day to do it all, hence why management goes through lots of prioritization.
So the most efficient way to get work done is to survey the landscape of prioritized work and find creative ways to include parts of things you want done into work that is definitely going to happen. This lets you get closer to your long term goal without requiring everyone stop to work on your project. When you’re out of work that can piggy-back off of, you’re hopefully only a few short steps from the goal, which is easier to justify.
Let’s be honest here — this strategy isn’t exclusive to any discipline and is probably one of the most important ways to get things done for any team.
One common example is that UX has been wanting to fix the registration flow with major overhaul for ages, but there hasn’t been any appetite from the other teams. But this year, Eng is doing a major upgrade of the front-end frameworks. They’re going to need to reimplement a bunch of forms and things anyway.
Such a situation would be the perfect time to make improvements to that page, and other pages, since engineers would be touching the code during their rewrite process anyways. Maybe you can throw in a few architectural changes to accommodate future design ideas. This work doesn’t come free, time will still need to be spent building and testing out the new design, but the incremental cost can be small enough that you won’t find a lot of resistance. Assuming you get in the conversation early enough.
Similarly, the data science team can make a case for finally putting a bunch of missing instrumentation, or trying to get an instrumentation framework implemented at the same time as the rebuild. It’s only adding a few extra lines of code to a file you have open anyway.
Once you get in the habit of looking for these opportunities to piggyback work you want onto work that’s already happening, you’ll find that they’re quite common. Oh, the product leads are doing market research on a new feature? Maybe we can slip in an extra research question that we’ve always wanted to ask people. UX is going to undertake a big redesign project? Maybe they’ll agree to take a look at the aging email templates also.
Influencing teams with data
The current trend in business is to be “Data driven”. It’s up to debate exactly how much companies are actually driven by data as opposed to gut feelings justified with shaky data. But despite that nuance, everyone likes to use the trappings of data to push their arguments forward. This means that data teams have an easy time building relationships with everyone — they can produce a resource that everyone finds valuable.
It’s pretty common for other teams to ask the data folk “we have this idea, can you see if there’s data to support working on this?” They want to make sure what they’re about to expend effort on is meaningful and worth doing. Any answer you give, whether it’s “this will probably be impactful” or “this will unlikely to move any needles” is welcome, because the analysis you’ve done was hopefully fair. This is a direct way of to have an effect on a team.
But there’s also times when a team is about to embark upon something and they haven’t asked for data, but you hear of the plans and take it upon yourself to check their assumptions against available data.
Let’s say you find something positive, and they should go on as planned. Then by all means show them the data and analysis because most people won’t turn down supporting arguments in favor of their case. It’s a great chance to build relationships. “Hey, I heard you were working on this and was curious and it looks like it’ll affect a gajillion users positively! Yay!”
Putting the brakes on a project
But what if you find something negative? That there’s data that points to either stopping the project outright, or making a course correction. This takes a bit more diplomacy to handle.
Ultimately you’d still want to make the information known. But depending on exact culture of your organization, you need to figure out what’s the best way to go about it. People won’t be overjoyed at being shown contrary data, but they are likely to at least give it consideration. It’s all in the presentation and framing.
Having a good prior relationship of trust beforehand definitely makes the process easier. When there’s a base level of trust already established, you’re MUCH less likely to sound like you’re randomly checking up on someone’s work (that never feels good). But even with trust, it’s also important to not just say “this is a bad idea” but also to use it as a chance to partner up with the team in figuring out how to chart a path forward. There’s usually some parts that are reusable.
This situation is where everyone being on the same team is important. What you’re doing is not much different that if you were the company lawyer hearing about a new feature that’s a giant lawsuit waiting to happen. Putting a stop to disasters, be it lawsuits, revenue lost, or very angry customers, is just as important as launching something new. Finding data that says something is a bad idea is a similar, albeit less clear-cut, case.
Convincing others to invest in data infrastructure
While other teams aren’t able to help data teams very much in doing data analysis, infrastructure is often something we’d always love more help with. Whether it’s instrumenting parts of a product, setting up infrastructure to handle complex pipelines, or getting help with building out tools, having help is nice.
The problem with such projects is that people who don’t work with data may not know how important certain things like good instrumentation can be. Until they know the potential value it brings, they’re not going to be able to weigh appropriately against their own priorities. End result is you don’t get the help you need.
So what can we do here? Lots of hard work advocating and pushing for things. Obviously it should be brought up during planning and we need to advocate for our position. But in addition to that, we should try to frame our work in terms of how it benefits everyone involved. If you make a good case, it could raise it to be competitive with more pressing projects.
A new testing framework, which would require changes to the general dev process and lots of potential implementation details, can’t just make the lives of data scientists matter for other teams to care strongly. It also should address the concerns of other teams, like being easier to maintain and reducing the number of times we have to say “we can’t measure that”.
Hopefully you’ve been consistently beating the drum at appropriate points about how certain data questions that are coming up need tooling upgrades that you need resources to help implement. That way, when it comes time to propose a new tracking system, people already have a sense of what they’re going to get out of it.
And finally, sometimes schedules magically align and there’s a short lull in work! In these instances you can sometimes convince others to help out by sheer goodwill. Again, since we’re all on the same overall team, taking the time to get rid of a hugely inefficient time-sink for one group is worth it for everyone involved.
Acting as a data dealer
Since every likes having data to back up their arguments, one thing I’ve noticed over the years of being dropped into new teams all the time is that people get addicted to having data analysis on fast turnaround.
Once teams get the first taste of having their product launch instrumented and upper management nods approvingly at the metrics being reported, they’re usually enthusiastic for more.
It usually starts with something very simple and formal, like setting success metrics and doing instrumentation for a new feature launch. Clear benefits to the team up front. Then they have a good experience with having those metrics, and they’ll ask to continue with that process. They’ll also have questions that “maybe you can help with?”
Things snowball from there. As demand grows, so does visibility. Eventually, if you’re swamped and bottle-necked, you have lots of supporting voices asking for the data team to get expanded.
This pattern of behavior is part of the reason why I keep a very open door and encourage everyone to talk to me about any data topic under the sun. You’ll never know when a bit of analysis might snowball into something important, so I like maintaining a sense of what the overall teams are thinking of.
Leveraging people who hold power
Sometimes, work that needs to be done is a complete drag. For example, going through and labeling a bunch of UI elements so that telemetry is labeled correctly downstream. Everyone recognizes that it’s important work at some level, but people understandably don’t really want to do it. No amount of sugar-coating will make it a fun project, so people tend to be pass over them.
In such situations, I don’t (and shouldn’t) have any actual power to tell other teams to do drudge work. But, if I can make a strong case on how useful the work is to someone who does have that power, there’s a chance it can happen. Those higher level managers: product leads, project managers, directors, etc. are able to see the value they can get from having a something a common element naming/tracking system. They can use their own leverage to signal boost the importance of a project and hopefully raise it above the cut line for work during planning sessions.
While this is going on it helps soften the blow if you also take on a bunch of drudgery yourself, so that at least it’s shared pain instead of completely transferred pain. I tend go out of my way to minimize pain inflicted so that people can see we’re not just offloading crap to them. Social gestures are important here.
This process is why I always feel it’s important to have leaders that do understand all the aspects of what goes into making a great product. They can’t be experts at everything, but they needs to have a broad enough viewpoint that there’s room for everyone to make a case for important work. They can see the value in work that other teams might not fully see the implications of, and use their position to help things along.
Be clear about your capacity budget
When I’m supposed to plan out work for a quarter, or a half year, I use a rough rule of thumb: I aim for spending about 1/3rd of my time handling things I want to do, 1/3rd handling things that other teams need me to do, and 1/3rd handling the random influx of ad-hoc coming in.
The 1/3rd ad-hoc time includes various meetings and fire drill work, so effectively 2/3rds of my time winds up being taken up by “work for other teams”. The difference is whether it’s time registered before hand or not. It’s admittedly quite a lot, so I make sure other teams know when they’re starting to cross the line by pushing back.
Still, the rule of thirds is only a guideline and there have been emergency crunch periods where maybe 90% of my time is taken up by other teams. But these NEED to be the rare exception because being constantly buried like that sets you up for failure in the future because you can’t take care of your own work debt.
When things get really bad, I’ll hash it out with my manager and other team leads in a conversation of “we have Big Projects A, B, C on deck. We suddenly want to add Project E and that’s too much. Which of the four do I drop in priority?”. One of these projects will wind up being deemed less important, but the most important part of the process is making sure everyone reaches consensus as to which is less important.
No one should have their project dropped without having a chance to understand why.
Expect horse trading and compromise
Given that there’s a finite amount of resources, no one ever gets exactly what they want. No amount of employing any strategy you’d like will get a perfect result. If the production system is literally on fire (like, the data center has actual black smoke coming out of it), you’re not going to be spinning up that new Hadoop cluster today, maybe not even this quarter.
Maybe you’ll do extra work on a project this quarter, in exchange you get some engineering help next quarter. On average it can balance out. Again, building good working relationships is important.
Favors and snacks are definitely a currency that gets used.
About this newsletter
I’m Randy Au, currently a quantitative UX researcher, former data anslyst/scientist and general-purpose data and tech nerd. The Counting Stuff newsletter is a weekly data/tech blog about the less-than-sexy aspects about data science and UX research and tech. With occasional excursions into other fun topics.
Comments and questions are always welcome. Tweet me.
All photos used are taken by Randy unless otherwise noted.