Staying afloat as a new-ish solo data scientist

It's hard until you learn to handle yourself, and then it's still hard

As is often the case, something from work and internet reading provides the inspiration for this week’s newsletter - techniques to staying above water when you’re a new solo data scientist.

There’s so many companies out there hiring data scientists nowdays, it’s almost inevitable that someone will wind up being the only data scientist attached to a large cross-functional team/group/division. They might not be the first data scientist of the company (which I’ve written about almost two years ago), but they’re floating alone without having a peer data team to support them.

Remember! As with all similar articles about strategies with dealing with humans in various situations, these are things that appear to work for me in the situations I’ve come across. They’re not universal, so take inspiration, but they’re far from being a law.

We’re aiming to avoid even approaching burnout

The road to burnout is a long and subtle road. People don’t burn out over the space of a week. It’s a situation that slowly builds up over time, creeping up on you, until you just on the edge of a breakdown. The most important thing in all of this is to recognize the signs of stress that are unique to yourself, and use those as a signal to hit the brakes before things get worse.

Personally, as my stress cranks up I’ll wind up being more irritable and snarky. There’s insomnia, lots of procrastination and distraction looking at random things on the internet (like new hobbies!). I’ll also get a very distinctive feeling in my chest like I’m short of breath from tension, this particular sign is the one I use to know I’m going to a bad state and need to chill out.

I only learned these signs when I had a horrible job in my early career that pushed me very close to burnout. I was lucky enough to take an emergency long-weekend vacation at the peak of it. I sat on the plane and I realized at the doors-closed/phones-off announcement “the system at work could explode but no one can reach me to deal with it. I’m free.” I could suddenly breathe freely again for the first time in like 6 months.

That incident inspired me to job hunt, find a non-abusive place (look, it was 2008 and the height of the financial meltdown) and move on.

Your own reactions to stress will largely be unique to you. Pay attention to your body and behavior and learn your signs. I hope it doesn’t take something as drastic to teach you.

Why “new” and why “solo”?

Because of the complexities of human and group interaction dynamics, I have to limit scope, or I’d be spouting useless general platitudes.

The people most vulnerable to burnout would be people who are relatively new to a position (perhaps not new to being a data scientist, but fresh to the company). It’s also more likely to happen to people who are the only data person in their local group, because there aren’t going to be existing processes and systems to provide much-needed protections against burnout.

Let’s go diving - a scenario

For the purpose of this post, let’s pretend we’re New-DS. We’ve got some prior experience in industry, not fresh out of school, but not a grizzled “I’ve seen things” veteran either. New-DS has just joined a new company, they’re the only data person in that company. They’ve been hired to help the company use data to make better product decisions, and to broadly just make use of the data that’s available.

Luckily, the position is recognized as a junior one and they’re not expected to be responsible for everything data-related and setting strategy.

New-DS has a pretty reasonable manager, but not someone with a quantitative/analyst background. They’re either a senior engineering manager, a UX researcher, or maybe a product manager. They’ve never really managed a data scientist before, but they mean well and put the effort into trying to understand and learn what New-DS needs, even if they don’t fully understand HOW New-DS will accomplish their job.

The manager (and the whole company) sees the value in having quantitative insights, and everyone’s hungry to become data-driven as soon as possible.

Early on, it’s common for everyone to be nice and try to give New-DS some time and space to get oriented and acclimated to their job. It’d usually last a month or two, depending on how (un)structured the new employee onboarding process is.

But at some point, people are going to start wanting some of their questions answered. Their eyes light up when they hear that there’s a quant on the team now, and they’re hungry for data to do their jobs better. They might be unconsciously circling, like very friendly sharks.

The pressure starts building

In a perfect idealized world, all work coming in to New-DS would go through their manager for the first 6 months or so. The main reason for this is because New-DS just doesn’t know what’s important to the company right now. The most important function of the manager at this point is to provide priorities so that New-DS can focus on learning how to do their job.

While that’s the ideal, reality is never that simple. New-DS will talk to people, will attend meetings, will have lunch with their coworkers. Questions will come up where New-DS will think “Oh, we should have the data for that, I should be able to help this person!”. These are the ad-hoc requests we always hear about, and they come from all directions. Many of these come via direct channels so there’s no way any manager could filter them before hand.

Between “official projects” handed down from on high, ad-hoc requests coming from all directions, and generally learning about the company in general, New-DS has a very full plate. There’s more work than time.

On top of this, New-DS miscalculates their own productivity because they’re new to a data environment. Something that would’ve been “a quick SQL query” at their previous job takes 5-10x longer because they’re working with an unfamiliar system with unfamiliar data sources.

As this continues over the course of a few weeks, suddenly New-DS feels underwater. Every project and question sounds important. New-DS is frustrated that it takes whole days to do a simple data pull and analysis. Work is piling up. They’re feeling stressed and overwhelmed.

This is the bad place. We want to avoid coming to this point.

First off, lean on your manager, heavily

Good managers don’t want their charges to burn out, and they need to recognize the signs of it before it progresses too far. That said, even the most well-meaning and careful manager isn’t omniscient. You can’t expect them to know everything going on without you saying something.

The most important thing to do is get prioritization from the manager.

Just because people ask you questions doesn’t mean they need to get an answer from you, nor even a commitment to do any work for them. This applies to everyone, even if the CEO walks up to you and hands you a project, get the prioritization of the project with your manager first. You’ll very likely work on that CEO project because it’s likely important, but you at the very least need to let your manager know that’s where you’re spending your time.

In a very simple form, New-DS should build up a big list of all potential work, ranked by importance. Then somewhere along that list is a cut line. Any work above the cut line, New-DS is committing to completing within the next week/month/quarter/whatever. Anything below the cut line does not get worked on because there’s more important stuff to do.

Later, as New-DS gets more experienced and can learn how to prioritize work themselves, they’ll lean on their manager less for this function. But even then, an open channel of communication should be maintained. It’s never a good thing if New-DS and the manager have irreconcilable disagreements on what work is high priority.

Yes, this view of a manger requires that New-DS has a decent-to-good manager that’s interested in New-DS’s continued long-term success. This is not true for all managers and situations. If things are less-than-optimal, take care to understand how work performance is evaluated and align yourself along those lines.

Learning how to diplomatically say no to work

It’s hard to say no. Especially when you’re new. You want to make a good impression, show that you’re a good teammate, and in general be helpful. But saying “Yes!” to every random ad-hoc request is what leads to our horrible burnout scenario. So you need to find strategies to say no.

Say no (if you can)

Some people learn to say no directly. They’ll just say “Sorry I can’t work on this right now. Because of [reasons]” If you’ve got the confidence in you to do that, great! People generally aren’t offended by being told no for a valid reason.

I don’t outright say no very often, since it’s rare to get a request that is so clearly a bad idea. But it does occasionally happen.

Let people know you’re working on something else

Ad-hoc requests are not coordinated. There’s no grand conspiracy of coworkers trying to flood New-DS with work from all sides. Everyone honestly doesn’t want to overload New-DS. The situation results from people not knowing that New-DS has gotten work from other places. New-DS just became a collection point for data requests because people don’t know where to direct their questions.

New-DS needs to learn to say is a simple: “I’m working on [important task] for [important person] right now, can we come back to this topic next week?”. People will very quickly go, “Oh, that’s more important than my thing, I’ll come back later.”

Redirect upwards

It’s never wrong to say a variation of “Would love to help, can you talk to my manager to have that prioritized against all my other work?” This is particularly effective when starting out, like with New-DS.

Now, the work of saying no has been shifted to someone else. If an ad-hoc request is really that important, you’ll also know where it stands in the grand scheme of things.

Add friction to requests

Another good way to lower your request volume is to weaponize a common UX truth — that adding friction to a process lowers engagement.

There are tons of ways to do this, but usually it's variations on “would love to help, please do this for me so I don't forget it”. Where “this” is something that takes a bit of effort.

It could be intake forms (like Caitlin Hudon’s Intake form for Data requests) , a quick email describing the problem, file a ticket, talk to a manager, etc.. Note that we’re not throwing artificial barriers for the sake of it. Intake forms, or even an email, provide written durable context and documentation for a request. It’s not deliberately wasting anyone’s time.

Sometimes these intake processes force people to think about their request more, like rubber duck programming. They realize once they try too articulate their question that they don't need the answer and they leave you alone. Worst case is, they go through the effort and you at least have a thought-out explanation of the problem instead of a vague notion.

Sometimes people just don't care enough about a question to follow up, so it’s ok to let those fall by the wayside.

Ask for more time up front

If you’re underwater already, but you just can’t bring yourself to say no to a request for some reason, then ask for more time. “I’m swamped right now, can I get back to you next week?” You’re not going to be able to help them today no matter what, so it won’t hurt to set realistic expectations.

Keep a task list, use it

Some people are natural list makers, some people (like myself) aren’t. But this situation isn’t time to pick sides. While you’re underwater, you need to put all the work down on a list so you can prioritize, track, and most importantly, cross things off of.

Know that vague sense of dread when you take a break? The “I think I forgot to do something important” feeling as you sit down to dinner or go to bed? That’s because you’re trying to manage your workload in your brain. When you condition yourself to constantly juggle the workload during working hours, you’re not going to be able to turn that off for the rest of the day. It’ll forever haunt you.

So you need to dump all that stuff out of your brain, put it on a prominent sheet of paper, a whiteboard by your desk, a spreadsheet on your computer, whatever works. Once you take that load off your brain and you start relying that as your source of “what should I be doing now” list, you’ll be able to get a bit more rest.

Having a list also lets you do all the prioritization and communication of what you’re working on to others. You can just point to the list to get people to stop bothering you. Time savings!

Over time, as you get more comfortable with work, you might be able to retire the list, or only use it for Big Things. But only when you’ve gotten significantly better at managing your workload.

Remember that nothing is easy, everything is hard

New-DS needs to recalibrate their sense of how fast they can complete work. If they don’t, they wind up agreeing to “simple” things that turn out to be week-long monster jobs. Without anyone with prior data experience to help them, things can go wrong in all sorts of ways.

New-DS has done work before. Back in their old position, they could probably answer questions within minutes and write complicated SQL queries from scratch without referencing any table schemas. This gives them lots of confidence in doing analysis. The problem is that much of that knowledge does not apply to their new company.

It’s very easy to misjudge how much time it takes to become familiar with a single table. Even for simple tables, the time needed to read the columns, figure out the join relationships, spot check a couple of rows, can easily take up a few hours of time. It’s also time you don’t notice passing because you’re heads down figuring out a table.

It’s also important to tell other people how difficult certain work is. If New-DS started on a “simple” project, but after exploring realizes that the dataset doesn’t exist and they have to use arcane magic to back it out of a log file, people need to know. Some things aren’t worth the trouble, and no one but New-DS would know how difficult something is.

Find out what is “enough”, do only that

Many data scientists are perfectionists. Even if they’re not perfectionists, many have a very high internal quality bar for work and rigor. In general it feels good to have bulletproof work that covers all the bases because then we’re safe from surprises and embarrassment.

Perfection does not have a linear relationship to time spent. You can usually get 60%, 80% of the way to a solid answer with very little work, and then spend an entire week (or worse) trying to get to 90% perfect.

This way of thinking will make projects run unnecessarily long and just further compound the stress and work backlog. New-DS needs to learn when to stop working. The stop point is almost always much earlier than feels “correct”.

Remember, before New-DS came onto the scene, the company in this scenario had no data scientist on staff (maybe the previous one left, or it’s a new position). Everyone managed to do their jobs without help from data science. They can live for another few weeks while New-DS gets further along in ramping up.

On top of that, people aren’t usually asking for bulletproof answers. Directional guidance, such as “A is more likely than B” “C is much bigger than D”” is often enough to help them make a decision. Directional work is much easier and quicker to do than getting exact counts. Sometimes you can just estimate numbers with simple ratios.

Checkpoint early and often

Since New-DS won’t know what “good enough” looks like yet, the easiest thing to do is just ask. “Hey, I found this new small tidbit of information, does this help you? Should continue?” There’s a chance they’d be satisfied with just that much. A quick 5 minute conversation could save you a ton of time. I’ve often been surprised at what people will find to be good enough for their purposes, especially when they find out a more complete analysis might take 5x longer.

New-DS isn’t some independent research consultant. There’s no need to disappear into a cave, run up billable hours, and come back out with perfect answers. New-DS is helping people with their problems, which means they should keep that person in the loop and get feedback.

Promise less up front

There’s no need to give a detailed account of the analysis you’re going to do up front. It’s up to the data scientist to pick the correct methods for the question at hand. Making that decision involves knowing the systems and data involved, information which New-DS does not have!

So just promise to look at the issue, possibly find some quick relationships. It sets a reasonably achievable bar of work quality to expect. If you wind up doing a more thorough analysis because it was worth it, that’s a bonus.

Breathe

Not joking.

Everything that is currently on fire and “SUPER URGENT, DO NOW” will happily continue to be on fire 30 minutes from now whether you stress about it or not. Any enterprise that is teetering on eminent disaster if a brand new hire doesn’t get their work done has deep structural issues that the new hire can’t resolve. New-DS literally has the least at stake in the company of anyone else in the entire company.

Stress will destroy what little work quality that can be mustered together. Plenty of studies link stress with job performance. Don’t become another statistic. Take a breath, take a break, even take a short nap if you need to. A 30min walk + 30 minutes of work when refreshed is often more productive than 60 minutes of sitting at your desk banging your head against a report.

These are all the self-defense mechanisms I can think of that I’ve developed over the years. I’m sure there are tons more that I’m just not conscious of. Feel free to hit me up in email replies, comments, or on the internet for further discussion.

---

About this newsletter

I’m Randy Au, currently a quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. The Counting Stuff newsletter is a weekly data/tech blog about the less-than-sexy aspects about data science, UX research and tech. With occasional excursions into other fun topics.

Comments and questions are always welcome, they often give me inspiration for new posts. Tweet me. Always feel free to share these free newsletter posts with others.

All photos/drawings used are taken/created by Randy unless otherwise noted.