Why do data scientists quit their jobs so often?

Picture this

You studied data science for years, hoping to break into the industry. Finally, after countless rejections, you hear back from an interviewer. You're getting hired. All your hard work finally paid off.

While this sounds like a really happy ending, the story doesn't end here.

Months into working for the company, you feel demotivated, drained, and tired. Your manager is constantly breathing down your neck because sales are low. The models you build aren't converting into purchases.

Finally, you decide you can't handle it anymore. You start looking for a new job and hand in your resignation letter.

Sounds scary, right?

I know.

Unfortunately, this is a very common situation in the data industry.

Of course, this won't happen to you if you work in a Fortune 500 company. If you end up working for Google or Facebook, then your experience will be very different from everything I mentioned above.

This is because companies like Google can afford separate positions for people who specialize in different things. They have separate roles for data scientists, data analysts, data engineers, statisticians, and decision scientists.

Because these companies have such highly specialized roles, they only hire people with really high qualifications (Masters or PhD holders) as data scientists. In fact, I remember reading somewhere that Google's data scientists don't even need to know how to code.

Apparently they specialize in model building, so somebody else writes the code for them and puts their models into production.

You can fact check this if you like, but the point I'm trying to make is that most data science jobs available out there aren't as highly specialized.

Most companies out there aren't like Google.

This means that you need to focus on a lot of other areas outside model building. You will need to code well, learn how to scale your models, perform statistical analysis, and most importantly: be able to derive business value from the models you build.

A lot of new data scientists find themselves caught unaware in a situation like this, and are unable to provide a company with business value.

The Problem

As I mentioned above, the problem is that most data scientists are unable to provide business value from the models they build.

And honestly, a lot of the time, this isn't their fault.

Many companies expect data scientists to perform magic. They will give you one month of consumer data and expect you to build a model that doubles their sales.

After spending weeks trying out different machine learning algorithms and picking the model with the highest accuracy, you send them the model results.

They use the insights derived from your model to promote different products to a wide range of customers.

And... nothing happens. There is no sudden increase in sales.

There is a lot of disappointment.

It comes as a huge shock to them that data science isn't magic after all. You try to explain that you did the best you could with the data at hand, but to no avail. It all goes downward from there.

The situation above is a slight exaggeration of events, but I'm sure you get the gist of it.

The real problem here that needs to be addressed is a mismatch between the technical and business aspect of the project.

You see, your boss doesn't come from a technical background. He doesn't understand that there is no one size fits all magical model that can solve all his company's problems.

On the other hand, you don't understand the business requirement of the project. You have no visibility on what the project is actually trying to achieve. You don't know how to relate a metric like accuracy to sales.

What can you do differently?

Firstly, avoid companies that don't have a proper data science team in place.

You don't want to be stuck in a situation like the one I described above - where the company has no idea of the data science lifecycle, and thinks that hiring one or two data scientists can solve all their problems.

These companies probably don't even have a proper data storage system, so it is better to avoid them altogether unless you want to end up building their entire data science pipeline.

Instead, go for a company that has a data team in place. Ideally, the company should also have consultants who help bridge the gap between the business requirement and models built.

When you work in a company, don't blindly do the things your manager tells you to do.

If the requirement is to "build a machine learning model that maximizes sales," then ask some more questions.

Make sure you understand the context behind the model being built. Do some domain research. Ask your manager for consumer insight reports and read those.

Understand trends between the data you have and projects created in the past. Get to know your consumer base better, and learn the marketing strategies used to target these customers.

You will be able to provide better recommendations based on your understanding of the business requirement.

Sometimes, a project is a dead end. If you feel like the data provided to you is insufficient to provide recommendations to drive sales, then be transparent about it. Let your manager know.

Not every problem requires machine learning to solve.

This way, even if they don't want to budge and push you to build a model, they aren't blindsided if they get unsatisfactory results. You have already made them aware that the data available didn't relate to the problem at hand.

Make sure to be involved in every part of the process. Don't be afraid to ask questions. Understand the impact your model has on sales, and make sure you have visibility on the results of your analysis.

I don't come from a business background, but learnt a lot about marketing once I joined my company as an intern. Initially, I was too technical and couldn't connect the dots. I just built models, but didn't understand how it related to the business requirement.

Over time, I learnt.

Every time I worked with a client from a different industry, I'd read about the industry and do some market research. This way, I could provide better recommendations on target audience and help maximize sales.

Model building is a very small part of being a data scientist.

The ability to bridge the gap between model building and the business requirement is very important. Arguably, it is the most important aspect of data science.

Note

If you are a beginner in the data industry and are confused about the different career options, don't know how to start learning data science, or are looking to land a data science job, you can schedule a 30 minute or 1 hour consultation session with me here: https://lnkd.in/gURAj82