first predictive model in healthcare looked like a home run.
It answered the business question. The performance metrics were strong. The logic was clean.
It also would have failed spectacularly in production.
That lesson changed how I think about data science and what it takes to be successful in healthcare in the age of AI.
Looking back, that failure would repeat itself throughout my career, but it was crucial to my growth and success as a data scientist: a complex model in a notebook is worth nothing if you don’t understand the environment your model is meant for.
Data Analyst
After three grueling months on the hunt for my first job in the real world, in a market with a fresh appetite for data but that was also teeming with talent, I was finally given my first big break. I landed an entry-level data analyst position on the Business Intelligence team at a large hospital system. There was so much to learn. A massive hurdle, and one that many people wanting to get into the healthcare data realm will also have to jump, was familiarizing myself with the ins and outs of Epic, the largest EHR (electronic health record) vendor by market share. Stretching my legs in SQL with the extremely complex data in an EHR was no easy feat. For the first few months, I was leaning on my senior coworkers to write the SQL I would need for analysis. This frustrated me; how could I have just finished a master’s degree in statistics and still be struggling to pick up the SQL mindset?
Well, with practice (a lot of practice) and patience from my coworkers (a lot of patience) it eventually all started to make sense in my head. As my comfort grew, I dove into the world of Tableau and dashboarding. I grew fascinated with the process of making aesthetically pleasing dashboards that told data stories that desperately needed telling.
Illustration by Luky Triohandoko on Unsplash
Throughout my first year, my manager was extremely supportive, checking in regularly and asking what my career goals were and how she could help me achieve them. She knew my background in school was more technical than the ad-hoc analyses I was doing as an entry level data analyst, and that I wanted to build predictive models. In a bittersweet end to my first chapter, she offered to transfer me to another team to get me this experience. That team was the Advanced Analytics team. And I was going to be a Data Scientist.
Data Scientist I
From day one, I worked closely with a data science guru who had a deep knowledge of healthcare and the technical capabilities to match, giving him the ability to deliver amazing products and pave the way for our small team. He was the first in our system to develop a custom predictive model and get it live in the production environment, producing scores on patients in real-time. These scores were being used in clinical workflows. When my manager asked me what my professional goals were for the upcoming year, I had an immediate and certain response: I wanted to get a custom predictive model into production.
I began with a few POCs (Proofs of Concept). My first model was a linear logistic regression model that attempted to predict the likelihood of complications from diabetes. While a good first attempt, my data sampling approach was all wrong, and in peer review, my colleague pointed it out. One of the key lessons I learned from my first attempt at a predictive model in healthcare was
When gathering data to train a predictive model, it is crucial you mimic the conditions, patient context, and workflow in which the model will be used within the production environment.
An example of this: You cannot simply gather each patient’s current lab values and use those as features in your model. If you are expecting the model to make predictions, say 15 minutes after arrival in the ED, you need to account for that. Thus, when gathering two years of historical data to train a model, you need to gather each patient’s lab values as they existed 15 minutes after arrival, i.e. at the time of their simulated prediction date and time, not what those lab values are today/currently. Failing to do so creates a model that may perform better in POC than it does in real-time production environments, because you are giving the model access to data it would not have available to it at the time of prediction, a concept known as data leakage.
Lesson learned, I was ready to try again. I spent the next few weeks developing a model to predict appointment no-shows. I was very intentional on how I gathered data, I used a more robust and powerful algorithm, XGBoost, and once again got to the peer review stage. The model’s AUC (Area Under the Receiver Operating Characteristic curve) was astounding, sitting in the low 0.9s and blowing everybody’s expectations for a no-show model out of the water. I felt unstoppable. Then, it all came crumbling down. During a deep dive into the surprisingly strong performance, I noticed the most important feature was the scheduled appointment time. Take that feature out, and AUC dropped into the mid-0.5s, meaning the model predictions were virtually no better than random guessing. To investigate this strange behavior, I jumped into SQL. There it was. Within the database, every patient who did not show up to their appointment also had a scheduled appointment time of midnight. Some data process retrospectively changed the appointment time of all patients who never completed their appointment. This gave the model a near-perfect feature for predicting no-shows. Every time a patient had an appointment at midnight, the model knew that patient was a no-show. If this model made it to production, it would be making predictions weeks before upcoming appointments, and it would not have this magic feature to pull up its performance. Data leakage, my arch nemesis, was back to haunt me. We tried for weeks to salvage the performance using creative feature engineering, a larger data set for training, more intensive training processes, nothing helped. This model wasn’t going to make it, and I was heartbroken.
I eventually hit my stride. My first big predictive model success also had an amusing name: the DIVA model. DIVA stands for Difficult Intravenous Access. The model was designed to notify nurses when they may have difficulty placing IVs on certain patients and should contact the IV team for placement instead. The goal was to reduce failed IV attempts, hopefully raising patient satisfaction and reducing complications that could arise from such failures. The model performed well, but not suspiciously well. It passed peer review, and I developed the script to deploy it into production, a process much harder than I could’ve imagined. The IV Team loved their new tool, and the model was being used within clinical workflows across the organization. I accomplished my goal of getting a model into production and was thrilled.
Illustration by Round Icons on Unsplash
Data Scientist II
Following the successful implementation of a few other models, I was promoted to Data Scientist II. I continued to develop predictive models, but also carved out time to learn about the ever-growing world of AI. Soon, demand for AI solutions increased. Our first official AI project was an internal department challenge where we would employ language models to summarize financial releases of publicly traded companies in an automated fashion. This project, like most other AI-related projects, was quite different than the typical ML model development I was used to, but the variety was welcomed. I had so much fun diving into the world of ETL processes, effective prompting, and automation. While we are just getting our feet wet with AI initiatives, I am excited for the new types of business problems we can now create solutions for.
My role as a data scientist has evolved as AI systems have improved. Developing DS/ML and AI solutions requires much less technical work effort now, and I almost think of myself as part data scientist, part AI project manager during the process. The AI systems we have access to now can write code, bug test, and make edits very effectively with tactical prompting on our end. That said, there is a growing concern about the impact and feasibility of AI initiatives, with various reports suggesting that most AI projects fail before seeing production. I believe
A Data Scientist with a strong technical foundation and subject matter expertise can be the greatest asset to combating the high failure rate of AI projects.
Our understanding of predictive models fundamentals coupled with domain knowledge from within our industries (healthcare, in my case), is still very much needed to create solutions that are effective and can provide value. Gone are the days when we could rely solely upon our technical acumen to provide value. Coding is now handled by LLMs. Automation is much more accessible with cloud providers. An expert that can translate the needs of the business into a strategic plan that guides AI to an effective solution is what is needed now. The modern data scientist is the perfect candidate to be that translator.
Illustration by muhammad noor ridho on Unsplash
Wrapping Up
Data science, as with any career path in tech, is always changing and evolving. As you can see above, my role has changed so much in the years since college. I have climbed a few rungs of the corporate ladder, going from an entry-level data analyst to a Data Scientist II, and I can say with confidence that the skills required to be successful have shifted as the years have gone by and technological advances have been made, but it is important to remember the lessons learned along the way.
My models failed.
Those failures shaped my career.
In healthcare, especially with AI magic at our fingertips, a successful data scientist isn’t the one who can build the most complex models.
A successful data scientist is one who understands the environment the model is meant for.
