Expert Insights into Digital Behavioral Data for Health Analytics

Sanja Šćepanović: I work a lot on different health topics and social media health discus­sions, some of which I will present during the IC2S2. We developed a method where you can extract any symp-tom mention from text, for which we have several applications. One was to study stress across companies and another one was to study public health across the United States. I am currently expanding my work on public health applications using various public health data sources, for example, prescriptions in the UK in relation to satellite environ-mental indices [¹].

Sanja Šćepanović: My master’s was in a very different area, in cybersecurity. But then I found myself in California, attending a lecture byJure Leskovec. He was presenting on computational social research, calling it social network analysis at that time. The main story that Jure was telling was that for the first time in history, we are collecting so much data. Before, social scientists, psychologists, they all had to spend so much time to collect data, mostly by interviewing people. Now all this data is at the tip of our fingers. The problem is not the data anymore, it is how we make sense of it. That for me was convincing, that this would be one great way to do science nowadays, to have the power to make sense out of this new data, big or small.

Sanja Šćepanović: Then I worked on various topics which were not particularly health related until I joined Daniele Quercia’s team. When I joined, there was the question on what we should start working on. There was an open project because a person left. This project was a little bit about what I mentioned earlier, about how we extract and parse health discussions. It was in a very early stage, and that was interesting to me. But at the time I did not imagine that it would draw me in so much, I thought it is just one of the projects as I had done them before. In the process of working with the project and the data, I realized that it is a very interesting and useful application. Before starting the project, I did have interest in mental health in particular, which I think is one of the main issues nowadays [², ³, ⁴]. Social media is a good reflector in some way. This one started spontaneously; it was this big project, and then, once I executed it well [⁵], it led to many applications. It started shaping as a research topic of mine.

Sanja Šćepanović: Yes. And I think not just for me, but for all of us. The development of AI is so significant! When I was working on my PhD, for example, I was trying to analyze the homophily and semantic relatedness of discussions on Twitter. To be able to do this, I had to download the entirety of Wikipedia and create an embedding space, because there was only an old method before all this embedding deep-learning came. There is an old method by a researcher at Google, Evgeniy Gabrilovich which was to download Wikipedia, then do tf–idf analysis on the articles to find how related some words are [⁶] then build your own embeddings, and finally apply it on your own data, for me Twitter. I did this, which was an enormous effort. Fast forward three years and it is two lines of code with SBERT now.

There are inflection points where you realize that the developments of technology are so tremendous that it is important to have good questions. If you are into applications, having good questions might even be more important than honing your technical skills, because that is getting developed on the side, if technical and method development is not your main area.

Sanja Šćepanović: I am not a hundred per cent convinced the we found the best usage of digital data for health yet. If we could work with prescription data on an individual level, if we could interview all people, or be using doctors’ records, all of that would be better. What we do with digital traces is a proxy. But I do think, especially for mental health, that social media can give us even more, because people are not always truthful when they talk to their doctors. Right now, for example, I am working with prescription data. You do have biases, where maybe some population is more aware that they can use mental health medications; if they assume they are good, they will use them more. This is not because they have more mental problems, but just because they are more aware of the medical options. So even those data come with biases, and I think social media there can reveal things that even doctors do not understand.

The other thing which is useful, but I still do not know how we can do it, is to see health from people’s perspective, not just from the medical experts’ perspective. Our work on symptoms revealed that medical taxonomies define conditions with some common signs that doctors think define a condition, but people actually talk about other symptoms, so doctors and patients do not express it in the same way. This might be useful in some ways, but I do not have a concrete idea on how yet.

Sanja Šćepanović: I should mention data from wearables, even though that is not direct­ly my main area. I do have two pieces on wearables [⁷, ⁸].Because this data is usu­ally indi­­vidual level data, it is going to give you more information than social media data alone can. That is my perspective now. For types of wearables I think about smartwatches as well as more unusual types of sensors, like an abdominal body sensor. You can sense sounds, for example, coughs for COVID, audio data, or PPG signals and so on.

The other part is about methods. Now we have conferences devoted to machine lear­ning for health, where methods are developed particularly for health data (for example, https://chilconference.org/) Another huge part is electronic health record data from collaborations with hospitals; these notes, written by doctors, are just like social media data, free-form text. They are just also natural language processing tasks, and a big part of the challenge in that community is to transcribe these records in a unified way so that we can analyze them well. There is a lot of work there, it is not my work, and I think now this is in some way solved. Then we have radiology scans and image data. So this con­fe­rence is devoted to methods and finding ways of analyzing these texts, or even radiology scans and image date, while being careful about biases, which is super-important.

Another important thing is how we can make these methods trustworthy and trusted by doctors. The definition is not clear yet, because this is subjective and personal. The research there is not going into technical definitions, but into how we can put this in practice with doctors. We had a lot of examples for epidemiology and for COVID, where social media was tracked and it was shown that you can use it for health analysis. I personally have seen very few examples of this approach still being used by public agen­cies. Part of the challenge is explainability. What are you really analyzing? Can you prove to me that what we are analyzing is truly what we want to track? Because if I am a public health agency and I survey people, I know what I collected, even if people did not fully and correctly answer. Those are some challenges in terms of methods for digital traces.

There are two professors at Duke University (Cynthia Rudin) and at UBC (Margo Seltzer) that are building interpretable machine learning models for tabular data [⁹, ¹⁰] Tabular data is actually a big part of health data. Instead of giving the doctor a model without further information, they are finding a set of all equally performing models, or models performing within a reasonable boundary, and show the doctor all these alternative models. The doctor can then see which variables which model is using, and may choose the model accordingly. This seems to be one way of breaking this trustworthiness barrier.

Sanja Šćepanović: One should think about this, even if you do pure research. Your goal is in the end for it to be used.

Sanja Šćepanović: There are different challenges. With sensor data it is harder to collect enough data because you often need an ethical approval for your study, and then you need participants willing to participate and to consent. You can then collect data for a week or ten days maybe, but usually it is hard to collect over a longer period. My colleague Marios Constantinines is working a lot on this, and the key word they are using all the time is ‘in the wild’, which means: how can we make this go beyond collecting a week’s time and ten participants? Can we have devices which we can give to people to wear for a year or half a year and collect? This is not solved fully yet.

There are some studies which collected a lot, like the Cambridge University study with cou­gh­ing [¹¹], because it was during the COVID Pandemic and it was easy to get partici­pants.

In general, such studies are of smaller scale now and cover a shorter time span; as a consequence, the validity is always a limitation and we do not know how well our findings generalize. I can talk concretely about our work with the abdominal body sensor [8], where you need to put a sensor on people’s stomachs. This was again not easy, you really need to find students or colleagues who want to help. Even then when it comes to analy­zing, you throw out a big part of this data, because some people did not wear the sensor properly, or maybe they were not supposed to eat but they ate in the morning, or they just forgot it this day. There is a lot of data cleaning needed, and a lot of missing data to account for.

Sanja Šćepanović: The symptom extraction work that I have spent one and a half year to develop, we could now just do it with ChatGPT. It would be pricey, but that is the only problem. In the past I had to work really hard to label my data with Mechanical Turk or other crowdsourcing platforms, to train different models and to find the best one – I would then eventually have a model that extracts the symptoms pretty well, but not perfect. I believe now we could show the same text to ChatGPT and ask it to find all the symptoms. We could even ask it to do something which we could not achieve before, like dividing it into different symptoms that map to the different conditions.

This is called concept normalization. For example, alopecia means losing hair. So if some­one on Reddit says “My hair is thinning”, you would want to have those two linked to­gether. ChatGPT could actually do it, but I have not yet seen this work, and it would be quite important to evaluate this process in order to know that you can trust it. This could be applicable, but it is still expensive on a large scale, it might cost some money to pro­cess billions of Reddit posts, as we did in one project. The other set of applications that starts to appear and is being discussed is patients that used to go to Google to search are now going to ChatGPT to search. Some early works show that this apparently works well. It again brings the question of how much trust people can have in this.

Sanja Šćepanović: For some time in the past, there was this idea of using people and their web searches to study population health, which was adopted even by Google. They published a paper on it and even developed a platform, but then they had to shut it down again after a very short time (Google Flue Trends [¹²]). The reason was that they had fine-tuned their model to work on all the searches, thinking it is picking up only flu. But − and this goes back to explainability and interpretability of the models − the machine learning model also picked up on many other factors, e.g., since flu cases mostly happen in the season of winter, the model would connect other things that happen in winter to flu, and not just the symptoms of the illness. It was a failure. It initially worked well, and then really disagreed with the official statistics.

So we should be careful with what we are capturing. What is my advice there? I do not know, it is very tricky. Maybe my advice is to really try and think if you can explain or interpret your results, and try to double- or triple-check them. Be very honest about the limitations.

Also, a lot of the work we do is on population level, ecological studies. It might then be good to always discuss the type of study, and that for any true policy implications, this needs to be picked up by individual type of studies. We might propose some hypothesis, that there is some effect on health, but this would need to be validated by individual level studies. Our studies are a first step, but they are not a final step at the moment.

Sanja Šćepanović: In general, as I said in the beginning, I feel anything that deals with men­tal health is a bit more suited for this digital analysis, because of people being less stigmatized or maybe feeling more free to express themselves when they can do it anony­mous­ly. This gives more space to understand people’s well-being and health. But if you want to arrive at a precise understanding of some conditions, we really need additio­nal data sources, because people may not talk about all those conditions voluntarily and un­prompted.

For wearable data, I think well-being and stress is something that we should track, because these are less clearly defined by doctors. Proper diagnostics really require close collaboration with medical experts. One very important thing about these population health or across-the-country studies is that we always need to acknowledge whether or not our studies are − and I think 99 percent are − association type of studies, which means we do not find causal links. We simply cannot know if it was the environment that caused the increase in asthma percentage we found, or if it was something else. Causal methods, once they are suitable for our data, will be important.

Sanja Šćepanović: I think ethical approval, for the studies where it is needed, which is many of those using wearable, should really be thought of from the beginning, already when designing the study. If you are not experienced, it might be good to write an early email or have some conversation with your university’s ethical board, just so that they are able to steer you early on, showing you what is completely impossible.

The other thing is to really take care of not hurting or harming your participants, which can happen in those studies. This includes being really, really careful about the protec­tion of the data, for instance not sharing them unless it is strictly necessary. And also being transparent with the people about this.

There was a recent study at CHI [¹³], a conference in the field, and I was expecting that this type of conference would be adhering to these standards to the most, but a big percentage of papers did not have any consent forms. And a big percentage of the papers did not adhere to the principles of openness and transparency fully. This second part, which is somehow related, is sharing your code. You often cannot share data, but what you can and should share is your code.

Sanja Šćepanović: I think this is a very good question. Obviously, do not share data exposing individuals. Even when you share Twitter or Reddit data, it would be good to be careful to minimize the possibility of linking. Because this is already all public data, you could share the data, especially in some post-processed forms. You might still want to check with your supervisors or board. For the wearable data, there are already some data­sets which are made available for the community to develop models. Those are benchmarks which are carefully shared. For the data, the good thing is that there are frame­works like Datasheets for Datasets [¹⁴] or the Dataset Nutrition Labels [¹⁵], depen­ding on the type of your data.

Sanja Šćepanović: Two things. One is going from connecting one dataset and some outcome to connecting multiple datasets to build a fuller picture, a very multimodal approach. Starting from my PhD, I have been dreaming to do that, but it has been proven to be a bit challenging. People start to do it, but it is not easy to have complementary data. This is really going to change a lot, because sometimes we might discover one hypothesis in one dataset and in another, complementary dataset another hypothesis, and this might start sparking among us the discussion on why.

I can give you an example from another area I am studying; satellite indices for green areas. So you can derive green areas from satellite, it is called NDVI, a normalized diffe­ren­ce vegetation index [¹⁶]. There is lots of research, association studies, that have linked green areas to better mental health, less diabetes prevalence, less obesity, and so on. But then, for all of those studies, you find at least a couple – and sometimes even more – con­tra­dicting studies. We have found that sometimes it is not enough just to look at how green the area is, but that you need to disentangle where this green area is, and who has access to it.

It is this multimodal approach: A green area in the backyards is not that useful, but green areas on the streets, which anybody can access, are much more useful. Moving beyond these simple associations by combining different dataset and modalities, and thereby trying to reconcile inconsistent research findings, this type of multimodality is one area that I am very excited about. The second area is causality, which I already mentioned. A lot of our work is stifled in being applied because you will in the end get a question from policy makers if what you are finding is causal – because if it is not, we cannot apply it.