Our Explainable Life: Symbolic Models for Political Forecasting

Transcript:

Brian Munz:

Hey everybody. Welcome to the NLP Stream, which is our weekly livestream to talk about all things related to NLP. I am Brian Munz, I’m a product manager at Expert.ai. And what we like to do is every week we have this live stream where we bring in guests to speak about something relevant to NLP and the world of language and AI. And sorry, I think we had a few technical difficulties, so there may be streaming this at a different place, but you may be joining us after the actual date, but we’ll be back next week as usual. But I wanted to jump right into it because we have really interesting topic with a very knowledgeable person this week. And so just want to jump right in and give the floor to Veronica Villa. Welcome.

Veronica Villa:

Thank you Brian. Hi everybody. I’m going to share my screen.

Brian Munz:

Okay. Oh, there we go.

Veronica Villa:

Can you see my presentation?

Brian Munz:

Yeah, I can see it.

Veronica Villa:

Perfect.

Veronica Villa:

So again, hello, everybody. Today I would like to talk about symbolic models for political forecasting with a focus on explainability. This all started with an article I wrote a few months back earlier this year about a personal experience developing a symbolic model for political forecasting. And after the reactions and talks I had with readers and colleagues, I decided to reframe my view about political forecasting and in order to understand how we should develop such symbolic models nowadays in the big picture of AI becoming mainstream. First of all, is it possible nowadays to develop a symbolic model for political forecasting? Is it possible? Is it preferable compared to machine learning? Or maybe the two approaches can be merged or can be pipelined somehow in order to reach good results. And what are the consequences? Is it even safe? What are we doing actually when we develop a political forecasting model? Is there a feedback we get from the real world we are trying to predict with our model? What are the implications?

Veronica Villa:

So I decided to present my thoughts about this issue and I think the problem should be approached, first of all, by understanding what we are trying to do when we decide to develop a model for political forecasting. So what is the scenario right now? We have surveys and polls, we have regression models based on statistics, we have prediction markets and we have the AI based models. So what users, people, the public nowadays get through social media and conventional broadcasting news is mash up of all these approaches.

Veronica Villa:

So I think it’s interesting first of all to understand what are these different approaches and how they work and how we can understand from them when developing our AI models. So surveys and polls are very popular and are currently, unfortunately, a bit unreliable as we could notice in the past few years. So we actually know why they are a bit unreliable. The first factor is the shy Tory factor. So we know from other areas of knowledge that normally when asked directly “what are you going to vote?” people tend to lie and especially conservative voters tend to lie. It’s a known phenomena. So it should be taken into account when producing survey and poll results, when presenting them and especially when interpreting them. Still, this is something very difficult to obtain. So how much should we account for the shy Tory factor when interpreting our surveys and polls results?

Veronica Villa:

So this is something that is interesting to know I think for everybody, not only for people working in this area but also for people at the other end of it. So reading or informing themselves about polls. And it’s also interesting for us developing AI models because the shy Tory factor and generally knowing that people lie when answering to direct questions about their vote is something that can be extremely useful when we develop symbolic code. So human readable code.

Veronica Villa:

As for regression models based on statistics, their reliability varies nowadays. Something we know is that according to econometrics, conservative models seem to be more accurate and regression diagnostics has improved dramatically. But still once again at the two ends of the process we need to realize that the reliability is variable. So we need to understand what we are looking at, what we are developing and what we’re looking at.

Veronica Villa:

As for prediction markets, they exist. So now in the world it exists such a thing where you can trade outcomes for money. So this can lead to controversial effect. So it can be to some extent considered gambling and it can lead to controversial situations like assassination markets. So trading outcomes based on actual dangerous actions can be by itself dangerous.

Veronica Villa:

So I’m mentioning this because I strongly believe that whatever is our approach be its surveys or regression models or AI or prediction markets, we do have an impact on the world itself. So maybe in AI, especially natural language processing, we were used to be quite a niche, so we were doing our things, developing our models and studying and doing research and development, but then suddenly we became mainstream and the world is looking at us and this has increased our impact and we need to understand that now we do have an impact just like maybe prediction markets and surveys. So people are checking out our results and their view can be modified by our results.

Veronica Villa:

In prediction markets this is particularly dramatic, especially because based on the efficient market hypothesis, when you bet money or I don’t want to say that, when you trade outcomes for money, you are assuming that the price actually reflects all the elements of your hypothesis, which we don’t really know if this is correct. I mean if it’s actually like that or it’s just a hypothesis, but still we need to take into consideration that what we are doing in a prediction market is we are trading our outcomes and this has an impact. And if we base our actions on the efficient market hypothesis, taking it for granted as an objective fact, we might be communicating something that we consider to be the truth. So at the other end of the process, the public, the users, people who are checking out our results or our market, their views might be modified by what we do.

Veronica Villa:

So in AI based models we are exploring uncharted territories right now. So that’s bare innovation. So right now we have machine learning which is growing dramatically and we have symbolic approaches and we have composite approach where we try to use both approaches together and try to profit from them both and to balance them. My personal experience is in the symbolic area, but I’m trying to experiment with machine learning. It’s very interesting to me, especially in terms of pipelining my models but my field of expertise right now is symbolic models.

Veronica Villa:

So first of all, if we want to approach political forecasting, we need to understand our use case. So these are a few examples. The first example I mentioned is the Brexit because is the one where I actually developed a model with actual results that we will be discussing later. New Italian government, so very fresh. New UK prime minister and USA midterm elections. So political forecasting, very different use cases. What are the main elements?

Veronica Villa:

In Brexit, we do have two sides. This makes things much easier because two sides you can set up a model, it can be a categorization model or an extraction model. It really depends on what you prefer in your development, but it’s still two sides. So it’s theoretically easier. But when you have very polarized voters you have a high risk of mistakes in your natural language processing because when voters are very polarized, they vocalize themselves very differently and the shy Tory factor might be more important and voters, especially referendum voters who are for the yes tend to be much more vocal. So in natural language processing, all this must be taken into account in order to code correctly and obtain relevant results. Also, an element in the Brexit referendum was the long campaign. So how long are you going to monitor the trends in your data in order to obtain relevant results? This still is the Brexit use case which is really approachable in my opinion because of the two sides and because we know how polarized voters express themselves linguistically and also long campaign really… It has its drawbacks, but it also allows us to decide how long should be our monitoring of the data.

Veronica Villa:

While the new Italian government scenario is very difficult in my opinion to approach. So actually we did not develop a model for this scenario because there is very scattered parties, uncertain alliances. We got the official parties with the logo and everything and the programs at the beginning of September, so less than one month before the actual elections. So people, voters did not have enough time, I don’t want to say to decide for the vote, but to discuss on social media or to discuss in a relevant way apart from just fighting. There is a strong problem with undecided voters and we also had low attendance which was expected and also we are experiencing worldwide crisis, post covid and war and energy. So we all doubt this scenario as not ideal for developing a model.

Veronica Villa:

Same applies to the new UK prime minister. So citizens don’t vote as you know, it’s ballot stages and it’s the political party in charge that decides the new prime minister. Again, crisis and a lot of resignations during the process. So this is why we decided not to develop a model for this scenario. And we are not developing a model for the USA midterm selections. Very different scenario. So it’s federal and state and local and tribal elections at the same time. So there are two sides really, but they can be inferred from the general database, but it’s very complicated.

Veronica Villa:

What I see here as a potential is that it really depends on the year. So I think that some years midterm elections are, people are, voters are grading the party in power while other years they are voting for future politics. And in my opinion this is the case this year because there are huge urgent issues, internal and worldwide. So it’s very interesting scenario. But I think that here instead of trying to predict the winning party, we could try to mine for specific trends. So finding specific trends maybe on specific internal or worldwide issues and monitoring those trends, that would be interesting. So it’s really crucial that we understand our scenario and that we select correctly a scenario where we do have a potential for our natural language understanding model.

Veronica Villa:

So as for the Brexit, this is the article I wrote earlier this year towards data science. It’s more storytelling piece compared to a technical article. It tells the story of how I ended up developing a symbolic model for the Brexit referendum. So that was, of course, back in 2016. So I was working in a European funded project under the umbrella of the [inaudible 00:19:19], which is a program for knowledge sharing between academia and industry. So I was there for Expert.ai and we were working together with colleagues from other companies and together with professors and students at the University of Aberdeen. And it was quite a long project actually. And there were a couple of articles and also a book related to the project. And at the very end of the project we decided to showcase our work by developing a small model for predicting the Brexit outcome. And I was in charge of developing the symbolic engine.

Veronica Villa:

So this article I wrote tells the story of my personal experience developing this model and especially regarding the biases I was suspecting in my code and I was maybe able to define in the end. So like I was saying before, I decided to reframe my considerations and in order to make them relevant in the current scenario. So first of all, I would like to tell something more about how the model was actually developed and especially I would like to point out how it is fully explainable and what this means for us right now at this stage of artificial intelligence developing.

Veronica Villa:

So I use Twitter as a source. So we decided together in the project to use Twitter as a source because the Twitter API allow us quite easily to retrieve the data and to save them and to monitor them for days and weeks and months. And also, we decided for Twitter because of the nature of the language on Twitter. So sure to the point sentences. This is not always an advantage having sure to the point sentences. Sometimes when you develop other kinds of models, you really want to have a text. So a fully developed text with strong sentences, with long considerations and with an organic treatment of the issue. But in this case we decided that having single sentences was more suitable. And also, Twitter is very strong in the use of hashtags and hashtags are extremely useful when you develop symbolic engines. It’s maybe a little bit counterintuitive, but it’s exactly like that. So you can count hashtags in symbolic models exactly like you would do with a statistic model. And in addition to this you can develop linguistic rules that take hashtags into considerations together with more complicated linguistic conditions. So we decided that Twitter was the ideal source for our data and we decided to develop a categorization engine as opposed to an extraction engine. Completely rule based.

Veronica Villa:

So our code can be accessed, discussed, modified, reused and hits can be shown. This is very important in my opinion because hits been shown is very useful when you develop but I think they do have a potential in explainability even for the end user, maybe not in this specific scenario, but broadly speaking, when developing symbolic engines being able to show where the linguistic rules have hit can be crucial and in my opinion is a huge point in not only marketing symbolic models, but also making them really useful not only for a customer but for a user in general.

Veronica Villa:

This model relies heavily on the knowledge graph. So the knowledge graph is a linguistic knowledge base we use in Expert.ai. It’s a very mature graph. We’ve been using it for more than 20 years now. It contains concept. So concepts are linked together according to the meaning and it’s very rich in lexicon, in vocabulary, in jargon. And it’s the core of Expert.ai linguistic technology together with this ambiguous engine. So it’s a very strong core we can rely on when we develop rule based engines because, of course, we are going to use keyword words in our rules. But mostly, especially in this kind of use case, we are using concepts quite a lot. Personally, I used mostly concepts in developing this engine both for understanding concepts like positive or negative opinion on the two sides of the referendum, but also in coding concepts related to the different issues.

Veronica Villa:

So for example, in the Brexit discussion online there were some key issues, for example, immigration, taxes. Those were the two main points. So it’s really important when you develop a rule based engine to be able to retrieve from a strong core technology where you have access on concepts. So what we did was monitoring trends over weeks, over six weeks as I remember correctly. So we would daily acquire the data from Twitter using the API and monitor them in order to find trends. And what happened was that the trends were very stable, which surprised me, especially because the trend was clearly towards yes for Brexit, while all the polls and other kinds of analysis that were being shared, both on social media and on conventional broadcasting were for a no outcome. So it was a little bit unsettling because, of course, being out of the chair can be uncomfortable, but at the same time every day we would get a yes for the Brexit from this monitoring.

Veronica Villa:

So a little bit more on the knowledge graph. It’s pre-trained and each node is a concept. Pre-trained by humans. So when I joined the company 20 years ago, I was able to meet the people who are actually training the knowledge graph and this graphic here is very realistic. So that’s a concept in our knowledge graph and it’s connected to other concepts. So that’s the core of my model. So when we talk about symbolic engines and rules, of course, rules are crucial, but the core technology is much more crucial, I believe, because it enables us to develop strong rules, reliable rules.

Veronica Villa:

Linguistics rules are readable especially if they are developed in a language that has a low access point, so you can read it even if you’re not a programmer. You can really share knowledge through the rules. So linguistic condition allow us meaningful human control on our engine. They allow us to code into the model. The lexical markers, we decide to use. So the lexical markers that according to our analysis are signals for a yes or a no in a referendum. And they also allow us to code sociolinguistic criteria into our code. So for example, we can take account of the shy Tory factor because we are in full control of the linguistic conditions we are writing.

Veronica Villa:

Also, we can take into account of the undeutsch hypothesis. So according to the undeutsch hypothesis, when people lie, they actually use different language constructs and in lexical as opposed to when they are telling the truth. And this is something that we can recognize to some extent when we read a text and when we are able to recognize a linguistic signal of the undeutsch hypothesis in our text, then we are able to code it into a linguistic role. So in the end, our model automatically will be able to retrieve that knowledge we have coded into it.

Veronica Villa:

Same applies to Stylometry. Also, this engine included embedded statistics scores. So when all triggers is activated, so when the linguistic condition is true in my text, a score is added to my category. So automatically when a lot of rules trigger, my score is higher. So it’s very simple embedded statistics, but it’s very reliable and relevant. And again, we can control it, we can change the score, change the weights. So it’s a layer that allows us to be more precise and to have more control on our model.

Veronica Villa:

So what about machine learning? Back then when I developed this model, I wasn’t working with machine learning yet, it was still in the making and this led me to think in the past couple of years, would I make it different today? So would I incorporate machine learning algorithms or maybe I would create a composite model? So I think according to my experiments in the past few months, a composite model can be especially effective in terms of having a composite pipeline. So here we have a taxonomy with two categories. We have linguistic rules relying on the knowledge graph. So what I’m experimenting right now is adding linear SVM algorithms in the loop because it works very well when you have two categories and also because in our technology you can rely on the knowledge graph when you are feature engineering.

Veronica Villa:

So when you are setting your machine learning algorithm, for example, the linear SVM in this case, you have access to the knowledge graph. So you can decide, you can toggle, you can decide, I want to use concepts or not, I want to keep two keywords or maybe [inaudible 00:33:32] so I can make decisions. And my pipeline is a composite one so there are linguistic rules that are falling under my control and there are machine learning algorithms that work autonomously, but still I can code, not really code, I can set them in my feature engineering so that they can talk to the knowledge graph, which I think is a crucial element of our pipelines. So this is what I am experimenting right now and I would like to find a good use case for this in order to experiment better. So like I show you the Italian political voting and the UK scenario and the USA scenarios are not ideal right now, but that’s my goal in the future. So incorporating machine learning algorithms in a pipeline.

Veronica Villa:

So the biases. My article towards data science was strongly focused on the biases that my model might have been infected with. So I think we must be very honest with ourselves when coding and also honest when reading our own outcomes. So in the case of the Brexit model, I asked myself, “Is my source biased?” Because I was using Twitter, so social media, demographics, geographics, hashtags and filter bubbles do play a strong part in the scenario. So by using the Twitter API, we selected the geographics very carefully. So we were careful to select the same number of tweets from Scotland, from Wales and from England and from Northern Ireland. Locating demographics is much more difficult and also filter bubbles can be tricky, but we were aware of such elements. So what we were prepared to was having in our trend shifting and we told ourself, okay, when the trend shifts it’s probably going to be because of a filter bubble, so for demographic issues, but the trend remains stable. So we never really actually came to the point of considering geographics and filter bubbles.

Veronica Villa:

In the end, what I think was the main problem of my model was my invent, so what I had around me as a coder. So of course, I had a selective exposure to the Brexit itself. So this happens to everyone. You cannot really avoid it because you have a workplace, you have a family, you have friends, you have your circumstances. So you end up to some extent being in an echo chamber and you end up wanting confirmation biases for yourself. So when you are coder, this can change things because you might be coding your biases into your linguistic rules without even noticing. So in my personal experience, I think that because I was mainly surrounded by brexiters, I ended up learning their language better. And because I knew how they talked, because I was surrounded by them, I was able to write better rules for the yes, while I never got to talk to someone who was for the no, or very few people were against Brexit where I was living back then. So I didn’t have the chance to learn the language. I could see their tweets, I could see them being very vocal on social media, but in a little bit stereotypical way. So I think in the end I wrote better rules for the yes. So that’s I think, in the end, the bias of my code, which ended up to be the reason why it succeeded. So we were able to predict the Brexit output correctly.

Veronica Villa:

And in the end I think okay, it was a bias. You don’t want biases in your code, but if no people were actually the majority, or at least the majority of those who voted it’s normal that I was surrounded by them. Still, ideally, I would’ve preferred to be able to learn from both the voting groups and to develop good rules for both. So that’s what happened in my opinion. And it’s interesting to know, I mean it’s interesting as a coder to be able to find your own biases. And also your pipeline can be biased. So especially this is something that worries me in terms of developing composite models. So what am I going to do? Am I going to give more weight to the machine learning part or to the symbolic part? Which one should come first or just come first in the pipeline? I don’t want a short circuit and that’s what I’m trying to understand better.

Veronica Villa:

Also, we need to understand that also the interpretation of the results can be biased. This is something that was very clear to me in the next days after the Brexit because after the first shock, all social media and also the news, they started to give a very standard explanation of what happened and which in my opinion was a little bit limited compared to what was happening in the real world. I mean I was in the UK back then, so I was surrounded by it and I found that the depiction in the media was a little bit limited. So peer reviewing is extremely important. Also, understanding the difference between factuality and intention, and again, understanding that everybody’s prone to confirmation bias. It’s just the way our mind works.

Veronica Villa:

We do have responsibilities in the end. So right now we have the hype. We are no longer in a niche. We had the hype. AI is perceived as objective. The narrative in social media is a lot towards tech solutionism, I find and tech determinism, which worries me a little bit. I mean I should be happy about it because I work in the industry, but at the same time it worries me a little bit. So I think we should be careful about that, all of us but also as coders.

Veronica Villa:

Storytelling, narrative, abstraction. Many people in the public is expecting a crystal ball forecast. And I ask myself, do I want to give a crystal ball forecast? Maybe, maybe not. And I thought about it quite a lot. And I think that a black box can be excellent in some situations, but we should also consider that there are other options. So right now in Expert.ai we have this concept, the green glass. So understanding the limits of the black box and being able to be explainable and to show what we are doing and to make people understand what we are doing and why our outcomes are what they are. Because we are part of the social machine this is something that we must accept that we do have a responsibility and we need to act accordingly. Maybe the user will want a crystal ball forecast, but in some cases we should insist on explainability.

Veronica Villa:

We are trying to reach transparency and accountability. And what we are also trying to do is avoid algocracy. Algocracy reinforces the status quo. It reinforces biases. It limits digital self-determination in the end user and it uses individualization, it produces anxiety. So I think we should be careful around our own models.

Veronica Villa:

What is the value of explainability? In my opinion, I think symbolic model has the great advantage of being explainable, but it’s not only an ethical problem, I don’t want to be explainable because that’s good in itself. I want to be explainable with my symbolic model because I think that’s a constructive approach that can provide added value. I was mentioning the highlighted hits of my linguistic rules. So I’m recognizing as a coder the linguistic signals in the text. I’m coding them into my linguistic rules and while I work I can highlight the hits. It’s very useful for me in my developing. But what if I highlight the hits also for the end user? So let’s imagine a linguistic engine where the end user is able to look at the text and instead of having a crystal ball outcome, yes, no, or this party, the other party, whatever, the user gets the text highlighted where the linguistic rules trigger. This is information. It’s also providing evidence of how I got the final result, which can still be a crystal ball result of yes or no. And it’s also improving comprehension and autonomy and promoting media literacy.

Veronica Villa:

This is particularly relevant for people who are developing fake news detection models. I think Jose Peres is talking about this subject and doing the NLP streams, and I find that his work on this area is extremely interesting. In my opinion, when you are able as a final user to see how the engine was able to obtain a certain result, you learn by the process and you become more autonomous. So for example, in fake news recognition, we know that there are linguistic signals. For example, emotional language or using special punctuation. So if I can highlight such linguistic signals, I’m providing added information for the end user. So I’m very focused on this aspect of the model. So providing not only the bay results, but also highlighting the text where the linguistic rules have triggered, where they are true in the text because I think that they can provide real value of explainability. So I’m not explaining how my model works because I think it’s ethically good in itself, but because it allows me to provide value, information and constructive information that the end user can rely on. The next time they read an article, the next time they read a text maybe they will be able to understand it better.

Veronica Villa:

So thank you very much for being with me in this presentation. I hope you find found it interesting.

Veronica Villa:

Brian, what about the feedback?

Brian Munz:

It was very interesting. There’s a few things that are always interesting when you read about, especially politics. One is that, of course, I think it really highlights the complexity of human beings because like you said, it’s not as simple as, I may have a perception in my head that everyone who lives in a certain part of the country, of course, is going to feel a certain way about a certain topic, and then next thing you know, everyone is surprised, including the media that it’s been completely different. And so it’s just very interesting to see the complexities and how in some ways NLP can cut through that and in other ways you need to have a human being who is kind of saying, “Well let’s not put everything onto this side.” Right?

Veronica Villa:

Exactly. In natural language processing, we are trying to simplify complex problems, of course, because we are drowned in data. So we really need tools to help us make sense of the data and to stream them, to organize them. But at the same time, we don’t want to make complex problems simplified. We want to understand their complexity. So we want to simplify them in terms of making them more understandable, not in terms of eliminating complexity because if we eliminate the complexity, we are no longer able to understand. So we need tools. We need tools that allow us to deal with all this huge amount of data, but to deal with them autonomously to some extent. So we really don’t want to give the model the full control of our decisions and of our understanding of the world. We want the tool to help us understand the world, to make us more able to understand the world.

Brian Munz:

Right. It’s interesting too, because you touched on it in the last slide about how it sort of educates people where I think the younger generations, when I see my kids and stuff, they seem to be more aware of fake news and how people are trying to manipulate them online. And so it’s interesting that NLP can kind of have a role in that to say, someone can go and am I being scammed? And then there’s different NLP ways that it’ll kind of educate a person, well, if you see this particular tactic, then maybe it’s fake.

Veronica Villa:

Exactly. Highlights. So something telling you this is this and this is what highlights that help you understand.

Brian Munz:

And then it’s not a dependency as much as you’re learning. And then NLP can help, of course, sift through a lot of obvious junk and things. So makes sense.

Brian Munz:

Well, I mean, I could talk about this for hours, but I think it was a very interesting presentation. It’s always interesting to hear, especially in the middle of election season when it’s nonstop people putting their predictions forth where I don’t know how useful that always is, like you said, in more complexities. But definitely was interesting and we’d love to have you back another time to talk about something else. So thanks for sharing.

Veronica Villa:

Thank you for having me.

Brian Munz:

And so next week we are going to be talking about looking towards sustainable technology, green approaches to NLP, something that Walt Mayor touched on when he was here a few weeks ago. But we hope to see you all there next week, same time. And thanks again Veronica for talking.

Veronica Villa:

Thank you.

Brian Munz:

Thanks. Bye.

Veronica Villa:

Bye.