<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Luke Lefebure</title>
    <description>My blog and personal site</description>
    <link>https://llefebure.github.io/</link>
    <atom:link href="https://llefebure.github.io/feed.xml" rel="self" type="application/rss+xml"/>
    <pubDate>Sun, 24 May 2020 21:51:00 -0700</pubDate>
    <lastBuildDate>Sun, 24 May 2020 21:51:00 -0700</lastBuildDate>
    <generator>Jekyll v3.8.5</generator>
    
      <item>
        <title>Implementing Dynamic Bernoulli Embeddings</title>
        <description>&lt;p&gt;Dynamic Bernoulli Embeddings (D-EMB), discussed &lt;a href=&quot;http://www.cs.columbia.edu/~blei/papers/RudolphBlei2018.pdf&quot;&gt;here&lt;/a&gt;, are a way to train word embeddings that smoothly change with time. After finding the the &lt;a href=&quot;https://github.com/mariru/dynamic_bernoulli_embeddings&quot;&gt;paper authors’ code&lt;/a&gt; a bit challenging to use, I decided to create &lt;a href=&quot;https://github.com/llefebure/dynamic_bernoulli_embeddings&quot;&gt;my own implementation&lt;/a&gt;. This post walks through that process and showcases some results of training the model on the UN General Debate corpus.&lt;/p&gt;

&lt;h2 id=&quot;motivation&quot;&gt;Motivation&lt;/h2&gt;

&lt;p&gt;I &lt;a href=&quot;https://towardsdatascience.com/exploring-the-un-general-debates-with-dynamic-topic-models-72dc0e307696&quot;&gt;previously wrote&lt;/a&gt; about using Dynamic Topic Models to model the evolution of topics in the UN General Debate corpus. This analysis produced some interesting results that gave a nice overview of how persistent topics of global interest have changed over time. For example, the model captured the evolution on discussion about Africa from the end of colonialism, to apartheid, and now most recently to democracy and elections.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/Pyfig/dbe_africa.png&quot; alt=&quot;africa&quot; /&gt;&lt;/p&gt;

&lt;p&gt;I have been interested in this idea of modelling the change of topics and language over time, so I wanted to take this analysis a bit further. Specifically, any kind of topic model forces you to work with the high level topics that the model discovers, but I wanted to be able to slice the data at a more granular level with questions like:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;What terms have changed the most over time?&lt;/li&gt;
  &lt;li&gt;For a specific topic like “Syria”, can I chart it’s evolution through the years?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;deciding-on-the-d-emb-model&quot;&gt;Deciding on the D-EMB Model&lt;/h2&gt;

&lt;p&gt;There has been some research into the idea of word embeddings that evolve over time, such as &lt;a href=&quot;https://www.aclweb.org/anthology/W14-2517.pdf&quot;&gt;Temporal Analysis of Language Through Neural Language Models&lt;/a&gt; and &lt;a href=&quot;https://www.aclweb.org/anthology/P16-1141.pdf&quot;&gt;Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change&lt;/a&gt;. The former basically trains a Word2Vec model on each timestep, using as initial values the embeddings learned from the previous timestep. The latter builds on this idea slightly by adding an alignment step between the embeddings at different timesteps. The rationale for that is that Word2Vec can result in arbitrary orthogonal transformations that preserve meaningful cosine distance relationships between words in the same timestep but potentially confound any meaningful comparison across timesteps.&lt;/p&gt;

&lt;p&gt;After trying this approach and seeing subpar results, I turned to &lt;a href=&quot;http://www.cs.columbia.edu/~blei/papers/RudolphBlei2018.pdf&quot;&gt;Dynamic Embeddings for Language Evolution&lt;/a&gt;, the focus of this post. The authors of this paper have two main critiques for the above approaches. First, they require a lot of data for each timestep to train high quality embeddings (not the case in the UNGD corpus), and second, the necessity of having to do a weird post-training alignment step to make the dimensions of the embdedding comparable across timesteps is not ideal.&lt;/p&gt;

&lt;h2 id=&quot;the-model&quot;&gt;The Model&lt;/h2&gt;

&lt;p&gt;The model itself is quite simple with just two types of trainable parameters. The first is a separate embedding vector for each term in each timestep, and the second is a time independent context vector for each term.&lt;/p&gt;

&lt;p&gt;Training the model is conceptually very similar to training Word2Vec using negative sampling with one addition that encodes a smooth temporal transition for embeddings. This comes in the form of a penalty term on the difference between embedding matrices at different timesteps.&lt;/p&gt;

&lt;h2 id=&quot;implementation-notes&quot;&gt;Implementation Notes&lt;/h2&gt;

&lt;ul&gt;
  &lt;li&gt;
    &lt;p&gt;The embeddings are represented in a single matrix, with timesteps stacked on top of each other. Specifically, we have a tensor of shape &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;(V * T, k)&lt;/code&gt; where &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;V&lt;/code&gt; is the vocabulary size, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;T&lt;/code&gt; is the number of timesteps, and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;k&lt;/code&gt; is the dimension of the embedding. The first &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;V&lt;/code&gt; rows correspond to the embeddings from the first timestep, the second &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;V&lt;/code&gt; to the embeddings from the second timestep, etc.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;The implementation effectively concatenates all texts from a given timestep into one, treating it as a continuous stream of words. The impact of this is that the context for a target word at the very end of a text may include a few words from the beginning of some other unrelated text (within the same timestep). This is no problem if your texts are long, but it may be a problem with shorter texts. I may build in support for respecting such boundaries in the future.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;The range of values for the scaling factor on the random walk prior ($\lambda = [1, 10]$) that the paper cites are an order of magnitude different than what the defaults in the authors’ code suggests. I think this suggested range is supposed to refer $\lambda_0 = [1, 10]$ where $\lambda = 1000 * \lambda_0$ as this is more in line with the defaults in the code and what ended up working for me.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;The first epoch is used for initialization of the embeddings. To do this, I treat everything in the first epoch as coming from the first timestep and ignore the random walk prior term in the loss. When the epoch is finished, I initialize the embeddings for every timestep with those of the first timestep. The authors of the paper suggested a similar approach.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;10% of the documents are held out for validation by default. The only metric computed on the validation set is the $L_{pos}$ value, which is basically how good the model is at predicting the target word given the context. No negative samples are drawn for the validation set.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;results-on-the-general-debate-corpus&quot;&gt;Results on the General Debate Corpus&lt;/h2&gt;

&lt;h3 id=&quot;training&quot;&gt;Training&lt;/h3&gt;

&lt;p&gt;There are multiple components to the loss function that we’re minimizing, and the overall loss is a weighted combination of these.&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Positive: How good is the model at positively predicting the target word from the context?&lt;/li&gt;
  &lt;li&gt;Negative: How good is the model at negatively predicting the negative samples from the context?&lt;/li&gt;
  &lt;li&gt;Prior: How far are the embeddings drifting between timesteps?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;img src=&quot;/assets/Pyfig/dbe_2_0.png&quot; alt=&quot;png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Let’s look at how this decomposes. These three components all have their signs flipped when contributing to the loss, so we hope to see them increasing. The positive and negative curves both do so nicely, while the prior curve is decreasing. This means that the prior term is contributing more and more to the loss over time, despite the loss decreasing. Intuitively though, this makes sense because we want the embeddings to drift, just not too far. Thus, getting the scaling factor correct on this term is crucial to ensure quality embeddings within timesteps and meaningful comparisons across timesteps.&lt;/p&gt;

&lt;p&gt;Note the dips at the end of each epoch in the positive and negative curves. These are a consequence of the last batch being slightly smaller. They are normalized before contributing to the overall loss which is why the overall loss curve has no unusual behavior at the end of each epoch.&lt;/p&gt;

&lt;p&gt;Finally, we can look at the positive component on the validation set to ensure that the model is generalizing.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/Pyfig/dbe_4_0.png&quot; alt=&quot;png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/Pyfig/dbe_5_0.png&quot; alt=&quot;png&quot; /&gt;&lt;/p&gt;

&lt;h3 id=&quot;analysis&quot;&gt;Analysis&lt;/h3&gt;

&lt;p&gt;We can use the trained embeddings to do some cool analysis such as:&lt;/p&gt;
&lt;ul&gt;
  &lt;li&gt;finding which embeddings changed the most over time (absolute drift)&lt;/li&gt;
  &lt;li&gt;looking for change points where an embedding significantly changed from one timestep to the next indicating some significant event&lt;/li&gt;
  &lt;li&gt;looking up the nearest neighbors of an embedding and seeing how this “neighborhood” has changed over time&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Let’s start by looking at absolute drift, which is the euclidean distance between the embeddings of each term at the first and last timesteps. Terms that drifted the most indicate that their usage has shifted.&lt;/p&gt;

&lt;p&gt;Climate comes out on top, and European and Union appear which makes sense given the founding of the EU only in 1993 (and subsequent frequent usage of European and Union together). Millenium and Sustainable likely have to do in part with the advent of the Millenium and Sustainable Development Goals initiatives, and countries/regions that have seen significant change such as Syria and Africa also appear near the top.&lt;/p&gt;

&lt;div&gt;
&lt;style scoped=&quot;&quot;&gt;
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
&lt;/style&gt;
&lt;table border=&quot;1&quot; class=&quot;dataframe&quot;&gt;
  &lt;thead&gt;
    &lt;tr style=&quot;text-align: right;&quot;&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;Term&lt;/th&gt;
      &lt;th&gt;Drift&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;climate&lt;/td&gt;
      &lt;td&gt;1.241559&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;union&lt;/td&gt;
      &lt;td&gt;1.221671&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;south&lt;/td&gt;
      &lt;td&gt;1.136769&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;3&lt;/td&gt;
      &lt;td&gt;co&lt;/td&gt;
      &lt;td&gt;0.985926&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;4&lt;/td&gt;
      &lt;td&gt;agenda&lt;/td&gt;
      &lt;td&gt;0.983631&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;5&lt;/td&gt;
      &lt;td&gt;change&lt;/td&gt;
      &lt;td&gt;0.951607&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;6&lt;/td&gt;
      &lt;td&gt;post&lt;/td&gt;
      &lt;td&gt;0.912055&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;7&lt;/td&gt;
      &lt;td&gt;european&lt;/td&gt;
      &lt;td&gt;0.889426&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;8&lt;/td&gt;
      &lt;td&gt;southern&lt;/td&gt;
      &lt;td&gt;0.888166&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;9&lt;/td&gt;
      &lt;td&gt;twenty&lt;/td&gt;
      &lt;td&gt;0.878703&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;10&lt;/td&gt;
      &lt;td&gt;millennium&lt;/td&gt;
      &lt;td&gt;0.869878&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;11&lt;/td&gt;
      &lt;td&gt;sea&lt;/td&gt;
      &lt;td&gt;0.865753&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;12&lt;/td&gt;
      &lt;td&gt;africa&lt;/td&gt;
      &lt;td&gt;0.851388&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;13&lt;/td&gt;
      &lt;td&gt;serbia&lt;/td&gt;
      &lt;td&gt;0.823497&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;14&lt;/td&gt;
      &lt;td&gt;african&lt;/td&gt;
      &lt;td&gt;0.774140&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;15&lt;/td&gt;
      &lt;td&gt;sustainable&lt;/td&gt;
      &lt;td&gt;0.770657&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;16&lt;/td&gt;
      &lt;td&gt;summit&lt;/td&gt;
      &lt;td&gt;0.762487&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;17&lt;/td&gt;
      &lt;td&gt;north&lt;/td&gt;
      &lt;td&gt;0.756751&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;18&lt;/td&gt;
      &lt;td&gt;rule&lt;/td&gt;
      &lt;td&gt;0.756369&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;19&lt;/td&gt;
      &lt;td&gt;syria&lt;/td&gt;
      &lt;td&gt;0.752415&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;

&lt;p&gt;We can take this idea of measuring embedding drift further by finding the term/timestep pairs corresponding to the largest single timestep changes. I call these “change points”, and they correlate with significant real world events. For example, the Arab Spring begain in Tunisia in 2010 and Iranian appears twice in consecutive years in this list after details of the Iran-Contra Affair emerged.&lt;/p&gt;

&lt;div&gt;
&lt;style scoped=&quot;&quot;&gt;
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
&lt;/style&gt;
&lt;table border=&quot;1&quot; class=&quot;dataframe&quot;&gt;
  &lt;thead&gt;
    &lt;tr style=&quot;text-align: right;&quot;&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;Year&lt;/th&gt;
      &lt;th&gt;Term&lt;/th&gt;
      &lt;th&gt;Change&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;1993&lt;/td&gt;
      &lt;td&gt;ukraine&lt;/td&gt;
      &lt;td&gt;0.257642&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;2010&lt;/td&gt;
      &lt;td&gt;tunisia&lt;/td&gt;
      &lt;td&gt;0.232226&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;2014&lt;/td&gt;
      &lt;td&gt;japan&lt;/td&gt;
      &lt;td&gt;0.231548&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;3&lt;/td&gt;
      &lt;td&gt;1987&lt;/td&gt;
      &lt;td&gt;iranian&lt;/td&gt;
      &lt;td&gt;0.226282&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;4&lt;/td&gt;
      &lt;td&gt;1986&lt;/td&gt;
      &lt;td&gt;iranian&lt;/td&gt;
      &lt;td&gt;0.218671&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;5&lt;/td&gt;
      &lt;td&gt;1985&lt;/td&gt;
      &lt;td&gt;rwanda&lt;/td&gt;
      &lt;td&gt;0.207307&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;6&lt;/td&gt;
      &lt;td&gt;1993&lt;/td&gt;
      &lt;td&gt;lebanon&lt;/td&gt;
      &lt;td&gt;0.201297&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;7&lt;/td&gt;
      &lt;td&gt;2000&lt;/td&gt;
      &lt;td&gt;sudan&lt;/td&gt;
      &lt;td&gt;0.201279&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;8&lt;/td&gt;
      &lt;td&gt;1985&lt;/td&gt;
      &lt;td&gt;italy&lt;/td&gt;
      &lt;td&gt;0.201008&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;9&lt;/td&gt;
      &lt;td&gt;1999&lt;/td&gt;
      &lt;td&gt;sudan&lt;/td&gt;
      &lt;td&gt;0.200704&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;

&lt;p&gt;Finally, we can look at a term’s neighborhood in the embedding space across time. Let’s look at climate since it was found to have changed the most.&lt;/p&gt;

&lt;div&gt;
&lt;style scoped=&quot;&quot;&gt;
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
&lt;/style&gt;
&lt;table border=&quot;1&quot; class=&quot;dataframe&quot;&gt;
  &lt;thead&gt;
    &lt;tr style=&quot;text-align: right;&quot;&gt;
      &lt;th&gt;&lt;/th&gt;
      &lt;th&gt;1970&lt;/th&gt;
      &lt;th&gt;1975&lt;/th&gt;
      &lt;th&gt;1980&lt;/th&gt;
      &lt;th&gt;1985&lt;/th&gt;
      &lt;th&gt;1990&lt;/th&gt;
      &lt;th&gt;1995&lt;/th&gt;
      &lt;th&gt;2000&lt;/th&gt;
      &lt;th&gt;2005&lt;/th&gt;
      &lt;th&gt;2010&lt;/th&gt;
      &lt;th&gt;2015&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td&gt;0&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;winds&lt;/td&gt;
      &lt;td&gt;winds&lt;/td&gt;
      &lt;td&gt;winds&lt;/td&gt;
      &lt;td&gt;climatic&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;1&lt;/td&gt;
      &lt;td&gt;détente&lt;/td&gt;
      &lt;td&gt;equilibrium&lt;/td&gt;
      &lt;td&gt;relaxation&lt;/td&gt;
      &lt;td&gt;relaxation&lt;/td&gt;
      &lt;td&gt;dynamics&lt;/td&gt;
      &lt;td&gt;winds&lt;/td&gt;
      &lt;td&gt;radical&lt;/td&gt;
      &lt;td&gt;radical&lt;/td&gt;
      &lt;td&gt;radical&lt;/td&gt;
      &lt;td&gt;ipcc&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;2&lt;/td&gt;
      &lt;td&gt;equilibrium&lt;/td&gt;
      &lt;td&gt;détente&lt;/td&gt;
      &lt;td&gt;détente&lt;/td&gt;
      &lt;td&gt;détente&lt;/td&gt;
      &lt;td&gt;winds&lt;/td&gt;
      &lt;td&gt;climatic&lt;/td&gt;
      &lt;td&gt;climatic&lt;/td&gt;
      &lt;td&gt;climatic&lt;/td&gt;
      &lt;td&gt;climatic&lt;/td&gt;
      &lt;td&gt;winds&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;3&lt;/td&gt;
      &lt;td&gt;dynamics&lt;/td&gt;
      &lt;td&gt;dynamics&lt;/td&gt;
      &lt;td&gt;equilibrium&lt;/td&gt;
      &lt;td&gt;equilibrium&lt;/td&gt;
      &lt;td&gt;détente&lt;/td&gt;
      &lt;td&gt;radical&lt;/td&gt;
      &lt;td&gt;atmosphere&lt;/td&gt;
      &lt;td&gt;ipcc&lt;/td&gt;
      &lt;td&gt;ipcc&lt;/td&gt;
      &lt;td&gt;radical&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td&gt;4&lt;/td&gt;
      &lt;td&gt;relaxation&lt;/td&gt;
      &lt;td&gt;relaxation&lt;/td&gt;
      &lt;td&gt;dynamics&lt;/td&gt;
      &lt;td&gt;realms&lt;/td&gt;
      &lt;td&gt;momentum&lt;/td&gt;
      &lt;td&gt;engendering&lt;/td&gt;
      &lt;td&gt;ipcc&lt;/td&gt;
      &lt;td&gt;impacts&lt;/td&gt;
      &lt;td&gt;resistant&lt;/td&gt;
      &lt;td&gt;bioenergy&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;

&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;

&lt;p&gt;Like Dynamic Topic Models, dynamic word embeddings provide an interesting lens into how language usage changes over time. Applied to the UN General Debate corpus, we saw how they can tell us what terms are changing the most and point us directly to interesting change points that correlate with significant real world events.&lt;/p&gt;

&lt;p&gt;The code is easy to use, pip installable, and located &lt;a href=&quot;https://github.com/llefebure/dynamic_bernoulli_embeddings&quot;&gt;here&lt;/a&gt;. I hope it can be useful for others interested in similar exploratory analyses!&lt;/p&gt;
</description>
        <pubDate>Sun, 24 May 2020 00:00:00 -0700</pubDate>
        <link>https://llefebure.github.io/articles/2020-05/implenting-dynamic-word-embeddings</link>
        <guid isPermaLink="true">https://llefebure.github.io/articles/2020-05/implenting-dynamic-word-embeddings</guid>
        
        
        <category>Text</category>
        
      </item>
    
      <item>
        <title>Coronavirus Earnings Season</title>
        <description>&lt;p&gt;Axios &lt;a href=&quot;https://www.axios.com/coronavirus-global-economic-risk-stock-market-china-24da74a5-c824-4c79-a8c7-220213d28824.html&quot;&gt;reported&lt;/a&gt; several weeks back on growing concern from economists about the understated risks to the global economy posed by coronavirus. The report cites an interesting statistic about the attention paid to coronavirus on recent earnings calls.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Of the 364 companies that have held Q4 earnings calls, 138 cited the term “coronavirus” during the call, and about 25% of those included some impact from the coronavirus or modified guidance due to the virus, according to FactSet.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Additionally, the author cites a &lt;a href=&quot;https://www.guggenheiminvestments.com/perspectives/global-cio-outlook/coronavirus-impact-on-the-global-economy&quot;&gt;note&lt;/a&gt; in which Scott Minerd, global CIO of Guggenheim Investments, compares the “cognitive dissonance” of market participants to the assurances given by Neville Chamberlain to Britain on the eve of World War II.&lt;/p&gt;

&lt;p&gt;One day after the Axios report, the S&amp;amp;P 500 index hit an all time high before fears materialized and markets crashed. My mind came back to that statistic, so I pulled the transcripts for a quick analysis.&lt;/p&gt;

&lt;h2 id=&quot;data-acquisition&quot;&gt;Data Acquisition&lt;/h2&gt;

&lt;p&gt;The Motley Fool provides earnings call transcripts for the companies that they cover in a &lt;a href=&quot;https://www.fool.com/earnings-call-transcripts/&quot;&gt;centralized place&lt;/a&gt;, making it quite easy to scrape and parse. Global media outlets picked up the story in early January, so I pulled transcripts from all earnings calls occurring in 2020 (as of March 28th). This leaves us with data on 2,788 unique earnings calls.&lt;/p&gt;

&lt;h2 id=&quot;analysis&quot;&gt;Analysis&lt;/h2&gt;

&lt;p&gt;1,284 (46%) of the transcripts in this dataset include a mention of “covid” or “coronavirus”. However, there is a strong temporal trend which can be seen in the graphic below as recent earnings calls are far more likely to mention it than earlier ones.&lt;/p&gt;

&lt;p&gt;The first recorded mention was on 1/23 during the Q&amp;amp;A section of the American Airlines call.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;Hi. Hey, how are you? My question has to do with the coronavirus but are kind of on three fronts. I know it’s early, but are you guys seeing any booking impact at all? Have you contemplated any travel waivers for the region? And lastly, maybe this one is for Robert. What measures is American taking to protect passengers and crew? -&lt;i&gt;Dawn Gilbertson, Consumer Travel Reporter at USA Today&lt;/i&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;blockquote&gt;
  &lt;p&gt;Thanks, Dawn. So first off, we haven’t – it’s too soon to see any impact. Our network isn’t that extensive in Asia. But we’re on top of it. We’re working with CBP, the CDC and public health officials, as well as our medical resources here to make sure that we’re following all best practices. We’re doing that with an intent to make sure that we take care of our customers and team members. We’ve seen viruses in the past that we’ve had to make accommodations for and to be prepared for. We’re doing all those same things right now. And we’re going to watch it and make sure that we take aggressive action if there is a need to. -&lt;i&gt;Robert Isom, President&lt;/i&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The following week, several additional companies took note of the growing outbreak. In Apple’s call on 1/28, for example, Tim Cook addressed the issue in the context of Apple’s support for the local community.&lt;/p&gt;

&lt;blockquote&gt;
  &lt;p&gt;This quarter, we also announced a $2.5 billion plan to help address the housing availability and affording crisis in our home state of California. We feel a great responsibility to help the region we have always called home, stay vibrant and to ensure that it remains a great place for everyone to live and raise a family, including those who do so much to serve the community like firefighters and teachers. In much more recent news, we are closely following the development of the coronavirus. We are donating to groups that are working to contain the outbreak. We are working closely with our Apple team members and partners in the affected areas, and our thoughts are with all of those affected across the region. -&lt;i&gt;Tim Cook, CEO&lt;/i&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Throughout February, around half of calls included a mention, and that number has climbed to nearly 100% in March as state and local governments around the US began to lock down businesses.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/Pyfig/coronavirus-earnings-graphic_1_0.svg&quot; alt=&quot;svg&quot; /&gt;&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;The code for this analysis (including the script for scraping the Motley Fool) can be found &lt;a href=&quot;https://github.com/llefebure/misc/tree/master/coronavirus-earnings-calls&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
</description>
        <pubDate>Sun, 29 Mar 2020 00:00:00 -0700</pubDate>
        <link>https://llefebure.github.io/articles/2020-03/coronavirus-earnings-graphic</link>
        <guid isPermaLink="true">https://llefebure.github.io/articles/2020-03/coronavirus-earnings-graphic</guid>
        
        
        <category>Viz</category>
        
        <category>Text</category>
        
      </item>
    
  </channel>
</rss>
