3. Combined filtering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last updated: July 5, 2023\n",
    "\n",
    "Last run: July 5, 2023\n",
    "\n",
    "**Additional filtering based on data (as described in the published study)**\n",
    "\n",
    "## Unreviewed science in the news: The evolution of preprint media coverage from 2014-2021\n",
    "\n",
    "\n",
    "Juan Pablo Alperin\n",
    "\n",
    "**Related Publication:**\n",
    "Fleerackers, A., Shores, K., Chtena, N. & Alperin, J.P. (2023). Unreviewed science in the news: The evolution of preprint media coverage from 2014-2021. *bioarxiv*. \n",
    "\n",
    "**Related Dataset:**\n",
    "Alperin, Juan Pablo; Fleerackers; Shores, 2023, \"Data for: Unreviewed science in the news\", https://doi.org/10.7910/DVN/ZHQUFD, *Harvard Dataverse*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprint filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/top outlets/preprints_mention_details_with_dates.csv')\n",
    "\n",
    "df['posted_on'] = pd.to_datetime(df.posted_on)\n",
    "\n",
    "df['best_published_pub_date'] = pd.to_datetime(df['best_published_pub_date'])\n",
    "df['best_preprint_pub_date'] = pd.to_datetime(df['best_preprint_pub_date'])\n",
    "\n",
    "preprints_original = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40,039\n"
     ]
    }
   ],
   "source": [
    "N = df.shape[0]\n",
    "print(\"{:,}\".format(N))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove everything before 2013 that appeared from problematic dates\n",
    "df = df[df.best_preprint_pub_date >= '2013']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3,619\n"
     ]
    }
   ],
   "source": [
    "print(\"{:,}\".format(N-df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "165\n"
     ]
    }
   ],
   "source": [
    "# filter those that appear to actually be postprints, not preprints\n",
    "df = df[(df.best_published_pub_date.isnull()) | (df.best_published_pub_date > df.best_preprint_pub_date)]\n",
    "\n",
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36,255\n"
     ]
    }
   ],
   "source": [
    "# remaining\n",
    "print(\"{:,}\".format(df.shape[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# date data too uncertain to decide if preprint or postprint (dates too closer together)\n",
    "# 7 days \n",
    "\n",
    "df = df[(df.best_published_pub_date.isnull()) | (abs((df.best_published_pub_date - df.best_preprint_pub_date).dt.days) > 7)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "332\n"
     ]
    }
   ],
   "source": [
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The news media mentioned happened more than 5 days before the preprint was published\n",
    "\n",
    "df = df[df.days_since_preprint >= -5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "327\n"
     ]
    }
   ],
   "source": [
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mention appears to be to the postprint, not preprint\n",
    "\n",
    "df = df[(df.days_since_publication.isnull()) | (df.days_since_publication < -5)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3,547\n"
     ]
    }
   ],
   "source": [
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1,021\n"
     ]
    }
   ],
   "source": [
    "# We have the same news story for the same DOI/arxiv_id more than once\n",
    "df = df.drop_duplicates(subset=['doi', 'arxiv_id', 'outlet', 'moreover_url'])\n",
    "\n",
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('data/top outlets/preprints_mentions_final.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprints = df.copy()\n",
    "del df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of preprint filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "31,028 mentions (77.5%)\n",
      "11,538 preprints (76.7%)\n",
      "25,249 stories (80.8%)\n",
      "\n",
      "In total, filtering led to the exclusion of 9,011 mentions (22.5% of original dataset)\n",
      "\n",
      "Our final preprint sample comprised 31,028 mentions of 11,538 preprints in 25,249 stories published by the 99 outlets in our sample (a preprint could be covered in several stories and a single story could mention multiple preprints)\n"
     ]
    }
   ],
   "source": [
    "N_mentions = preprints.shape[0]\n",
    "N_mentions_original = preprints_original.shape[0]\n",
    "\n",
    "N_preprints = preprints.altmetric_id.unique().shape[0]\n",
    "N_preprints_original = preprints_original.altmetric_id.unique().shape[0]\n",
    "\n",
    "N_stories =  preprints[['outlet', 'moreover_url']].drop_duplicates().shape[0]\n",
    "N_stories_original =  preprints_original[['outlet', 'moreover_url']].drop_duplicates().shape[0]\n",
    "\n",
    "N_outlets = preprints.outlet.unique().shape[0]\n",
    "\n",
    "print(\"{:,} mentions ({:.1f}%)\".format(N_mentions, 100*N_mentions/N_mentions_original))\n",
    "print(\"{:,} preprints ({:.1f}%)\".format(N_preprints, 100*N_preprints/N_preprints_original))\n",
    "print(\"{:,} stories ({:.1f}%)\".format(N_stories, 100*N_stories/N_stories_original))\n",
    "print()\n",
    "print(\"In total, filtering led to the exclusion of {:,} mentions ({:.1f}% of original dataset)\".format(\n",
    "    N_mentions_original-N_mentions, 100*(N_mentions_original-N_mentions)/N_mentions_original))\n",
    "print()\n",
    "print(\"Our final preprint sample comprised {:,} mentions of {:,} preprints in {:,} stories published by the 99 outlets in our sample (a preprint could be covered in several stories and a single story could mention multiple preprints)\".format( \n",
    "     N_mentions, N_preprints, N_stories))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WoS filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_file = 'data/top outlets/wos_with_doi_mention_details.csv'\n",
    "# columns = ['Altmetric_ID', 'Order', 'DOI', 'Arxiv_ID', 'First_Seen_On', 'PubDate', 'wos_year', 'wos_title', 'Author_Name', 'Author_Url', 'Posted_On', 'Title', 'moreover_url']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1,657,202\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(input_file)\n",
    "df.columns = [x.lower() for x in df.columns]\n",
    "df.rename(columns = {'author_name': 'outlet', 'author_url': 'outlet_url'}, inplace=True)\n",
    "df['posted_on'] = pd.to_datetime(df.posted_on)\n",
    "df['posted_on_year'] = df.posted_on.map(lambda x: x.year)\n",
    "\n",
    "print(\"{:,}\".format(df.shape[0]))\n",
    "\n",
    "articles_original = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove everything before 2013, to match what we did with preprints because of problematic dates\n",
    "N = df.shape[0]\n",
    "df = df[df.wos_year >= 2013]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "156,187\n"
     ]
    }
   ],
   "source": [
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merging WoS with Preprint data to find duplicates, etc.\n",
    "merged = df.set_index(['altmetric_id', 'order']).merge(preprints[['altmetric_id', 'order', 'posted_on']].set_index(['altmetric_id', 'order']), \n",
    "         left_index=True, right_index=True, \n",
    "         how='outer', \n",
    "        suffixes=['', '_p'],\n",
    "         indicator='source'\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing 579 WoS mentions that are in the preprint mentions folder\n"
     ]
    }
   ],
   "source": [
    "print(\"Removing {:,} WoS mentions that are in the preprint mentions folder\".format(merged[merged.source == 'both'].shape[0]))\n",
    "\n",
    "df = merged[merged.source == 'left_only']\n",
    "del df['source']\n",
    "del df['posted_on_p']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14,482\n"
     ]
    }
   ],
   "source": [
    "# We have the same news story for the same DOI more than once\n",
    "df = df.drop_duplicates(subset=['doi', 'outlet', 'moreover_url'])\n",
    "\n",
    "print(\"{:,}\".format(N - df.shape[0]))\n",
    "N=df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1,486,533\n"
     ]
    }
   ],
   "source": [
    "print(\"{:,}\".format(df.shape[0]))\n",
    "df.to_csv('data/top outlets/wos_mentions_final.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = df.copy()\n",
    "del df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of WoS filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1,486,533 mentions (89.7%)\n",
      "11,538 preprints (76.7%)\n",
      "1,084,048 stories (97.4%)\n",
      "\n",
      "In total, filtering led to the exclusion of 170,669 mentions (10.3% of original dataset)\n",
      "\n",
      "The final published research sample comprised 1,486,533 mentions of 397,446 distinct research outputs across 1,084,048 stories published by the 94 outlets in our sample (a research output could be covered in several stories and a single story could mention multiple outputs)\n"
     ]
    }
   ],
   "source": [
    "N_mentions = articles.shape[0]\n",
    "N_mentions_original = articles_original.shape[0]\n",
    "\n",
    "N_articles = articles.doi.unique().shape[0]\n",
    "N_articles_original = articles_original.doi.unique().shape[0]\n",
    "\n",
    "N_stories =  articles[['outlet', 'moreover_url']].drop_duplicates().shape[0]\n",
    "N_stories_original =  articles_original[['outlet', 'moreover_url']].drop_duplicates().shape[0]\n",
    "\n",
    "N_outlets = articles.outlet.unique().shape[0]\n",
    "\n",
    "print(\"{:,} mentions ({:.1f}%)\".format(N_mentions, 100*N_mentions/N_mentions_original))\n",
    "print(\"{:,} preprints ({:.1f}%)\".format(N_preprints, 100*N_preprints/N_preprints_original))\n",
    "print(\"{:,} stories ({:.1f}%)\".format(N_stories, 100*N_stories/N_stories_original))\n",
    "print()\n",
    "print(\"In total, filtering led to the exclusion of {:,} mentions ({:.1f}% of original dataset)\".format(\n",
    "    N_mentions_original-N_mentions, 100*(N_mentions_original-N_mentions)/N_mentions_original))\n",
    "print()\n",
    "print(\"The final published research sample comprised {:,} mentions of {:,} distinct research outputs across {:,} stories published by the {:,} outlets in our sample (a research output could be covered in several stories and a single story could mention multiple outputs)\".format( \n",
    "     N_mentions, N_articles, N_stories, N_outlets))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A total of 1,486,533 mentions of 397,446 publications across 1,084,048 news stories.'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"A total of {:,} mentions of {:,} publications across {:,} news stories.\".format(articles.shape[0], articles.doi.unique().shape[0], articles[['outlet', 'moreover_url']].drop_duplicates().shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}