Rachael Stickland
📆 🚧 💻 📖 🤔
Mahwish Mohammad
🚧 💻 📖 🤔 👀
Batool Almarzouq
👀 🤔
Ann-Marie Mallon
📆 🤔
Kirstie Whitaker
🤔
Rachael Stickland 📆 🚧 💻 📖 🤔 |
+ Mahwish Mohammad 🚧 💻 📖 🤔 👀 |
+ Batool Almarzouq 👀 🤔 |
+ Ann-Marie Mallon 📆 🤔 |
+ Kirstie Whitaker 🤔 |
+
![\"cprd-aurum-data-structure\"](\"cprd-aurum-data-structure.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Preview the `Medical dictionary` and associated lookup table\n",
+ "\n",
+ "\"The Medical Dictionary contains information on all medical history observations that have been recorded in EMIS Web®. Observations are coded using a combination of SNOMED, Read and local EMIS® codes. Further information is provided in later sections of this document.\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "\n",
+ "- Links to the `Consultation` and `Observation` data tables on 'medcodeid'\n",
+ "- Links to the `EMISCodeCat` lookup table on 'emiscodecategoryid'\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- MedicalDictionary table\n",
+ "SELECT * FROM MedicalDictionary \n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- EMISCodeCat lookup table\n",
+ "SELECT * FROM EMISCodeCat\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "source": [
+ "### Tip: Execute SQL cells more than once because 'ORDER BY RANDOM()' will show you different data each time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Preview the `Product dictionary` \n",
+ "\n",
+ "\"The Product Dictionary contains information on drug and appliance prescriptions recorded in EMIS Web®. This information is coded using the Dictionary of Medicines and Devices (DM+D). Further information is provided in later sections of this document.\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "\n",
+ "- Links to the `Drug Issue` data table on 'prodcodeid'\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- ProductDictionary table\n",
+ "SELECT * FROM ProductDictionary \n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Preview the data tables and associated lookup tables\n",
+ "\n",
+ "### `Patient` table \n",
+ "The `Patient` table \"contains basic patient demographics and patient registration details for the patients.\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "\n",
+ "- Links to the `Practice` data table on 'pracid'\n",
+ "- Links to the `Staff` data table on 'usualgpstaffid'\n",
+ "- Links to the `Consultation`, `Observation` and `Drug Issue` data tables on 'patid'\n",
+ "- Links to the `Gender` lookup table on 'gender'\n",
+ "- Links to the `PatientType` lookup table on 'patienttypeid'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Patient table \n",
+ "SELECT * FROM Patient \n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Gender lookup table\n",
+ "SELECT * FROM Gender;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- PatientType lookup table\n",
+ "SELECT * FROM PatientType\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Practice` table\n",
+ "The `Practice` table \"contains details of each practice, including the practice identifier, practice region, and the last collection date.\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "\n",
+ "- Links to the `Patient` data table on 'pracid'\n",
+ "- Links to the `Region` lookup table on 'region'\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Practice table\n",
+ "SELECT * FROM Practice\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Region lookup table\n",
+ "SELECT * FROM Region;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Staff` table\n",
+ "The `Staff` table contains practice staff details for each staff member, including job category. *CPRD Aurum Data Specification Version 3.4*\n",
+ "- Links to the `Patient` data table on 'staffid'\n",
+ "- Links to the `Practice` data table on the 'pracid' \n",
+ "- Links to the `JobCat` lookup table on 'jobcatid'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Staff table \n",
+ "SELECT * FROM Staff \n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- JobCat lookup table\n",
+ "SELECT * FROM JobCat\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Consultation` table\n",
+ "The `Consultation` table \"contains information relating to the type of consultation as entered by the GP (e.g. telephone, home visit, practice visit). Some consultations are linked to observations that occur during the consultation via the consultation identifier (consid).\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "- Links to the `Patient` data table on 'patid'\n",
+ "- Links to the `Practice` data table on 'pracid'\n",
+ "- Links to the `Staff` data table on 'staffid' \n",
+ "- Links to the `Observation` data table on 'consid'\n",
+ "- Links to the `MedicaDictionary` table on 'consmedcodeid'\n",
+ "- Links to the `ConsSource` look up table on 'conssourceid'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Consultation table\n",
+ "SELECT * FROM Consultation \n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- ConsSource lookup table\n",
+ "SELECT * FROM ConsSource\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Observation` table\n",
+ "The `Observation` table \"contains the medical history data entered on the GP system including symptoms, clinical measurements, laboratory test results, and diagnoses, as well as demographic information recorded as a clinical code (e.g. patient ethnicity). Observations that occur during a consultation can be linked via the consultation identifier. CPRD Aurum data are structured in a long format (multiple rows per subject), and observations can be linked to a parent observation. For example, measurements of systolic and diastolic blood pressure will be grouped together via a parent observation for blood pressure measurement.\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "- Links to the `Patient` data table on 'patid'\n",
+ "- Links to the `Practice` data table on 'pracid'\n",
+ "- Links to the `Staff` data table on 'staffid'\n",
+ "- Links to the `Consultation` data table on 'consid'\n",
+ "- Links to the `Problem` and `Referral` data tables on 'obsid'\n",
+ "- Links to the `MedicaDictionary` table on 'medcodeid'\n",
+ "- Links to the `NumUnit` lookup table on 'numunitid'\n",
+ "- Links to the `ObsType` lookup table on 'obstypeid'\n",
+ "- Links to itself on 'parentobsid' and 'probobsid'\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Observation table \n",
+ "SELECT * FROM Observation\n",
+ "WHERE value IS NOT NULL AND numunitid IS NOT NULL AND numrangelow IS NOT NULL AND numrangehigh IS NOT NULL AND probobsid != 'None'\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- NumUnit lookup table\n",
+ "SELECT * FROM NumUnit\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- ObsType lookup table\n",
+ "SELECT * FROM ObsType;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Referral` table\n",
+ "The `Referral` table \"contains referral details recorded on the GP system. Data in the referral file are linked to the observation file and contain ‘add-on’ data for referral-type observations. These files contain information involving both inbound and outbound patient referrals to or from external care centres (normally to secondary care locations such as hospitals for inpatient or outpatient care). To obtain the full referral record (including reason for the referral and date), referrals should be linked to the Observation file using the observation identifier (obsid).\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "- Links to the `Patient` data table on 'patid'\n",
+ "- Links to the `Practice` data table on 'pracid'\n",
+ "- Links to the `Observation` data table on 'obsid'\n",
+ "- Links to `RefServiceType` lookup table on 'refservicetypeid'\n",
+ "- Links to `RefUrgency` lookup table on 'refurgencyid'\n",
+ "- Links to `OrgType` lookup table on 'refsourceorgid'\n",
+ "- Links to `RefMode` lookup table on 'refmodeid'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Referral table\n",
+ "SELECT * FROM Referral\n",
+ "WHERE reftargetorgid IS NOT NULL AND refurgencyid IS NOT NULL AND refservicetypeid IS NOT NULL AND refmodeid IS NOT NULL \n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- RefServiceType lookup table\n",
+ "SELECT * FROM RefServiceType;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- RefUrgency lookup table\n",
+ "SELECT * FROM RefUrgency;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- OrgType lookup table\n",
+ "SELECT * FROM OrgType LIMIT 3;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- RefMode lookup table\n",
+ "SELECT * FROM RefMode;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `Problem` table\n",
+ "The `Problem` table \"contains details of the patient’s medical history that have been defined by the GP as a ‘problem’. Data in the problem file are linked to the observation file and contain ‘add-on’ data for problem-type observations. Information on identifying associated problems, the significance of the problem and its expected duration can be found in this table. GPs may use ‘problems’ to manage chronic conditions as it would allow them to group clinical events (including drug prescriptions, measurements, symptom recording) by problem rather than chronologically. To obtain the full problem record (including the clinical code for the problem), problems should be linked to the Observation file using the observation identifier (obsid).\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "- Links to the `Patient` data table on 'patid'\n",
+ "- Links to the `Practice` data table on 'pracid'\n",
+ "- Links to the `Staff` data table on 'lastrevstaffid'\n",
+ "- Links to the `Observation` data table on 'obsid' and 'parentprobobsid'\n",
+ "- Links to the `ParentProbRel` lookup table on 'parentprobrelid'\n",
+ "- Links to the `ProbStatus` lookup table on 'probstatusid'\n",
+ "- Links to the `Sign` lookup table on 'signid'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "--Problem table \n",
+ "SELECT * FROM Problem\n",
+ "WHERE lastrevdate IS NOT NULL AND lastrevstaffid IS NOT NULL AND parentprobrelid IS NOT NULL AND probstatusid IS NOT NULL AND signid IS NOT NULL\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- ParentProbRel lookup table\n",
+ "SELECT * FROM ParentProbRel;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- ProbStatus lookup table\n",
+ "SELECT * FROM ProbStatus;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Sign lookup table\n",
+ "SELECT * FROM Sign;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### `DrugIssue` table\n",
+ "The `DrugIssue` table \"contains details of all prescriptions on the GP system. This file contains data relating to all prescriptions (for drugs and appliances) issued by the GP. Some prescriptions are linked to problem-type observations via the Observation file, using the observation identifier (obsid).\" *CPRD Aurum Data Specification Version 3.4*\n",
+ "- Links to the `Patient` data table on 'patid'\n",
+ "- Links to the `Practice` data table on 'pracid'\n",
+ "- Links to the `Staff` data table on 'staffid'\n",
+ "- Links to `Observation` and `Problem` data tables on 'probobsid'\n",
+ "- Links to `ProductDictionary` data dictionary table on 'prodcodeid'\n",
+ "- Links to the `Common_dosages` lookup table on 'dosageid'\n",
+ "- Links to the `QuantUnit` lookup table on 'quantunitid'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- DrugIssue table \n",
+ "SELECT * FROM DrugIssue\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Common_dosages lookup table\n",
+ "SELECT * FROM Common_dosages\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- QuantUnit lookup table\n",
+ "SELECT * FROM QuantUnit\n",
+ "ORDER BY RANDOM() \n",
+ "LIMIT 5;"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "py38_default",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.16"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/code-for-aurum/Step2-Notebooks/Step2B-intro-cprd-aurum-synthetic-stats.ipynb b/code-for-aurum/Step2-Notebooks/Step2B-intro-cprd-aurum-synthetic-stats.ipynb
new file mode 100644
index 0000000..4a688f2
--- /dev/null
+++ b/code-for-aurum/Step2-Notebooks/Step2B-intro-cprd-aurum-synthetic-stats.ipynb
@@ -0,0 +1,416 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTEBOOK SET UP (1) - ask for credentials and db info from user\n",
+ "import getpass\n",
+ "my_username = input('Your username: ')\n",
+ "my_password = getpass.getpass(prompt='Your password: ', stream=None)\n",
+ "this_host = input('Host name: ')\n",
+ "this_db = input('Database name: ')\n",
+ "\n",
+ "# NOTEBOOK SET UP (2) - load Jupyter magic functions & connect to db (assumes db & tables already created)\n",
+ "%load_ext sql\n",
+ "%sql postgresql+psycopg2://{my_username}:{my_password}@{this_host}/{this_db}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Summary statistics for CPRD Aurum Sample (Synthetic) Dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The summary statistics created in this notebook follow the structure of those within the ['Release Notes: CPRD Aurum Sample Dataset October 2021'](https://www.cprd.com/sites/default/files/2022-02/CPRD%20Aurum%20Sample%20Dataset%20Release%20Notes.pdf) PDF. \n",
+ "\n",
+ "This notebook aims to replicate the numbers that CPRD provides using SQL commands, as an introduction to interacting with this dataset and the tables with SQL.\n",
+ "\n",
+ "This notebook assumes you have created a SQL database with the CPRD tables within. See code Step1A, Step1B and Step1C in `code-for-aurum` to see how the raw text files were transformed into tables within a SQL database.\n",
+ "\n",
+ "*We have not yet matched all the answers in the data specification - please let us know if you spot why!*\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Total number of acceptable patients (including transferred out and deceased patients)\n",
+ "Permanent registrations only. The ‘acceptable’ flag refers to a research quality threshold based on CPRD metrics."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Count total acceptable patients\n",
+ "SELECT COUNT(*)\n",
+ "FROM patient\n",
+ "WHERE acceptable = 1;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Current number of acceptable patients (i.e. registered at currently contributing practices, excluding transferred out deceased patients)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "SELECT COUNT(*)\n",
+ "FROM patient\n",
+ "WHERE acceptable = 1 \n",
+ "AND cprd_ddate IS NULL -- The data spec suggests to use cprd_ddate instead of emis_ddate \n",
+ "AND regenddate IS NULL; -- regenddate is null means no date of registration ending "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Percentage of UK population coverage (current patients only)\n",
+ "Based on latest UK population estimates from the Office of National Statistics.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "SELECT COUNT(*)/667968.00 as percent_coverage\n",
+ "FROM patient\n",
+ "WHERE acceptable = 1\n",
+ "AND cprd_ddate IS NULL\n",
+ "AND regenddate IS NULL;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Available follow-up time in years since 1st January 1995 (all patients including transferred out and deceased):\n",
+ "Follow-up time stated here does not incorporate the up-to-standard (UTS) date and the database includes records pre-dating the 1st of January 1995\n",
+ "\n",
+ "*In this section, we don't quite match the answers in the release note!*\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Defining follow up time as difference between enddate and startdate\n",
+ "SELECT regenddate,\n",
+ "regstartdate,\n",
+ "regenddate-regstartdate AS followup_days,(regenddate-regstartdate)/365.0 AS followup_years\n",
+ "FROM Patient\n",
+ "WHERE regenddate IS NOT NULL\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- AVERAGE for all patients\n",
+ "SELECT AVG(\n",
+ " (\n",
+ " CASE WHEN regenddate IS NULL \n",
+ " THEN '2021-10-01' ELSE regenddate END -- if patient had no regenddate we assume the enddate is the date of cprd publication\n",
+ " - \n",
+ " CASE WHEN regstartdate < '1995-01-01' -- we want to include all patients, but if regstartdate is before 1995-01-01, we only count from this date\n",
+ " THEN '1995-01-01' ELSE regstartdate END\n",
+ " )/365.0\n",
+ " )\n",
+ "FROM Patient;\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- MEDIAN for all patients\n",
+ "WITH cte AS (\n",
+ " SELECT \n",
+ " (CASE WHEN regenddate IS NULL \n",
+ " THEN '2021-10-01' ELSE regenddate END \n",
+ " - \n",
+ " CASE WHEN regstartdate < '1995-01-01'\n",
+ " THEN '1995-01-01' ELSE regstartdate END\n",
+ " )/365.0 AS followup_years\n",
+ " FROM Patient\n",
+ " )\n",
+ " --select * from cte \n",
+ " SELECT percentile_disc(0.25) WITHIN group (ORDER BY followup_years) FROM cte\n",
+ " UNION ALL\n",
+ " SELECT percentile_disc(0.5) WITHIN group (ORDER BY followup_years) FROM cte\n",
+ " UNION ALL \n",
+ " SELECT percentile_disc(0.75) WITHIN group (ORDER BY followup_years) FROM cte"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "---STDEV for all patients\n",
+ "SELECT STDDEV(\n",
+ " (\n",
+ " CASE WHEN regenddate IS NULL \n",
+ " THEN '2021-10-01' ELSE regenddate END \n",
+ " - \n",
+ " CASE WHEN regstartdate < '1995-01-01' \n",
+ " THEN '1995-01-01' ELSE regstartdate END\n",
+ " )/365.0\n",
+ " )\n",
+ "FROM Patient;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- AVERAGE for current patients\n",
+ "SELECT AVG(\n",
+ " (\n",
+ " CASE WHEN regenddate IS NULL \n",
+ " THEN '2021-10-01' ELSE regenddate END \n",
+ " - \n",
+ " CASE WHEN regstartdate < '1995-01-01'\n",
+ " THEN '1995-01-01' ELSE regstartdate END\n",
+ " )/365.0\n",
+ " )\n",
+ "FROM Patient\n",
+ "WHERE regenddate IS NULL\n",
+ "AND cprd_ddate IS NULL;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- MEDIAN for all patients\n",
+ "WITH cte AS (\n",
+ " SELECT \n",
+ " (CASE WHEN regenddate IS NULL \n",
+ " THEN '2021-10-01' ELSE regenddate END \n",
+ " - \n",
+ " CASE WHEN regstartdate < '1995-01-01'\n",
+ " THEN '1995-01-01' ELSE regstartdate END\n",
+ " )/365.0 AS followup_years\n",
+ " FROM Patient\n",
+ " WHERE regenddate IS NULL\n",
+ " AND cprd_ddate IS NULL\n",
+ " )\n",
+ " --select * from cte \n",
+ " SELECT percentile_disc(0.25) WITHIN group (ORDER BY followup_years) FROM cte\n",
+ " UNION ALL\n",
+ " SELECT percentile_disc(0.5) WITHIN group (ORDER BY followup_years) FROM cte\n",
+ " UNION ALL \n",
+ " SELECT percentile_disc(0.75) WITHIN group (ORDER BY followup_years) FROM cte\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- STDEV for current patients \n",
+ "SELECT STDDEV(\n",
+ " (\n",
+ " CASE WHEN regenddate IS NULL \n",
+ " THEN '2021-10-01' ELSE regenddate END \n",
+ " - \n",
+ " CASE WHEN regstartdate < '1995-01-01'\n",
+ " THEN '1995-01-01' ELSE regstartdate END\n",
+ " )/365.0\n",
+ " )\n",
+ "FROM Patient\n",
+ "WHERE regenddate IS NULL\n",
+ "AND cprd_ddate IS NULL;\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Total number of practices (current and historic) included in the database"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "SELECT COUNT(*) FROM practice;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Total number of distinct practices \n",
+ "SELECT COUNT(DISTINCT practice) FROM practice;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Currently contributing practices\n",
+ "\n",
+ "> Code to come!\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Percentage coverage of UK general practices (currently contributing practices only)\n",
+ "Expressed as a percentage of all practices currently contributing to CPRD Aurum\n",
+ "\n",
+ "> Code to come! "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### Regional distribution of currently contributing practices\n",
+ "\n",
+ "*In this section, we don't quite match the answers in the release note!*"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Regional distribution of currently contributing practices\n",
+ "SELECT re.description AS Region, COUNT(pr.pracid) AS TotalPractices \n",
+ "FROM practice PR\n",
+ "INNER JOIN region re \n",
+ "ON re.regionid = pr.region\n",
+ "GROUP BY pr.region, re.description"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "py38_default",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.16"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/code-for-aurum/Step2-Notebooks/Step2C-cprd-cohort-notebooks.ipynb b/code-for-aurum/Step2-Notebooks/Step2C-cprd-cohort-notebooks.ipynb
new file mode 100644
index 0000000..4671b4b
--- /dev/null
+++ b/code-for-aurum/Step2-Notebooks/Step2C-cprd-cohort-notebooks.ipynb
@@ -0,0 +1,896 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## CPRD Cohort Criteria Examples Workbook"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This workbook was created to replicate the example criteria given in CPRD Aurum FAQs v2.4 (see their [website](https://www.cprd.com/primary-care-data-public-health-research)) under the section *\"How will I know if the CPRD Aurum data are suitable for my research needs?\".* When we include text within quotation marks in the sections below, we are quoting from CPRD Aurum FAQs v2.4. \n",
+ "\n",
+ "We are using these examples to increase our understanding of the tables, how they all relate to eachother and how to write queries for example criteria. These types of queries would allow a research team to filter the CPRD data, to create a sample cohort that matches their research questions e.g. select patients within a certain age range and on a specific medication. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "----\n",
+ "Preliminary setup code:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTEBOOK SET UP (1) - ask for credentials and db info from user\n",
+ "import getpass\n",
+ "my_username = input('Your username: ')\n",
+ "my_password = getpass.getpass(prompt='Your password: ', stream=None)\n",
+ "this_host = input('Host name: ')\n",
+ "this_db = input('Database name: ')\n",
+ "\n",
+ "# NOTEBOOK SET UP (2) - load Jupyter magic functions & connect to db (assumes db & tables already created)\n",
+ "%load_ext sql\n",
+ "%sql postgresql+psycopg2://{my_username}:{my_password}@{this_host}/{this_db}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Below we've written queries for each example criterion step-by-step, hopefully making the complete query easier to understand."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Part 1. Counts based on one criterion:\n",
+ "\n",
+ "*\"The total number of patients in CPRD GOLD or CPRD Aurum with a first ever prescription for\n",
+ "metformin recorded during 01/01/2004 - 31/12/2015, stratified by calendar year\"*"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "a. Find all patients on metformin prescription"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- FILTERING all terms relating to metformin from product dictionary into a new table called A_MetforminProdID\n",
+ "DROP TABLE IF EXISTS A_MetforminProdID;\n",
+ "SELECT * INTO A_MetforminProdID FROM ProductDictionary \n",
+ "WHERE (\n",
+ " UPPER(DrugSubstanceName) LIKE '%METFORMIN%' -- input terms to include here\n",
+ " );\n",
+ " \n",
+ "-- Drug issue table contains data relating to all drug prescriptions issued by the GP, \n",
+ "-- so joining this table to the filtered product dictionary, we'll get all metformin related prescriptions\n",
+ "SELECT DI.*, '<----->' AS BOUNDARY, MP.* FROM DrugIssue DI\n",
+ "INNER JOIN A_MetforminProdID MP ON MP.prodcodeID = DI.prodcodeID\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "source": [
+ "b. Now we have a way of finding all patients with metformin prescriptions issued, we want to find the very first metformin prescription issued, grouped by patient: \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Grouping metformin prescription for each patient\n",
+ "SELECT patid, issuedate FROM DrugIssue DI\n",
+ "INNER JOIN A_MetforminProdID MP ON MP.prodcodeID = DI.prodcodeID\n",
+ "GROUP BY patid, issuedate\n",
+ "ORDER BY issuedate ASC\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Insert above grouping into a new table called A_MetforminPatients:\n",
+ "DROP TABLE IF EXISTS A_MetforminPatients;\n",
+ "SELECT patid, issuedate INTO A_MetforminPatients \n",
+ "FROM DrugIssue DI\n",
+ "INNER JOIN A_MetforminProdID MP ON MP.prodcodeID = DI.prodcodeID\n",
+ "--WHERE patid = 'XXXXXXXX' -- to look at single patient sample\n",
+ "GROUP BY patid, issuedate\n",
+ "ORDER BY issuedate ASC\n",
+ "\n",
+ "-- Ordering to find first metformin prescription for each patient\n",
+ ";WITH added_row_number AS (\n",
+ " SELECT\n",
+ " *,\n",
+ " ROW_NUMBER() OVER(PARTITION BY patid ORDER BY issuedate ASC) AS row_number\n",
+ " FROM A_MetforminPatients\n",
+ ")\n",
+ "SELECT\n",
+ " *\n",
+ "FROM added_row_number\n",
+ "--WHERE row_number = 1 --AND patid = 'XXXXXXXX' -- single patient sample\n",
+ "LIMIT 20;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "source": [
+ "c. We have the first metformin prescriptions issued for each patient, so now we want to filter out the those within the date interval given: 01/01/2004 - 31/12/2015 \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "WITH added_row_number AS (\n",
+ " SELECT\n",
+ " *,\n",
+ " ROW_NUMBER() OVER(PARTITION BY patid ORDER BY issuedate ASC) AS row_number\n",
+ " FROM A_MetforminPatients\n",
+ ")\n",
+ "SELECT\n",
+ " *\n",
+ "FROM added_row_number\n",
+ "WHERE row_number = 1 --AND patid = 'XXXXXXXXXX'\n",
+ "AND issuedate BETWEEN '2004-01-01' AND '2015-12-31' \n",
+ "LIMIT 3;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "d. The last condition is to stratify by calendar year, which we'll do below:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- 4. Group by calendar year\n",
+ "WITH added_row_number AS (\n",
+ " SELECT\n",
+ " *,\n",
+ " ROW_NUMBER() OVER(PARTITION BY patid ORDER BY issuedate ASC) AS row_number\n",
+ " FROM A_MetforminPatients\n",
+ ")\n",
+ "SELECT\n",
+ " DATE_PART('YEAR',issuedate), count(patid)--, issuedate\n",
+ "FROM added_row_number\n",
+ "WHERE row_number = 1 --AND patid = 'XXXXXXXXXX'\n",
+ "AND issuedate BETWEEN '2004-01-01' AND '2015-12-31' \n",
+ "GROUP BY DATE_PART('YEAR',issuedate)--,patid, issuedate\n",
+ "LIMIT 8;\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The above makes use of the product dictionary, and a sample string ('metformin') to filter out relevant drugs. You may however already have a list of drugs that you'd like to load in directly and use to filter out patients, for which we'll have an example below."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "----"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Part 2. Counts based on two criteria:\n",
+ "\n",
+ "\n",
+ "\n",
+ "1) \"Total number of patients with a medical diagnosis of Type 2 diabetes mellitus recorded in CPRD GOLD or HES APC \n",
+ "on or before 31/12/2005\" \n",
+ "OR\n",
+ "2) \"Prescriptions for anti-diabetic medication in CPRD GOLD (note - provided in one code list) \n",
+ "on or before 31/12/2005. Patients must have at least 12 months of prior registration before their earliest event date.\"\n",
+ "\n",
+ "Rephrased criteria: \n",
+ "\n",
+ "1) Total number of patients with a medical diagnosis of Type 2 diabetes mellitus recorded in CPRD Aurum on or before 31/12/2005 \n",
+ "OR\n",
+ "2) Prescriptions for anti-diabetic medication in CPRD Aurum (note - provided in one code list) on or before 31/12/2005. \n",
+ "Patients must have at least 12 months of prior registration before their earliest event date.\n",
+ "\n",
+ "We'll split up both criteria (1 & 2) and start with the first one, in segments, below:\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "1a. Load in pre-existing code list for Type2Diabetes, CPRD Aurum, from OPTIMAL's THINK repo (or any other of your own choice)\n",
+ "\n",
+ "link here: https://github.com/THINKINGGroup/phenotypes/tree/main/Medical%20conditions/Type2Diabetes_11_3_21_birm_cam\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Specify file path for your downloaded code list file (for example: User/Docs/codelists/type2diabetes.csv)\n",
+ "code_list_dir = input('Enter path for type 2 diabetes code list file')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Create a new table to load list into (make sure this matches the headers in your code list file!)\n",
+ "DROP TABLE IF EXISTS A_Type2Diabetes_CodeLists;\n",
+ "CREATE TABLE A_Type2Diabetes_CodeLists\n",
+ "(MEDICAL_CODE_ID TEXT,DESCRIPTION TEXT,READ_CODE TEXT,SNOMED_CT_CODE TEXT,DATASOURCE TEXT);"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Loading in code list from directory specified above\n",
+ "DO $$ \n",
+ "DECLARE\n",
+ " filepath TEXT := :code_list_dir;\n",
+ "BEGIN\n",
+ " EXECUTE 'COPY A_Type2Diabetes_CodeLists (MEDICAL_CODE_ID,DESCRIPTION,READ_CODE,SNOMED_CT_CODE,DATASOURCE) FROM ''' || filepath || ''' WITH (FORMAT ''csv'', DELIMITER '','', HEADER, QUOTE ''\"'');';\n",
+ "END $$;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- View table contents \n",
+ "SELECT * FROM A_Type2Diabetes_CodeLists LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Use code list to filter medcodeids in medical dictionary for Type2Diabetes\n",
+ "SELECT * FROM medicaldictionary\n",
+ "WHERE medcodeid IN (SELECT medical_code_id FROM A_Type2Diabetes_CodeLists)\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "1b. Find all type 2 diabetes related observations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- View patients with type2db medcodeid in observation table\n",
+ "SELECT * FROM observation \n",
+ "WHERE medcodeid IN (SELECT medical_code_id FROM A_Type2Diabetes_CodeLists)\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "--Total observations related to type2db\n",
+ "(SELECT 'Total obs related to typ2db' AS Totals, count(*) FROM observation\n",
+ "WHERE medcodeid IN (SELECT medical_code_id FROM A_Type2Diabetes_CodeLists))\n",
+ "UNION\n",
+ "-- Total distinct patients with observations related to/medical diagnosis of Type 2 diabetes mellitus \n",
+ "(SELECT 'Total distinct patients with obs related to typ2db', count(DISTINCT patid) FROM observation\n",
+ "WHERE medcodeid IN (SELECT medical_code_id FROM A_Type2Diabetes_CodeLists));"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "1c. Medical diagnosis (inferred by a medcodeid occurence) recorded on or before 31/12/2005:\n",
+ "\n",
+ " Note to check assumptions of methodology below: we might want to join the problem table and consider the parent problem label - we did it manually for now. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- Find all type2d observations grouped (least to most recent, ascending) for each patient\n",
+ "SELECT patid,obsdate,medcodeid,t.description FROM observation o \n",
+ "INNER JOIN A_Type2Diabetes_CodeLists t ON t.medical_code_id = o.medcodeid \n",
+ "WHERE medcodeid IN (SELECT medical_code_id FROM A_Type2Diabetes_CodeLists)\n",
+ "GROUP BY patid,obsdate,medcodeid, t.description\n",
+ "ORDER BY patid,obsdate ASC\n",
+ "LIMIT 5;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- Find earliest observation and add constraint for observation date\n",
+ "WITH patid_ranked AS (\n",
+ "SELECT *, \n",
+ "ROW_NUMBER() OVER(PARTITION BY patid ORDER BY obsdate ASC) AS row_number\n",
+ "FROM observation WHERE medcodeid IN (SELECT medical_code_id FROM A_Type2Diabetes_CodeLists)\n",
+ ")\n",
+ "SELECT * FROM patid_ranked\n",
+ "WHERE row_number = 1 -- most recent observation when ordered by obsdate ascending\n",
+ "AND obsdate <= '2005-12-31' -- add date constraint for the first observation \n",
+ "LIMIT 3; \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we'll look at the second criteria:\n",
+ "\n",
+ "2. Prescriptions for anti-diabetic medication in CPRD Aurum (note - provided in one code list) on or before 31/12/2005. \n",
+ "Patients must have at least 12 months of prior registration before their earliest event date.\n",
+ "\n",
+ "Edit: date modified from 31/12/2005 to 31/12/2015 as this was a more suitable range for the synthetic Aurum dataset. If you are running this notebook on the real Aurum dataset, try the original date threshold. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "2a. Load in multiple code lists from OPTIMAL's THINK repo (or any resource of your own preference) that relate to anti-diabetic medications.\n",
+ "\n",
+ "links here: \n",
+ "- https://github.com/THINKINGGroup/phenotypes/blob/main/Drug%20Codes\n",
+ "- https://phenotypes.healthdatagateway.org/phenotypes/\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- Create a new filename table, as there are multiple antidiabetic drug files\n",
+ "DROP TABLE IF EXISTS A_AntiDiabeticDrug_CodeList_Name;\n",
+ "CREATE TABLE A_AntiDiabeticDrug_CodeList_Name(FILEID INT GENERATED BY DEFAULT AS IDENTITY, DRUGFILENAME TEXT);\n",
+ "SELECT * FROM A_AntiDiabeticDrug_CodeList_Name;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "## Change directory here to the folder containing the code lists:\n",
+ "directory_path = input('Specify directory for folder containing the diabetic medication code lists')\n",
+ "\n",
+ "# Iterate over all the code lists in the specified directory\n",
+ "for filename in os.listdir(directory_path):\n",
+ " file_path = os.path.join(directory_path, filename)\n",
+ " if os.path.isfile(file_path):\n",
+ " # Insert filenames into the table\n",
+ " %sql INSERT INTO A_AntiDiabeticDrug_CodeList_Name (DRUGFILENAME) VALUES (:filename)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Create a new antidiabetic medication code list table (note this should match your code list file headers):\n",
+ "DROP TABLE IF EXISTS A_AntiDiabeticDrug_CodeLists;\n",
+ "CREATE TABLE A_AntiDiabeticDrug_CodeLists\n",
+ "(DRUG_CODE_ID TEXT,DESCRIPTION TEXT,BNF1 TEXT,BNF2 TEXT,BNF3 TEXT,ATC TEXT,DATABASE TEXT);"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Create function to load in data below, which we will then use to loop over multiple code lists\n",
+ "\n",
+ "*You will need to change the values according to your own code list file contents \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "-- Function to load code lists into table in a LOOP\n",
+ "CREATE OR REPLACE FUNCTION A_loaddata(filepathname TEXT)\n",
+ " RETURNS void\n",
+ " LANGUAGE plpgsql AS\n",
+ "$func$\n",
+ "BEGIN\n",
+ " EXECUTE format ('\n",
+ " COPY A_AntiDiabeticDrug_CodeLists(DRUG_CODE_ID,DESCRIPTION,BNF1,BNF2,BNF3,ATC,DATABASE) -- more columns\n",
+ " FROM %L (FORMAT CSV, HEADER)' -- modern syntax\n",
+ " -- WITH CSV HEADER' -- tolerated legacy syntax\n",
+ " , $1); -- pass 1st function parameter (filepathname) to format() \n",
+ "END\n",
+ "$func$;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, loop over drug list filenames and apply loaddata() function:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- SELECT * FROM A_AntiDiabeticDrug_CodeList_Name; -- contains filenames\n",
+ "-- SELECT * FROM A_AntiDiabeticDrug_CodeLists; -- table to insert into\n",
+ "\n",
+ "--Loop loaddata() function over drug list filenames to insert into table:\n",
+ "do $$\n",
+ "DECLARE fileName VARCHAR;\n",
+ "BEGIN\n",
+ "FOR fileName IN \n",
+ "SELECT drugfilename FROM A_AntiDiabeticDrug_CodeList_Name\n",
+ "LOOP\n",
+ " PERFORM A_loaddata(\n",
+ " :directory_path || '/' || fileName\n",
+ " );\n",
+ "END LOOP;\n",
+ "END;\n",
+ "$$;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "--Check table for loaded data:\n",
+ "SELECT * FROM A_AntiDiabeticDrug_CodeLists\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "2b. Now we want to use the antidiabetic code lists table to filter out the relevant drugs from dictionary (or we can use code list table directly, since it already has cprd aurum identifiers if it's a complete match)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- a. Use code list to filter prodcodeids in product dictionary for anti-diabetic medication\n",
+ "SELECT * FROM productdictionary \n",
+ "WHERE PRODCODEID IN (SELECT drug_code_id FROM A_AntiDiabeticDrug_CodeLists)\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- You can also check if there's a count match between all the codes in your code list vs. codes in the product dictionary table as a sanity check\n",
+ "(SELECT COUNT(DISTINCT drug_code_id) FROM A_AntiDiabeticDrug_CodeLists) --\n",
+ "UNION ALL\n",
+ "(SELECT count(*) FROM productdictionary \n",
+ "WHERE PRODCODEID IN (SELECT drug_code_id FROM A_AntiDiabeticDrug_CodeLists)\n",
+ ");"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "2c. Find patient observations relating to antidiabetic drugs"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "SELECT DI.*, ADC.* FROM drugissue DI \n",
+ "INNER JOIN A_AntiDiabeticDrug_CodeLists ADC ON DI.prodcodeid = ADC.drug_code_id\n",
+ "WHERE DI.issuedate <= '2015-12-31'\n",
+ "ORDER BY enterdate ASC\n",
+ "LIMIT 2;\n",
+ "\n",
+ "-- Total patient prescriptions with antidiabetic drugs\n",
+ "SELECT COUNT(DISTINCT patid) FROM drugissue DI \n",
+ "INNER JOIN A_AntiDiabeticDrug_CodeLists ADC ON DI.prodcodeid = ADC.drug_code_id\n",
+ "WHERE DI.issuedate <= '2015-12-31'\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "2d. Patients must have at least 12 months of prior registration before their earliest event date (earliest prescription).\n",
+ "\n",
+ "So below we have a table with earliest prescription issue date for each patient on antidiabetic drugs, where patients also have AT LEAST a one year interval between first issue of metformin and registration date at GP clinic \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "-- Table with earliest prescription issue date \n",
+ "-- for each patient on antidiabetic drugs\n",
+ "WITH patid_ranked AS (\n",
+ " SELECT DI.*, \n",
+ " ROW_NUMBER() OVER(PARTITION BY patid ORDER BY issuedate ASC) AS row_number,\n",
+ " ADC.*\n",
+ " FROM drugissue DI \n",
+ " INNER JOIN A_AntiDiabeticDrug_CodeLists ADC ON DI.prodcodeid = ADC.drug_code_id\n",
+ " WHERE DI.issuedate <= '2015-12-31'\n",
+ " )\n",
+ "SELECT P.regstartdate,R.issuedate,R.issuedate - P.regstartdate AS DIFF, R.issuedate - interval '1 year' AS IssueDateMinus1Year,R.* FROM patid_ranked R\n",
+ " INNER JOIN Patient P ON P.patid = R.patid\n",
+ " WHERE row_number = 1\n",
+ " AND P.regstartdate <= (R.issuedate - interval '1 year')\n",
+ " ORDER BY DIFF ASC\n",
+ " LIMIT 3;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "------\n",
+ "\n",
+ "#### Part 3. Counts based on three criteria:\n",
+ "1) The total number of patients in CPRD Aurum with an incident medical\n",
+ "diagnosis of Type 2 diabetes mellitus recorded in during 01/01/2004 - 31/12/2015 OR\n",
+ "2) Incident prescription of anti-diabetic medication (note-provided in one code list) documented during\n",
+ "01/01/2004 - 31/12/2015 AND\n",
+ "3) Have a test record for HbA1c recorded in CPRD Aurum (note - test value not assessed)\n",
+ "\n",
+ "First and second criteria have already been previously covered; here we focus on the third criteria only.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "3a. Find all terms related to '%HbA1%' and insert into a new table called A_hba1_medcodes\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql \n",
+ "--SELECT * FROM Observation LIMIT 10;\n",
+ "--SELECT * FROM obstype ORDER BY 1; -- lookup table\n",
+ "DROP TABLE IF EXISTS A_hba1_medcodes;\n",
+ "SELECT * INTO A_hba1_medcodes FROM medicaldictionary\n",
+ "WHERE term LIKE '%HbA1%';\n",
+ "--View table values:\n",
+ "SELECT * FROM A_hba1_medcodes LIMIT 2;\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "3b. Finding all patients with HbA1C related observations, where the observation type is a 'value' and the value is populated (not a NULL value)\n",
+ "\n",
+ "*Note, assumption made below, that 'value' obstype gives the test record\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "\n",
+ "-- if we want to see all the distinct observation types \n",
+ "SELECT DISTINCT obs.obstypeid FROM Observation obs\n",
+ "INNER JOIN obstype ot ON ot.obstypeid = obs.obstypeid\n",
+ "WHERE obs.medcodeid IN (SELECT medcodeid FROM A_hba1_medcodes)\n",
+ "LIMIT 2;\n",
+ "\n",
+ "--checking what different ids correspond to\n",
+ "SELECT * FROM numunit;\n",
+ "\n",
+ "--Joining tables to see observations for patients where \n",
+ "--type of observation is a 'value' type, not null, \n",
+ "-- and related to 'hba1c' term in dictionary \n",
+ "SELECT obs.*, ot.*, md.* FROM Observation obs\n",
+ "INNER JOIN obstype ot ON ot.obstypeid = obs.obstypeid\n",
+ "INNER JOIN medicaldictionary md ON md.medcodeid = obs.medcodeid\n",
+ "WHERE obs.medcodeid IN (SELECT medcodeid FROM A_hba1_medcodes)\n",
+ "AND ot.obstypeid = 10 -- value\n",
+ "AND obs.value IS NOT NULL\n",
+ "LIMIT 2;"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "3c. We can also now count the distinct number of patients:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "--count distinct patients:\n",
+ "SELECT COUNT(DISTINCT obs.patid) FROM Observation obs\n",
+ "INNER JOIN obstype ot ON ot.obstypeid = obs.obstypeid\n",
+ "INNER JOIN medicaldictionary md ON md.medcodeid = obs.medcodeid\n",
+ "WHERE obs.medcodeid IN (SELECT medcodeid FROM A_hba1_medcodes)\n",
+ "AND ot.obstypeid = 10 -- value\n",
+ "AND obs.value IS NOT NULL\n",
+ "LIMIT 2;\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "source": [
+ "------\n",
+ "\n",
+ "#### Drop tables created in this notebook:\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "vscode": {
+ "languageId": "sql"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "%%sql\n",
+ "DROP TABLE IF EXISTS a_antidiabeticdrug_codelist_name;\n",
+ "DROP TABLE IF EXISTS a_antidiabeticdrug_codelists;\n",
+ "DROP TABLE IF EXISTS a_metforminpatients;\n",
+ "DROP TABLE IF EXISTS a_metforminprodid;\n",
+ "DROP TABLE IF EXISTS a_type2diabetes_codelists;\n",
+ "DROP TABLE IF EXISTS a_hba1_medcodes;\n",
+ "DROP FUNCTION IF EXISTS a_loaddata;"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "py38_default",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.16"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/code-for-aurum/Step2-Notebooks/cprd-aurum-data-structure.png b/code-for-aurum/Step2-Notebooks/cprd-aurum-data-structure.png
new file mode 100644
index 0000000..1d8f6ca
Binary files /dev/null and b/code-for-aurum/Step2-Notebooks/cprd-aurum-data-structure.png differ
diff --git a/code-for-aurum/installation-setup.md b/code-for-aurum/installation-setup.md
new file mode 100644
index 0000000..6d2db74
--- /dev/null
+++ b/code-for-aurum/installation-setup.md
@@ -0,0 +1,147 @@
+# Software installation
+
+Step 1 of the workflow uses **Python** and **PostgreSQL**. To create this workflow, we used a [Data Science Virtual Machine (DSVM)](https://azure.microsoft.com/en-gb/products/virtual-machines/data-science-virtual-machines)(Ubuntu 20.04) which already had Python and PostgreSQL installed.
+
+- **Python**: the installation of Python will depend on your operating system, reference [the main Python docs](https://www.python.org) for more details. The python files (`.py` below) were created with Python 3.8, using the Python extension within [Visual Studio Code](https://code.visualstudio.com).
+
+- **PostgreSQL**: On the DSVM, it was likely installed by `sudo apt install postgresql postgresql-contrib`. Reference [the main PostgreSQL docs](https://www.postgresql.org/download) for more details on how to install on your operating system.
+
+It is likely that your research institution will already have their existing infrastructure and compulsory/recommended software set-ups. If not, we include some extra information in the next section, to cover:
+- how we configured PostgreSQL for our set-up and created users with different permissions
+- how we executed sql commands via the psql terminal, a sql script, and a shell script
+- how we used Visual Studio Code to provide a graphical user interface to execute sql commands, as well as our favoured way to connect to the VM via the ssh connection
+- how we integrate PostgreSQL with Jupyter Notebook
+
+# PostgreSQL on Ubuntu DSVM
+
+## Configure & create users
+Resources: [1-Azure Quick Start](https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/postgresql-standalone-server-ubuntu/) and [2-PostgreSQL on DSVM](https://learn.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/linux-dsvm-walkthrough?view=azureml-api-2#postgresql-and-squirrel-sql)
+
+1. Check configuration files for postgresql (resource 2)
+> They talk about `/var/lib/pgsql/data/pg_hba.conf` but ours was here: `/etc/postgresql/12/main/pg_hba.conf`
+> They say to *'Change the IPv4 local connections line to use md5 instead of ident, so we can log in by using a username and password.'* This was already done for our files, so did not need to change anything.
+2. Launch psql, the interactive terminal for PostgreSQL, as the built-in postgres user:
+
+`sudo su - postgres psql`
+
+3. As postgres user, check you can create a database, create table, and insert into this table (resource 1)
+
+5. Create a user account (resource 2), and assign the appropriate [roles](https://www.postgresql.org/docs/current/predefined-roles.html) and [privileges](https://www.postgresql.org/docs/current/ddl-priv.html).
+
+7. Quit the interactive terminal with `\q`
+> It said could not save history to file `"/var/lib/postgresql/.psql_history": No such file or directory`. It's trying to save the history of what we've done so far in psql as user the 'postgres'. Fortunately this does not seem like it really needs immediate solving because when we go on to use psql with our user accounts, it saves the history here `/home/your-username/.psql_history`.
+
+6. Write `su your_username` to get back to user profile
+
+7. Log back into the the interactive psql terminal by writing `psql` in the terminal and checking that your desired default username is being used
+
+## Exploring PostgreSQL file layout on the DSVM
+
+Using [this documentation](https://www.postgresql.org/docs/current/storage-file-layout.html) and cross referencing with our set-up on the DSVM:
+
+Cluster configuration files are here:
+> etc/postgresql/12/main/
+
+The docs talk of a PGDATA directory where data files used by the database cluster are stored. We don't have anything called that but it seems to be here:
+> ls /var/lib/postgresql/12/main/
+
+We also have postgresql files in these locations:
+> /usr/share/postgresql/12/
+> /usr/share/postgresql-common/
+
+And, as expected, there are executables in `/bin` which start with `pg_`
+
+## Test psql commands
+
+:bulb: A cheat sheet of helpful commands: https://tomcam.github.io/postgres/#using-psql
+
+### Interactive psql terminal
+
+When logged in to psql terminal with your username, try these commands:
+
+ - `CREATE DATABASE your_database;`
+ - `\c your_database`
+ - `CREATE TABLE products (product integer, name text, price numeric);`
+ - `INSERT INTO products VALUES (1, 'Cheese',4.99);`
+ - `SELECT * FROM products;`
+ - Make a csv file with multiple rows, with these columns `n, name, price`
+ - `COPY products (n, name, price) FROM 'test_files/products.csv' WITH (FORMAT 'csv', HEADER, DELIMITER ',');`
+ - `SELECT * FROM products;`
+
+### Execute from sql script
+- Create a new file on the shell terminal `touch test.sql`
+- Paste the above commands that you ran on the psql terminal into the test.sql file
+- Log into psql `psql`
+- Run the sql script, with outputs displayed: `\i test.sql`
+- Run the sql script, with outputs pasted into a text file: `\o output.txt` `\i test.sql`
+
+### Execute from shell script
+- Create a new file on the shell terminal `touch test_sql.sh`
+- Inside this test_sql.sh file write:
+
+> #!/bin/sh
+>
+> dbname="database-name-that-exists"
+>
+> username="your-username"
+>
+> psql $dbname $username << EOF
+>
+> SELECT * FROM table-that-exists;
+>
+> EOF
+
+All the postgreSQL commands are provided inside the EOF block. You can also use shell variables inside the sql command e.g.
+
+`wherecond="something"`
+
+`SELECT * FROM table-that-exists WHERE col_name = '$wherecond';`
+
+## ssh to the DSVM with VSCode (ignore if no VM being used)
+
+On our local MacOS we installed *Remote Explorer* and *Remote - ssh* by Microsoft. Instead of using an ssh connection via the terminal app, we can ssh connect to the VM via our local [Visual Studio (VS) Code](https://code.visualstudio.com/) app, with these extensions.
+
+**After installing extensions:**
+- Note a small green box in bottom left of VSCode, with the '**><**' icon
+- Click on **><** to open a remote window
+- In the dialog box, select Connect to Host (Remote SSH) > Add new SSH host
+- Enter ssh connection details - it needed username, server IP and private key. We located and selected the SSH configuration file previously created. This added the host to our configuration file.
+- Select **><** , Connect to Host, and select the server IP address from the drop down. This prompt us to enter a password
+- Once done, a new remote window will open. You can confirm this is connected to the host as the >< icon will change in the bottom left
+
+## Visual Studio Code to interact with PostgreSQL
+
+As an alternative to using PostgreSQL on terminal only, we can use an extension to VS Code to give us a graphical user interface (GUI). There are other GUIs available!
+
+We installed the *PostgreSQL Management Tool* extension by *Chris Kolkman* in two locations: on VSCode on the VM (access via remote desktop) and on VSCode on our local MacOS.
+
+When in the postgreSQL explorer on VS Code, press the + button to add a database connection.
+- Hostname was 'localhost'
+- Put in postgreSQL credentials (created earlier)
+- Kept the default port
+- Chose a secure connection
+
+## PostgreSQL Integration with Jupyter Notebook
+
+These python libraries must be installed: ipython-sql, sqlalchemy, Psycopg2
+
+Then run this in a Python cell:
+
+``
+%load_ext sql
+from sqlalchemy import create_engine
+``
+
+Connecting to a PostgreSQL database run:
+
+``
+%sql dialect+driver://username:password@host:port/database
+``
+
+For example:
+
+``
+%sql postgresql://postgres:password123@localhost/mydatabase
+``
+
+
diff --git a/code-for-aurum/workflow_idea.png b/code-for-aurum/workflow_idea.png
new file mode 100644
index 0000000..70180d4
Binary files /dev/null and b/code-for-aurum/workflow_idea.png differ
diff --git a/cprd-code-browser.md b/cprd-code-browser.md
new file mode 100644
index 0000000..2c95a71
--- /dev/null
+++ b/cprd-code-browser.md
@@ -0,0 +1,13 @@
+## CPRD's Code Browser
+
+👉 Refer to [the CPRD website](https://cprd.com/defining-your-study-population) for information on CPRD's Code Browser tool, particularly the *'How to find the codes used in primary care data'* section, which links to a user guide.
+
+👉 A researcher can request access to the Code Browser tool for free by emailing *enquiries@cprd.com*.
+
+The Code Browser software is designed for Windows OS only.
+
+## Why use the Code Browser?
+
+- It contains additional information to the data specification PDFs as it gives you access to the medical and product dictionaries, allowing you to browse and create code lists that can be used to build your cohort.
+- It contains some observation counts (for real data, not synthetic). These counts are rounded to one significant figure for disclosure control. However, the observation counts only show the number of times each code has been used in the Observation file, and not the number of patients who have that code in their record (one patient may have same code multiple times during different consultations).
+- For a patient count for a specific condition, CPRD first recommends looking at publications in your area of interest in their bibliography, otherwise you can request a feasible count from CPRD based on inclusions/exclusion criteria.