99_writing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Genomic BigData Warehousing with Apache Spark and LakeHouse Architecture\n",
    "\n",
    "**DRAFT** | **DRAFT** | **DRAFT** | **DRAFT**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Abstract\n",
    "\n",
    "In this repo/article, we explore and discuss Cloud-native BigData engineering approach for handling Genomics data warehousing. The technique underpin BigData processing and data warehousing with Apache Spark and Data Lakehouse[0] architecture on Genomics data of \"gold\" label VCF files. We set up Genomic Table along with Metadata Table in Cloud-native fashion to bring both Genotype and Phenotype together within single or federated queryable interface. By leveraging Cloud-native setup, we bring Data & Compute closer together to avoid unnecessary data staging for further Genomic analysis or Cohort data building process.\n",
    "\n",
    "Genomic Table and, as such sourcing data warehouse directly from VCF could easily get out of hand. Typically, a \"best practice\" BioInformatics Pipeline should have finer, distilled, end-of-chain, \"gold\" label VCF product. This VCF file or metrics/fact files should contain a handful of records per patient that have already analysed & annotated well for a given use case. If sourcing Genomic Table from VCF is not appropriate then one could leverage more summarised tables output from MultiQC or some Cancer Reporter as see fit.\n",
    "\n",
    "Hence, this kind of Genomic BigData warehouse would still rely on high quality output of BioInformatics Pipeline and, continuation of overall Data Pipeline.\n",
    "\n",
    "### Problems\n",
    "\n",
    "Today, Genomics analysis pipeline generates unprecedented amount of data and, therefore it becomes Big Data problem[1][2].\n",
    "\n",
    "The \"best practise\" BioInfomatics Pipeline[3][4][5][6] involve multiple steps that produce rounds of data that get reduced into end of chain analysis summary metrics; often store in various formats -- from general formats such tsv, csv, txt to VCF. In this article we will focus on post analysis-ready, variants calling output, typically stored in VCF format. As details in NYGC's Somatic Pipeline[5] and UMCCRISE Pipeline[6] variant records get filtered and annotated against various databases and previously-studied nomenclature. This creates divergence among VCF files produced by different BioInfomatics Pipeline. This could also happen even within the same pipeline of different versions. Therefore, the content inside VCF file is heterogeneous by nature.\n",
    "\n",
    "Typically, BioInformatics only requires minimal metadata (such as individual details, clinical info, phenotypic data) to preform their pipeline analysis to study a case. Often, this rich-metadata is required to reconnect after post-analysis stage of BioInfo pipeline such compiling clinical reporting or further data sharing and cohort discovery purpose.\n",
    "\n",
    "As discussed, Genomic data is big. Therefore, keeping up and maintaining 24/7 online traditional database server is not favourable option in terms of cost and scalability consideration. Work load is often spike usage load or, it is stochastic data processing nature in most use case scenario. We wish to leverage Serverless Cloud computing services as much as possible such as AWS S3 object store, AWS Athena and AWS Lambda, etc. At the same time, we wish to perform streamlined migration into more performant system such as Redshift data warehouse service with minimal engineering efforts.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Steps\n",
    "\n",
    "In the following, we attempt to set up local development environment to study how to use Spark, Glow to explore some \"gold\" label VCF output from some \"best practice\" BioInfo Pipelines. Then, we make some data transformation and write final dataframe as Deltatable format.\n",
    "\n",
    "We will use JupyterLab as our development editor and, execute each notebook cell block to observe the output interactively.\n",
    "\n",
    "### Toolchain\n",
    "\n",
    "As prior exercise, it is recommended to try the following repos to understand basic of Spark and LakeHouse tables.\n",
    "\n",
    "* https://github.com/victorskl/deltalake-tute\n",
    "* https://github.com/victorskl/iceberg-tute\n",
    "* https://github.com/victorskl/hudi-tute\n",
    "\n",
    "First, we need to have local development toolchain ready. It is recommended have it installed the following JDK version:\n",
    "\n",
    "- JDK 11\n",
    "- Apache Maven 3.8\n",
    "\n",
    "If you are on macOS, this can be like:\n",
    "\n",
    "```\n",
    "brew install maven openjdk@11\n",
    "```\n",
    "\n",
    "The following are versions print out as guideline:\n",
    "```\n",
    "java -version\n",
    "  java version \"11.0.12\" 2021-07-20 LTS\n",
    "\n",
    "mvn --version\n",
    "  Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)\n",
    "```\n",
    "\n",
    "Next, create our main conda environment:\n",
    "```\n",
    "conda create -n genomic-bigdata-spark python=3.10\n",
    "conda activate genomic-bigdata-spark\n",
    "```\n",
    "\n",
    "Pull our main dependencies:\n",
    "```\n",
    "pip install -r requirements.txt\n",
    "```\n",
    "\n",
    "Pull additional Java packages:\n",
    "```\n",
    "mvn dependency:tree\n",
    "```\n",
    "\n",
    "```\n",
    "[INFO] Scanning for projects...\n",
    "[INFO]\n",
    "[INFO] ---------------< com.sankholin.genomic.bigdata:SparkApp >---------------\n",
    "[INFO] Building SparkApp 0.0.1\n",
    "[INFO] --------------------------------[ jar ]---------------------------------\n",
    "[INFO]\n",
    "[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ SparkApp ---\n",
    "[INFO] com.sankholin.genomic.bigdata:SparkApp:jar:0.0.1\n",
    "[INFO] +- net.sf.jopt-simple:jopt-simple:jar:5.0.3:compile\n",
    "[INFO] \\- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile\n",
    "[INFO] ------------------------------------------------------------------------\n",
    "[INFO] BUILD SUCCESS\n",
    "[INFO] ------------------------------------------------------------------------\n",
    "```\n",
    "\n",
    "### Data\n",
    "\n",
    "First, we need to download data from bcBio somatic-like tumor-normal Pipeline using Somatic truth sets from Genome in a Bottle (GiAB) samples output at [8]\n",
    "\n",
    "```\n",
    "wget https://s3.amazonaws.com/bcbio/giab/NA12878-NA24385/2018-07-05/na12878-na24385-somatic-hg38-truth.vcf.gz\n",
    "wget https://s3.amazonaws.com/bcbio/giab/NA12878-NA24385/2018-07-05/na12878-na24385-germline-hg38-truth.vcf.gz\n",
    "```\n",
    "\n",
    "Next, we create VCF stats file using `bcftools` as follows.\n",
    "\n",
    "```\n",
    "bcftools stats na12878-na24385-somatic-hg38-truth.vcf.gz > na12878-na24385-somatic-hg38-truth.stats.txt\n",
    "bcftools stats na12878-na24385-germline-hg38-truth.vcf.gz > na12878-na24385-germline-hg38-truth.stats.txt\n",
    "```\n",
    "\n",
    "Next, we create `metadata.csv` that mock minimal meta information about individual details, some clinical and Phenotypic data mock up. To simplify thing, we arrange it in flat denormalised table structure.\n",
    "\n",
    "### JupyterLab\n",
    "\n",
    "Launch Jupyter server by entering from terminal:\n",
    "\n",
    "```\n",
    "jupyter-lab\n",
    "```\n",
    "\n",
    "From JupyterLab interface, we open each notebook one by one in order. Execute each cell to make our observation.\n",
    "\n",
    "```\n",
    "01_glow_vcf_explore_somatic.ipynb\n",
    "02_glow_vcf_explore_germline.ipynb\n",
    "03_glow_vcf_explore_merged.ipynb\n",
    "04_glow_vcf_deltatable_somatic.ipynb\n",
    "05_glow_vcf_deltatable_germline.ipynb\n",
    "06_metadata_deltatable.ipynb\n",
    "07_querying_lakehouse.ipynb\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VCF Exploration with Glow\n",
    "\n",
    "### Single VCF\n",
    "\n",
    "First 3 notebooks simply explore VCF file content using Glow. Under the hood, Glow use `htsjdk` [9] for initial parsing then it parses VCF lines into Spark dataframe rows.\n",
    "\n",
    "```\n",
    "01_glow_vcf_explore_somatic.ipynb\n",
    "02_glow_vcf_explore_germline.ipynb\n",
    "03_glow_vcf_explore_merged.ipynb\n",
    "```\n",
    "\n",
    "There are few Glow parameters that can be configured such as\n",
    "* `includeSampleIds` to whether include Sample ID from VCF file\n",
    "* `flattenInfoFields` to whether to flatten all VCF INFO headers\n",
    "\n",
    "We typically like to retain as much info as possible, therefore, we keep these flags default `true`.\n",
    "\n",
    "From notebook exploration, we can observe that Glow attempt to rename some columns from VCF header such as\n",
    "```\n",
    "CHROM -> contigName\n",
    "POS -> end\n",
    "REF -> referenceAllele\n",
    "ALT -> alternateAlleles\n",
    "```\n",
    "\n",
    "Note that, Glow added `start` column which is 1 less of `POS` or now `end` column. This explains in Glow documentation\n",
    "> \"VCF reader uses the 0-start, half-open (zero-based) coordinate system [...] When writing to a VCF file, Glow converts positions back to a 1-based coordinate system as required by the VCF specification.\"\n",
    "\n",
    "### Merged VCF\n",
    "\n",
    "Next notebook, we attempt to merge read both VCF at one Spark dataframe and/or in-memory SQL-like \"Table View\". We can observe that Glow/Spark auto-merge the column schema to union. From previous, we dump each parsed VCF schema into text files. So, we can compare to see the difference as follows.\n",
    "\n",
    "```\n",
    "diff schema_bcbio_giab_somatic.txt schema_bcbio_giab_germline.txt\n",
    "\n",
    "22,23d21\n",
    "<  |-- INFO_FREQ: array (nullable = true)\n",
    "<  |    |-- element: string (containsNull = true)\n",
    "37,38d34\n",
    "<  |-- INFO_SOMTYPE: array (nullable = true)\n",
    "<  |    |-- element: string (containsNull = true)\n",
    "```\n",
    "\n",
    "Therefore, it tells Somatic VCF contains two more columns `INFO_FREQ` and `INFO_SOMTYPE`. We then continue to explore single merged table for some basic Genomic data exploration.\n",
    "\n",
    "#### Filtering Somatic or Germline\n",
    "\n",
    "Single merged `vcf_table` is great! However, can we still filter, say, to query Somatic records only or vice versa? Recall that `INFO_SOMTYPE` or `INFO_FREQ` columns only present in Somatic VCF header. Hence, we can approximate `NULL` record present of either column as pivotal data filter. Then we try example with `INFO_SOMTYPE` column being `NULL` or otherwise to infer Somatic variant records or Germline variant records.\n",
    "\n",
    "### Summary\n",
    "\n",
    "We can process VCF by their study type Somatic or Germline calling or both. Assumption is that within each \"best practice\" BioInformatics Pipeline; it should generate similar VCF structure with minor differences in header annotation.\n",
    "\n",
    "Couple of strategy possible:\n",
    "* by arranging all Somatic VCF type of the same BioInfo Pipeline output into one table\n",
    "* similarly, all Germline VCF type of the same BioInfo Pipeline output into one table\n",
    "* if we merged VCF, make sure to have very discriminator column that can filter data records better\n",
    "    * depends on data pipline setup, this discriminator column can be inserted during post-processing VCF files into warehouse table\n",
    "    * or, could add as part of BioInfo Pipeline VCF annotation process\n",
    "\n",
    "By default, Spark/Glow auto-discover \"schema\" out of VCF. Hence, \"schema evolution\" is possible. Alternatively, we can prescribe \"pre-defined schema\" to Spark/Glow during parsing. Hence, enforcing \"strict schema\". _FIXME: This the expectation and yet to confirm._\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VCF to Deltatable\n",
    "\n",
    "In this section, we perform ETL process --\n",
    "\n",
    "* _Extract_ data from VCF file\n",
    "* _Transform_ Genotype column to flatten structure\n",
    "    * Perform QA on transformed Spark dataframe\n",
    "* _Load_ final dataframe into LakeHouse as Deltatable\n",
    "\n",
    "```\n",
    "04_glow_vcf_deltatable_somatic.ipynb\n",
    "05_glow_vcf_deltatable_germline.ipynb\n",
    "```\n",
    "\n",
    "We use `flatten_struct_fields()` function from [10] as reference implementation.\n",
    "\n",
    "Please also observe that, we choose per deltatable by pipeline analysis type i.e. `somatic_table` and `germline_table` for better separation of concern and data isolation. Often, Germline kind of data requires more Data Use Access Control [11][12][13] than Somatic (cancer tumor mutation variants) kind of data.\n",
    "\n",
    "### Summary\n",
    "\n",
    "As we read VCFs through Spark/Glow, we could reduce the information as need be. Or, perform aggregation and store that much only. However, our focus for now is to keep data as-is as much as possible for simplicity.\n",
    "\n",
    "We perform data transformation on Genotype column. We flatten (i.e. apply data normalisation) the nested vectors of Genotype column. So that it can be performant and ease of use at data query (i.e. trade off for information retrieval time) by its predicates such as Sample ID. We then write this dataframe out as multi-parts compressed Parquet files through Deltatable framework.\n",
    "\n",
    "We can write Spark dataframe in multiple Deltatable write modes such as:\n",
    "* We would use `upsert` mode when/if we were to update or insert new records into existing delta table.\n",
    "* Or, `append` mode, if destination table is immutable and, so on so ford.\n",
    "\n",
    "If we wish, we could also write as-is Parquet only format and arrange them in \"Traditional Datalake Structure\".\n",
    "* e.g. `bcbio_somatic_df_exploded_flatten.write.format(\"parquet\").mode(\"overwrite\").save(\"./datalake/pipeline=bcbio/type=somatic/year=2022/month=01\")`\n",
    "\n",
    "In this case, we will be responsible for maintaining Datalake structure and its key-value partitioning such as `pipeline=bcbio`, etc. With Deltatable framework, we abstract away this and leave it up to Deltatable framework; and achieve \"Logical Table\" that perform like relational database table. Hence, Deltatable as such technology underpin \"LakeHouse\" architecture pattern for BigData data warehousing possibility.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metadata as Deltatable\n",
    "\n",
    "Next, we ingest mock metadata as deltatable.\n",
    "\n",
    "\n",
    "```\n",
    "06_metadata_deltatable.ipynb\n",
    "```\n",
    "\n",
    "Fist, this `meatadata.tsv` is total make up mock for demonstration. To simplify thing, it is prepared in denormalized flat table form.\n",
    "\n",
    "### Data Warehouse Models\n",
    "\n",
    "> As a demonstration, let us consider some ETL complex transformation have done here!\n",
    "\n",
    "At this point, we can take one step back and think about leveraging Data Warehouse data models. DataBricks has series of articles [14][15][16] for this.\n",
    "\n",
    "Some examples but not limited to:\n",
    "\n",
    "* Multi-Dimensional data model (_FIXME_ cite: Kimball et al Data Warehouse Model) such as Star schema, Snowflake schema\n",
    "    * For example, wrap around our `somatic_table` as Fact table and, build metadata as surrounding dimension table(s)\n",
    "* Data Vault data modelling (FIXME cite: Data Vault 2.0 standard) such as arranging in Hub, Satellite, Links concepts\n",
    "    * depends on data stage and use case, it may or may not be over engineering using DV2.0\n",
    "* Relation model with Data Mart (_FIXME_ cite: Inmon et al Data Warehouse Model) if that suit for the use case\n",
    "* Or even, just simple flat table as example in `metadata.csv`\n",
    "\n",
    "This modelling effort is crucial for query performance and use case towards BigData 4Vs: volume, velocity, variety, veracity [18].\n",
    "\n",
    "### Summary\n",
    "\n",
    "In real world scenario, this could involve much more complex structure and/or integration with upstream systems. For example, interfacing with systems such as RedCAP, FHIR, or Pathling look up, ontology and some kind of clinical Phenotype look up server. There may be possibility with dynamically looking them up from these systems.\n",
    "\n",
    "Often, we may not be able to extract out such metadata due to PII (Personally Identifiable Information). So, consider, we need to ingest some minimal metadata info and, need to ETL out minimal Phenotype info from these systems. We may or may not need to ingest these meta information into data warehouse -- which depends on case by case basis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying LakeHouse\n",
    "\n",
    "In this final notebook, we are going to consume data from the data warehouse that we just built.\n",
    "\n",
    "```\n",
    "07_querying_lakehouse.ipynb\n",
    "```\n",
    "\n",
    "### Cloud Native\n",
    "\n",
    "At this point, we could simply sync up the whole directory `./lakehouse` into Cloud Object store.\n",
    "\n",
    "As an example for AWS, we could perform these steps:\n",
    "\n",
    "* Upload our warehouse to S3: `aws s3 sync ./lakehouse s3://org-datalake-prod/v1/lakehouse`\n",
    "* Setup [AWS Glue to crawl Deltatables](https://aws.amazon.com/blogs/big-data/crawl-delta-lake-tables-using-aws-glue-crawlers/)\n",
    "* Then query with AWS Athena\n",
    "\n",
    "Similar Cloud services should exist in Azure or GCP. For other Cloud and private Cloud infrastructure, we could leverage PrestoDB, Trino, Hive to front our Genomics data warehouse. For more advanced cases, we could utilise a better High throughput dedicated managed data warehouse services such as Redshift, Synapse, BigQuery, etc.\n",
    "\n",
    "> Key point: Our Genomic BigData warehouse is in Cloud-native and can be scale-out by leveraging _state-of-the-art_ Cloud auto-scaling services.\n",
    "\n",
    "### Load Lakehouse Tables\n",
    "\n",
    "For our local development purpose, we mimic this Cloud-native effect by loading each Deltatables into separate spark dataframes. Then, we create in-memory \"Table View\" and, use SparkSQL to mimic SQL like query experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "### Perks\n",
    "\n",
    "In real world scenario, join tables call between Genomic Tables and Metadata table may or may not make sense. However, it demonstrates the use case possibility. Often, it is a desired missing feature in Genomic community i.e. combined vantage view of both Genotype and Phenotype data together which could help in Cohort discovery process.\n",
    "\n",
    "Often, these warehouse tables may span across multiple sources; AWS Athena, Trino as such distributed query engine enable \"Federated Query\" interface. For example, as such Genomic data warehouse could be backend of [GA4GH Beacon](https://github.com/ga4gh-beacon/beacon-v2/) and/or [GA4GH Data Connect](https://github.com/ga4gh-discovery/data-connect) REST endpoint interfaces.\n",
    "\n",
    "We also note that it opens up data into more general tooling for information retrieval e.g. SQL. As such warehouse tables enable OLAP -- Online Analytical Processing and, further bridge into data science such ML/AI. Rather than data sit in \"dark\" and only have knowledgeable by sudden key person in a team.\n",
    "\n",
    "Depends on data arrangement, we may treat such Genomic Table as central Fact table -- which we could use surrounding Metadata table(s) as dimensional look up. Or, we may be just focusing on Genomic Tables for some number crunching or aggregation for reporting. The process brings the data to one step closer to Cloud for computation -- i.e. concept such as \"Data & Compute Closer Together\". Hence, avoid data staging time and leverage more \"Cloud-native\" approaches for data analysis.\n",
    "\n",
    "### Contrary\n",
    "\n",
    "To note that;\n",
    "\n",
    "Certainly, experienced/trained BioInformatician can leverage more efficient data manipulation tools (such as `bcftools`) in a specialised-specified ad-hoc setup. This would work efficiently well so that we would just have to allocate resource for personnel to man the process; if we agree on this view.\n",
    "\n",
    "Genomic Table and, as such sourcing data warehouse directly from VCF could easily get out of hand. Typically, a \"best practice\" BioInformatics Pipeline should have finer, distilled, end-of-chain, \"gold\" label VCF product. This VCF file or metrics/fact files should contain a handful of records per patient that have already analysed & annotated well for a given use case. If sourcing Genomic Table from VCF is not appropriate then one could leverage more summarised tables output from MultiQC or some Cancer Reporter as see fit.\n",
    "\n",
    "Hence, this kind of Genomic BigData warehouse would still rely on high quality output of BioInformatics Pipeline. And continuation of overall Data Pipeline as depict in figure below.\n",
    "\n",
    "![GenomicBigData.png](./assets/GenomicBigData.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future Work\n",
    "\n",
    "### Point 1: Scalable and Feasibility\n",
    "\n",
    "We hypothesize that, this approach should scale out well in theory. However, this has yet to evaluate further on real world data workload (case by case basis) and, setting up some decent size cohort-wide data warehousing. Performance and feasibility study benchmarking on various technologies that underpin the setup. There is also to explore nuances of data partitioning within Deltatable or chosen LakeHouse table framework.\n",
    "\n",
    "### Point 2: BioParquet and GenomicTable\n",
    "\n",
    "We could also explore a more specialised Cloud-native BioInfo formats: `BioParquet` _ala_ GeoParquet[19]? We can also do comparison study with other LakeHouse table frameworks: Iceberg[20], Hudi[21] -- of which may or may not work better than Deltatable[22] in Genomics data context, for example. Similarly, this entails more specific to Genomic such that pursuit into `BioTable` or `GenomicTable` LakeHouse table specification effort?\n",
    "\n",
    "### Point 3: Column and Row Level Security\n",
    "\n",
    "Since underlay data structure are now arranged in tabular form, we could explore data governance in terms of:\n",
    "- Column Level Data Masking\n",
    "- Row Level Data Masking\n",
    "- Cell Level Data Masking\n",
    "\n",
    "Data masking or data access control list (ACL) is often desirable  in dealing with Human Genomics data; example scenario such as \"Can we lock down or mask out by Chromosome number\"; whereas there exists `chrom` or `contigName` column; or by masking some Genomics coordinate such as `POS` or `start` to `end` ranges, or well-known gene regions.\n",
    "\n",
    "Another hypothesis here is that, this should also work well in theory. As such Column, Row or Cell level data masking technique is well studied topics in traditional relational database.\n",
    "\n",
    "For those who are of special interest in this topic, the following are further pointers:\n",
    "- [AWS Governed Tables](https://www.google.com/search?q=aws+governed+tables)\n",
    "- [parquet column data masking](https://www.google.com/search?q=parquet+column+data+masking)\n",
    "- [parquet column level encryption](https://www.google.com/search?q=parquet+column+level+encryption)\n",
    "- [relational database column level data masking](https://www.google.com/search?q=relational+database+column+level+data+masking)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acknowledgement\n",
    "\n",
    "* First to HBC/BcBio repo (https://github.com/hbc/projects/tree/master/giab_somatic) to make data publicly available for use.\n",
    "* Thanks to Oliver Hoffman, _UMCCR Genomic Platform Group_ lead for allow me to use some of my UMCCR time and resources to perform this exploration.\n",
    "* Also thanks to colleagues at UMCCR team to providing me with all knowledgeable pointers.\n",
    "* Finally, thanks to all stakeholders to make these technology and tools available as Open source."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REF\n",
    "\n",
    "```\n",
    "[0] https://www.databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html\n",
    "[0] https://www.databricks.com/blog/2021/08/30/frequently-asked-questions-about-the-data-lakehouse.html\n",
    "\n",
    "[1] https://www.genome.gov/about-genomics/fact-sheets/Genomic-Data-Science\n",
    "[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4494865/\n",
    "\n",
    "[3] https://gatk.broadinstitute.org/hc/en-us/articles/360035894731-Somatic-short-variant-discovery-SNVs-Indels-\n",
    "[4] https://gdc.cancer.gov/about-data/gdc-data-processing/genomic-data-processing\n",
    "[5] https://www.nygenome.org/bioinformatics/3-cancer-cell-lines-on-2-sequencers/\n",
    "[6] https://github.com/umccr/umccrise/blob/master/workflow.md\n",
    "[7] https://samtools.github.io/hts-specs/\n",
    "\n",
    "[8] https://github.com/hbc/projects/tree/master/giab_somatic\n",
    "[9] https://glow.readthedocs.io/en/latest/etl/variant-data.html#vcf\n",
    "\n",
    "[10] https://github.com/microsoft/genomicsnotebook\n",
    "\n",
    "[11] https://ga4gh-duri.github.io/\n",
    "[12] https://pubmed.ncbi.nlm.nih.gov/34820660/\n",
    "[13] https://github.com/ga4gh-duri/ga4gh-duri.github.io\n",
    "\n",
    "[14] https://www.databricks.com/blog/2022/06/24/data-warehousing-modeling-techniques-and-their-implementation-on-the-databricks-lakehouse-platform.html\n",
    "[15] https://www.databricks.com/blog/2022/05/20/five-simple-steps-for-implementing-a-star-schema-in-databricks-with-delta-lake.html\n",
    "[16] https://www.databricks.com/blog/2022/06/24/prescriptive-guidance-for-implementing-a-data-vault-model-on-the-databricks-lakehouse-platform.html\n",
    "[17] https://datavaultalliance.com\n",
    "[18] https://opensistemas.com/en/the-four-vs-of-big-data/\n",
    "\n",
    "[19] https://github.com/opengeospatial/geoparquet\n",
    "[20] https://iceberg.apache.org\n",
    "[21] https://hudi.apache.org\n",
    "[22] https://delta.io  (_FIXME_ cite: chronologically it should come earlier)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}