Resolve several TODOs

Author: Joe Roe

paper.pdf

@@ -72,12 +72,13 @@ execute:
   cache: true
 ---
 
-```{r setup}
+```{r setup, cache=FALSE}
 #| include: false
 library("countrycode")
 library("cowplot")
 library("dplyr", warn.conflicts = FALSE)
 library("dm")
+library("english")
 library("giscoR")
 library("ggplot2")
 library("glue")
@@ -92,6 +93,7 @@ library("RPostgres")
 library("sf")
 library("spatstat")
 library("stars")
+library("stringr")
 library("tidyr")
 library("webshot2")
 
@@ -124,7 +126,7 @@ As a necessary prerequisite to understanding the context of any past event or pr
 If archaeology is to be an open science [@Lake2012], it is therefore critical that effective open access to chronological information be placed front and centre.
 
 Over the last two decades, archaeologists have answered this call by publishing an increasing number of compilations of dates from archaeological contexts as open data.
-These efforts have facilitated major reevaluations of previously-established chronologies [e.g. @HighamEtAl2014; @LoftusEtAl2019; @PratesEtAl2020; @KatsianisEtAl2020], important new insights into past processes [@RirisEtAl2024], and the development of novel ways of using chronological data [@Crema2022].
+These efforts have facilitated re-evaluations of chronologies themselves [e.g. @HighamEtAl2014; @LoftusEtAl2019; @PratesEtAl2020; @KatsianisEtAl2020] but also the development of novel ways of using chronological data [@MoodyEtAl2021; @Crema2022; @RirisEtAl2024].
 <!-- TODO: Would welcome more citations here -->
 The focus has been overwhelmingly on radiocarbon dating and most compilations focus on a single region and/or period.
 The profusion of open radiocarbon data in particular has prompted several initiatives towards a global synthesis [e.g. @SchmidEtAl2019; @BronkRamseyEtAl2019; @BirdEtAl2022].
@@ -147,7 +149,6 @@ Since we envisage both XRONOS as a dataset and XRONOS as software to be continua
 ## Compilations of radiocarbon dates {#sec-c14-compilation}
 
 Though an explicit emphasis on 'open data' is a relatively recent phenomenon in archaeology [@Lake2012], the open publication of compiled radiocarbon dates has a substantial prehistory.
-<!-- TODO: check and update this about date lists... -->
 Arnold and Libby [-@ArnoldLibby1951] initiated the tradition of regularly publishing all the dates they had obtained.
 This practice was subsequently continued as radiocarbon laboratories periodically shared and compiled their own 'date lists', published mainly in the journals *Radiocarbon* and *Archaeometry*.
 However, as the number of labs and volume of radiocarbon dates being produced grew, this paper-based format became impractical and mostly disappeared [with exceptions, e.g. @NdeyeEtAl2022] without being replaced by another form of systematic data-sharing or dissemination [@BronkRamseyEtAl2019].
@@ -169,7 +170,7 @@ Our review of the literature identified `r n_c14_datasets` published since 1994.
 This is almost certainly an undercount, because our firsthand knowledge of regional literature was limited to Europe and West Asia and many resources only ever existed in 'grey' formats (e.g. websites that were not indexed and no longer exist).
 We also restricted ourselves to structured datasets disseminated primarily in a digital format; 
 'date lists' in printed periodicals and gazetteers were excluded.
-A full list of the datasets we identified is presented in appendix XXXX<!-- TODO -->.
+A full list of the datasets we identified can be found in the supplementary materials.
 
 ```{r fig-c14-datasets-time}
 #| fig-cap: Cumulative number of radiocarbon compilations published since 1995 according to our survey (see Supplementary Material).
@@ -194,7 +195,7 @@ Notable early examples include ANDES 14C in 1994 [Central Andes, @MichczynskiEtA
 From 2010, coinciding with broader shifts in scientific publishing [@TenopirEtAl2011], it became more common to publish standalone 'open data' products in the form of journal supplements, archives in repositories and/or data papers;
 the *[Journal of Open Archaeology Data](<https://openarchaeologydata.metajnl.com>)*, launched in 2012, has been a prominent venue for this latter category.
 Most recently there has been a trend towards providing version-controlled plain text data via platforms such as [GitHub](https://github.com), reflecting the broader adoption of these tools amongst computational archaeologists over the last decade [@BatistRoe2024].
-The shift from online databases towards more static but more preservable open data products is welcome, given how many databases from the first generation have subsequently ceased to be accessible.  <!-- TODO: can this be quantified or visualised? -MH -->
+The shift from online databases towards more static but more preservable open data products is welcome, given how many databases from the first generation have subsequently ceased to be accessible.
 Version-controlled repositories are particular well-suited to data compilation projects because they allow for continued updates whilst still providing snapshot 'releases' that are citeable and can be archived in long-term repositories.
 
 ```{r data-basemap}
@@ -251,10 +252,11 @@ Most radiocarbon datasets we reviewed were compiled with a specific goal in mind
 Laboratory databases solve the problem of currency, but tend to have more arbitary coverage, since the inclusion of data is determined by who submits dates to that lab, not any form of principled curation.
 There are also comparatively few of them – most active labs no longer directly publish dates that they produce (if they ever did).
 
-In addition, the temporal and geographic coverage of these resources is uneven [@ChaputGajewski2016; @AlcantaraPedrozainpress], systematically biased [@ClistEtAl2023], and duplicative of each other.
-<!-- TODO: example, X databases for Europe, Y for the rest of the world? -->
-By our count, <!-- TODO: X --> of the databases are not 'open' according to the Open Knowledge Foundation's definition of data openness ["Open data and content can be freely used, modified, and shared by anyone for any purpose", @OpenKnowledgeFoundation], which both limits the access to and reuse potential of these datasets.
-<!-- TODO: X --> are not currently available in readily machine-readable formats (e.g. plain text or database files rather than PDFs or hypertext).
+The temporal and geographic coverage of these resources is uneven [@ChaputGajewski2016; @AlcantaraPedrozainpress], systematically biased [@ClistEtAl2023], and duplicative.
+For example, we identified `r english(sum(str_detect(c14_datasets$m49_region, "Western Europe"), na.rm = TRUE))` different databases covering Western Europe but none covering South Asia.
+The quality and accessibility of published compilations is also variable.
+`r str_to_sentence(english(sum(c14_datasets$open, na.rm = TRUE)))` of the `r english(nrow(c14_datasets))` resources we reviewed are not 'open' according to the Open Knowledge Foundation's definition of data openness ["Open data and content can be freely used, modified, and shared by anyone for any purpose", @OpenKnowledgeFoundation], which both limits the access to and reuse potential of these datasets.
+And even of these, many are not currently available in readily machine-readable formats (e.g. plain text or database files rather than PDFs or hypertext).
 
 The fragmentation of the radiocarbon record into regional datasets also hinders analysis at larger scales.
 Although the core elements of a radiocarbon date—laboratory identifier, radiocarbon age, measurement error—are more or less standardised, there is no such consistency in contextual information on the sample or site.
@@ -427,7 +429,8 @@ This is also typically present in many other forms of systematic compilation wor
 Aggregated typological information from such sources are often used in aoristic analysis and related methods [@Mischka2004; @Crema2024].
 What is lacking in this presentation of typological dating is metadata on how the determination was made and how exactly it is to be understood.
 Like any archaeological date, a typological date is derived from a physical sample – the object or set of object from which a chronological estimate was derived.
-Typological dates on one class of object may well clash with other classes of object, or for that matter with scientific dates, but without this kind of metadata such inconsistencies are difficult to resolve. <!-- TODO: clarify - MH -->
+Typological dates on one class of object may well clash with other classes of object, or for that matter with scientific dates – does one trust the date on pottery, the date on architecture, or the radiocarbon date?
+Without additional metadata on e.g. who made the typological determination or what the radiocarbon date was obtained on, such inconsistencies are difficult to resolve 
 Similarly the absolute date range corresponding to a typological determination (e.g. "Late Neolithic") can be interpreted in multiple ways depending on the region and intentions of the expert making the determination.
 PeriodO [@RabinowitzEtAl2016] is a linked open data infrastructure that includes a shared vocabulary of typological periods and corresponding calendar age estimates, and an important step towards addressing the latter problem.
 However, it remains to be systematically linked to actual compilations of typological dates.
@@ -444,22 +447,7 @@ Our overall aims in developing XRONOS is to bring this model, which RADON has op
 ## Design goals
 
 XRONOS is our answer to Kintigh's call [@Kintigh2006] for digital infrastructures that don't just provide access to chronological data but enables researchers to "archive, access, integrate, and mine disparate data sets".
-It parallels and draws inspiration from several similar initiatives within and outwith archaeology, such as <!-- TODO examples
-Examples:
-https://link.springer.com/article/10.1007/s10816-018-9399-6
-https://link.springer.com/article/10.1007/s10816-010-9098-4
-https://www.cambridge.org/core/journals/latin-american-antiquity/article/public-database-of-archaeological-resources-on-easter-island-rapa-nui-using-google-earth/F30677F6FC99B4C2BC762BCC98FD0966
-https://www.cambridge.org/core/journals/advances-in-archaeological-practice/article/data-integration-in-the-service-of-synthetic-research/E297C57441239AB35053F987AA758EB9
-https://www.cambridge.org/core/journals/advances-in-archaeological-practice/article/tdar/FCE1056A210A452E5FE65560763D4CC7
-https://www.cambridge.org/core/journals/advances-in-archaeological-practice/article/googlebased-freeware-solution-for-archaeological-field-survey-and-onsite-artifact-analysis/7914F673D1E1CAD96E5270E794CDFFD9
-https://link.springer.com/article/10.1007/s10816-015-9272-9
-https://link.springer.com/article/10.1007/s10816-015-9251-1
-https://link.springer.com/article/10.1007/s10816-015-9240-4
-https://anatomypubs.onlinelibrary.wiley.com/doi/10.1002/ar.23130
-https://academic.oup.com/dsh/article/29/3/349/2938128
-https://www.sciencedirect.com/science/article/pii/S0167739X13000678?via%3Dihub
-
--->.
+It complements several similar open data infrastructures within and outwith archaeology, such as SEAD for environmental archaeology [@Buckland2014], IMPACT for mummified human remains [@NelsonWade2015], Neotoma for palaeoecological data [@WilliamsEtAl2018], IsoArcH for stable isotope data [@PlompEtAl2022], and the 'Big Interdisciplinary Archaeological Database' (BIAD), an ambitious new initiative to combine many of these individual domains, including chronology [@ReiterEtAl2024].
 To improve upon existing global syntheses of radiocarbon dates (see @sec-global-compilations), we aimed to develop a living infrastructure that both continually collected data from diverse sources and presented a seamless single database to the user.
 The Global Biodiversity Information Facility [GBIF, <https://gbif.org>, @CanhosEtAl2004]—which provides a single, consistent interface to many sources of global biodiversity data—has served as an exemplar for us in this regard.
 
@@ -526,7 +514,7 @@ xronos_dm_svg <- xronos_dm |>
   dm_add_fk("versions", "item_id", "sites") |>
   dm_add_fk("versions", "item_id", "taxons") |>
   dm_add_fk("versions", "item_id", "typos") |>
-  # TODO: Self-references (primarily persuadable models), or too messy?
+  # Exclude self-references (primarily supersedable models) for readability, e.g.
   # dm_add_fk("contexts", "superseded_by", "contexts") |>
   dm_draw("BT", view_type = "title_only", column_types = TRUE) |>
   DiagrammeRsvg::export_svg()
@@ -537,7 +525,7 @@ write(xronos_dm_svg, xronos_dm_svg_file)
 ggdraw() + draw_image(xronos_dm_svg_file)
 ```
 
-<!-- TODO: shorten -->
+<!-- TODO: shorten? -->
 
 At the base of the XRONOS data model (@fig-data-model) are sets of spatiotemporal coordinates or, as we call them, *chrons*.
 In an archaeological context, we conceptualise a chron as an assertion linking human activity with a particular point in space and time.
@@ -638,9 +626,9 @@ This basic REST pattern is augmented by seven 'actions' (following the standard
 The 'show' action represents interaction with a single resource, as described above. 
 The 'index' action, which lists resources of a given type (e.g. <https://xronos.ch/c14s> for radiocarbon dates), is worth special mention because it is through this that the filtering logic at the core of XRONOS' two interfaces is implemented.
 By passing a query as HTTP GET parameters to the index action of a resource, the list returned the user is modified to only include records that match that query.
-For example, <https://xronos.ch/sites?site[country_code]=CH> (the part of the URL after the `?` character encodes the SQL WHERE clause `country = 'CH'` as a GET parameter) lists sites in Switzerland <!-- TODO: check this actually works -->.
+For example, <https://xronos.ch/sites?site[country_code]=CH> (the part of the URL after the `?` character encodes the SQL WHERE clause `country = 'CH'` as a GET parameter) lists sites in Switzerland.
 More complex queries can be executed using nested parameters.
-For example, <https://xronos.ch/c14s?c14[sample][material][name]=charcoal> (encoding that the `c14` table should be joined to the `material` table via `sample`, followed by the WHERE clause `material.name = 'charcoal'`) lists radiocarbon dates obtained from charcoal samples <!-- TODO: check that this actually works... -->.
+For example, <https://xronos.ch/c14s?sample[material][name]=charcoal> (encoding that the `c14` table should be joined to the `material` table via `sample`, followed by the WHERE clause `material.name = 'charcoal'`) lists radiocarbon dates obtained from charcoal samples.
 Uniquely, index actions can also respond with the result in a tabular data format (i.e. `.csv`).
 
 ## Data ingestion and curation {#sec-implementation-data}

paper.qmd

@@ -2,6 +2,7 @@ countrycode
 cowplot
 dplyr
 dm
+english
 giscoR
 ggplot2
 glue
@@ -14,10 +15,6 @@ purrr
 readr
 RPostgres
 sf
-abind
-stars
-tidyr
-webshot2
 spatstat.data
 spatstat.geom
 spatstat.random
@@ -27,3 +24,8 @@ rpart
 spatstat.model
 spatstat.linnet
 spatstat
+abind
+stars
+stringr
+tidyr
+webshot2