ANNIS for dates

[gcelano]

I would like to specify the date of composition for each work in my corpus, which I am likely to represent as a range (a fictitious example: -120:-110). Ideally, users should be able to specify a range in their query in ANNIS, and be able to retrieve all works/tokens falling at least within part of that range (for example, -130:-115 should also retrieve -120:-110, because the latter partly falls within the former).

Are there corpora where the date issue has already been tackled? In any case, what would be the best strategy to express this kind of annotation in PAULA XML? Should one first identify one "chunck" for all tokens of a work and then attach the date annotation to it?

[thomaskrause]

Hi!
Unfortunately we currently only have strings as annotation value type and this kind in ANNIS. I am not sure about Paula XML, but when it is converted, it would be represented as string anyway. We have a similar use case when we convert corpora with timeline items that include time codes. There we have something like start-end where start and end can be seconds with a decimal dot.

You should choose an encoding scheme that is easy to query using regular expressions. What easy means is in the eye of the user 😃 E.g. when using a fixed length for a single year (like 0100 instead of 100) so you can make range queries like [01][0-4].. which matches all years between 0 and 1499. Without the fixed length, this would still be possible, but would probably need more regex alternation to consider the different length. Or if you expect negative dates, always include the minus or the plus for the same reasons.

Also, you should consider whether it is more practical to have this encoded into two different annotations for the start and end date. This probably does not make it easier to find open-ranges, but it allows using the "value identity" operator == (or != for not equal values) as an additional constraint and filter for dates having the same start/end date. Maybe also @amir-zeldes can give some advise on this, especially on typical regex patterns in this case.

[amir-zeldes]

Generally I find ISO dates easiest to work with, because they sort properly and have predictable numbers of digits, e.g. today is:

2023-05-19

including leading zeros etc. So you can get the thirteenth century like this:

meta::date=/12.*/

Or thirteenth to fourteenth:

meta::date=/1[23].*/

If you need dates BC it's a little more complicated but if they're all BC it's easy, and otherwise you can use a disjunction.

[amir-zeldes]

This can be a little unpredictable for users - I think TEI and dialects recommend using attributes like notBefore, notAfter, etc., where you can still use ISO-like time formats for each one, then constrain on two metadata fields (though this is less useful for aggregation, if you want to group by period or the like)

[gcelano]

The notBefore and notAfter solution would be desirable if Annis could understand numbers. Creating regular expressions to match numbers can be confusing, especially if this involves two metadata fields.

[amir-zeldes]

Oh definitely, if we had proper number/date support that would be better, but at least numbers are predictable values for users, whereas things like "p1_2" can't be guessed, and don't generalized across a collection of corpora (we have dozens of Coptic corpora, so using actual date ranges seems the most comparable across the collection)

[gcelano]

Users do not have to guess anything, because there is documentation where date labels for each work can be checked.

The logic behind labels like "p5_2" or "m3_2" is clear: "p" means plus (i.e., CE), "m" minus (i.e., BCE). The following number (in my case, with the range 1-14) is the century and the last one is the half (1 or 2). The problem with knowing the possibile number ranges and how to write a proper regular expression to catch them is similar to the one you would have with notBefore/notAfter. Actually, I find it less perpiscuous to process REs with notBefore/notAfter: how would you write them to search for a work written in the 5th, 4th, or 3th century BCE (corresponding to /m[543]_[12]/ in my annotation style)?

[amir-zeldes]

Yeah, you're right, it's actually not so simple with notBefore/After... But if we are using a single number for a range like you are then I prefer to use:

• 550-599

Instead of p5_2, simply because it corresponds to a more standard notation. Then we can do the same kind of search:

• /^5[5-9][0-9]-5[5-9][0-9]$/

Or if we don't know whether it's an exact date or range:

• /^5[5-9]0-9?$/

Definitely could be nicer if we had support for real numbers 😅