Ann Matcher

[joancf]

I want to check different parts of a text. I want to detect figure references. I could do a complex Regular expression but i want to do it using rules an annotate different parts of the text , but I don't know how to express the conditions over tokens
so I want to find expressions like and all the combinations...
Figures 2a, 3 and 5 or Figs. 15-17

I came to a solution like:

r = Rule(
        Seq ( Text(re.compile("fig(ure)?s?\\.?",re.IGNORECASE)),
                N(Ann("Token", features=dict(lemma=".")), min=0,max=1 )
                ,Ann("Token", features=dict(is_digit=True),name="ref")
                ,N( Seq( Ann("Token", features=dict(lemma=",")) ,
                        Ann("Token", features=dict(is_digit=True),name="ref1")
                        ), min=0,max=1 )
                ,N( Seq( Ann("Token", features=dict(lemma="and")) ,
                        Ann("Token", features=dict(is_digit=True),name="ref2")
                        ), min=0,max=1 )                     
            ,name="seq"),
        AddAnn(name="seq", type="figRefs")

An has the following parameters:
type=None, features=None, features_eq=None, text=None, name=None, useoffset=True
but I don't know hot to use all the potential they have
so ...

• instead of Ann("Token", features=dict(is_digit=True) I would like to indicate that the shape is like the regular expression "d+x"
• instead of Ann("Token", features=dict(lemma=",") i would like the lemma to be "," or "-" how can I do that?
• instead of Text(re.compile("fig(ure)?s?\\.?",re.IGNORECASE)) i would like to say that this is the content of the token (the covered text?)
  So arrived to this one... but still not happy

NumtextPattern = re.compile("d+x{0,2}$")
figRef=re.compile("fig(ure)?s?\\.?",re.IGNORECASE)
r2 = Rule(
        Seq ( Ann("Token", text=figRef),
                N(Ann("Token", features=dict(lemma=".")), min=0,max=1 )
                ,Ann("Token", features=dict(shape=NumtextPattern))
                ,N( Seq( Ann("Token", features=dict(lemma=",")) ,
                        Ann("Token", features=dict(is_digit=True))
                        ), min=0,max=1 )
                ,N( Seq( Ann("Token", features=dict(lemma="and")) ,
                        Ann("Token", features=dict(shape=NumtextPattern))
                        ), min=0,max=1 )                     
            ,name="seq3"),
        AddAnn(name="seq3", type="figName")

a-b still missing... but now on the way.!
(i think you can use them as examples... )

And then when processing I would like to annotate the different numbers ( ref, ref1 ref2) and the full reference or maybe better,
be able to generate the list of references. Like [2a,3] and [15,16,17] for this part maybe i need 2 rules (one for "," and the other for "-" to do so i need i need to run a process (my own function instead of AddAnn, but wich parameter would such a function receive? )

[joancf]

This one that seems simpler... a sequence of elements of different type does not work to me. I'm not sure what I'm doing wrong
A Sequence of
NounChunk of NounChunk

with this rule, i get no matches...

nlp = spacy.load("en_core_web_sm")
doc=Document ("The wheel of the car was found near the restaurant of the gas station of my friend. Nobody knows how the wheel arrived there. ")
doc=apply_spacy(nlp,doc,  setname="Original markups")
outset = doc.annset("Out")
r5 = Rule(
       Seq ( Ann("NounChunk"),
            Ann("Token", features=dict(lemma="of")), 
            Ann("NounChunk")
            , name="chunk"),
        AddAnn( name="chunk", type="extendedChunk")
)

annSet=doc.annset("Original markups").with_type(["NounChunk","Token"])
Pampac(r5).run(doc, annSet, outset=outset)

The expected result would be:

• The wheel of the car
• the restaurant of the gas station
• the gas station of my friend

[johann-petrak]

I want to check different parts of a text. I want to detect figure references. I could do a complex Regular expression but i want to do it using rules an annotate different parts of the text , but I don't know how to express the conditions over tokens
so I want to find expressions like and all the combinations...
Figures 2a, 3 and 5 or Figs. 15-17

I am not sure what exactly your question is/are?

* instead  of `Ann("Token", features=dict(is_digit=True)` I would like to indicate that the  shape is like the regular expression "d+x"

you can use a regex to match either the underlying text or the value of a feature, would that help?

* instead of   `Ann("Token", features=dict(lemma=",")`   i would like the lemma to be ","  or "-" how can I do that?

Again, since you can match a regex against a feature you can do something like feature=dict(lemma=re.compile(r",|-"))

* instead of  `Text(re.compile("fig(ure)?s?\\.?",re.IGNORECASE))` i would like to say that this is the content of the token (the covered text?)

This should be possible using Ann("Token", text=yourregexp, ...)

And then when processing I would like to annotate the different numbers ( ref, ref1 ref2) and the full reference or maybe better,
be able to generate the list of references. Like [2a,3] and [15,16,17] for this part maybe i need 2 rules (one for "," and the other for "-" to do so i need i need to run a process (my own function instead of AddAnn, but wich parameter would such a function receive? )

You can use any callable that takes the parameters: succ, context, location. So you could implement a class that has the method __call__(self, succ, context, location) or just any function with that argument list.

• succ: an instance of pam.pampac.data.Success which holds all the results that match
• context: an instance of pam.pampac.data.Context which contains the document we match on, the sequence of annotations, and the output annotation set
• location: an instance of pam.pampac.data.Location which roughly speaking contains the next location in the annotation sequence and document offset

[johann-petrak]

With the second example there are several issues:

• if you use the annotations "NounChunk" and "Token" then NounChunk will always start at the same offset as a Token but in the sequence of annotations we match against, one of the two will occur first. So in order to match something like Seq(Ann("Token"), Ann("NounChunk")) the NounChunk annotation would have to be the next in the sequence after the token, but since NounChunk gets added later it will be after the Token at the same offset. As I said before, to match any of the annotations at the offset of the next annotation, use AnnAt instead: Seq(AnnAt("Token"), AnnAt("NounChunk")) should work. In almost all cases you probably want to use "AnnAt" except when you really want to consider the sequence of annotations as it consists even at the same offset.
• In order to get all three matches, the third match would need to get checked at a location before the end of the previous match. By default this is not happening. In order to make this work, the additional Pampac init parameter skip="one" is needed. HOWEVER: the version of gatenlp has a bug for this parameter. I have committed a fix so if you use the latest master version, it should work.

For illustration I attach a notebook which illustrates this and should work with the very latest master version of gatenlp.
pampac-nested.zip

BTW thanks for you questions and feedback: PAMPAC is a very complex piece of software and there is a huge range of possible situations, of which only a small set have been tested yet. So the more it gets used the more of any bugs in there we should be able to squash!

[joancf]

Thanks for the sample notebook. Yes, this was difficult to understand from documentation. I think I was confused by this example here:
https://nbviewer.jupyter.org/urls/gatenlp.github.io/python-gatenlp/pampac.ipynb

Complex Patterns

More complex patterns can be created from the simple Terminal Parsers, for example, matching a sequence of terminal parsers:

Seq(Ann(anntype="Person"), Ann(anntype="Token"), Ann(anntype="Person"))

would match only if the annotation sequence contains annotations of exactly those types in sequence. This can also be written down using operator syntax, using the "Pampac sequence operator":

Ann(anntype="Person") >> Ann(anntype="Token") >> Ann(anntype="Person")

The ones with the >> also gave me errors...

I have found a strange behaviour. if i want to detect any sequence of [NP of]* NP I would write

outset1 = doc2.annset("Out1")
outset1.clear()
outset2 = doc2.annset("Out2")
outset2.clear()
annset = doc2.annset("Original markups").with_type("NounChunk", "Token")
r5 = Rule(
    Seq(
        N ( Seq(AnnAt("NounChunk"),
            AnnAt("Token", features=dict(lemma="of"))), min=0,max=4), 
        AnnAt("NounChunk"),
        name="chunk"),
    AddAnn( name="chunk", type="extendedChunk")
)

Pampac(r5).run(doc2, annset, outset=outset1)

# do not advance to the end of the match but to the next offset after the start of the earliest match
Pampac(r5, skip="one").run(doc2, annset, outset=outset2)

If minimum is 1 I get these results
extendedChunk: The wheel of the car
extendedChunk: the restaurant of the gas station of my friend
extendedChunk: the gas station of my friend

That looks fine, but if I set the minim to 0 then is seems to advance form char to char getting:

extendedChunk: The wheel of the car
extendedChunk: f the car
extendedChunk: the car
extendedChunk: the car
extendedChunk: the restaurant of the gas station of my friend
extendedChunk: the gas station of my friend
extendedChunk: f my friend
extendedChunk: my friend
extendedChunk: my friend
extendedChunk: Nobody
extendedChunk: Nobody
extendedChunk: ow the wheel
extendedChunk: w the wheel
extendedChunk: the wheel
extendedChunk: the wheel

instead of
extendedChunk: the car
extendedChunk: the wheel
extendedChunk: The wheel of the car
extendedChunk: the restaurant of the gas station of my friend
extendedChunk: the gas station of my friend
extendedChunk: the gas station
extendedChunk: the restaurant
extendedChunk: my friend
extendedChunk: the wheel
extendedChunk: Nobody