Spacy step by step

[joancf]

When running Spacy, from gate using the spacy_annotator, GATE nlp incorporates all the annotations produced by spacy into the GATE document. But:

• can Spacy get any annotations from GATE document?
• Can I run Spacy over a tokenized text? or, Can I run some Spacy steps (for example tokenization and sentece split) , then use another annotator to add/modify some of the results produced by Spacy and then run some other spacy components?

[johann-petrak]

Currently, it is not possible to pass information about tokens or other information to Spacy by standard gatenlp methods. TBH I am not too familiar with all the details of how to use Spacy with pre-annotated text, but of course there is the complication that gatenlp supports arbitrary nested and overlapping annotations while Spacy does not.

I think it should be possible to implement something where gatenlp would pass an existing token sequence to spacy instead of the bare text, but it would be necessary to make sure that the token annotations cover the WHOLE document text, that all whitespace gets passed to spacy as well, and maybe that some gatenlp token features get converted to spacy token attributes?

If you could give a more detailed description of what you are thinking of, maybe it is possible to come up with something that is flexible and generic enough to justify adding to gatenlp?

Of course it is always possible to implement the missing pieces oneself in one's own Annotator and make use of existing methods or code from existing methods where necessary. For example if you have a specific way you want to pass on your tokens to spacy, you could implement the conversion from gatenlp to spacy yourself, then run the spacy pipeline and then maybe use gate.lib_spacy.spact2gatenlp(spacydoc, ...)` to convert back to a gatenlp document?

I wonder, if you need to do a large part of your processing in Spacy, which part would you still need to do with gatenlp?

[joancf]

I'll checkif spacy2gatenlp can do part of the job (maybe I will add an extra parameter or create a new function based on that one..)

• So I can take some (text) spans (this could be related to Generic annotator to annotate a segment of a document only #127 ) process them and add the annotation back to the document.
  Often spacy is used a full pipeline, but some it could also be run step by step. Basically, the pipeline does that:

doc = nlp.make_doc("This is a sentence") # Create a Doc from raw text
for name, proc in nlp.pipeline: # Iterate over components in order
doc = proc(doc) # Apply each component

So if we have: "gatenlp2spacy" that generates a spacy doc, then we could run som spacy procs and then back "spacy2gatenlp"
The most complex issue in this solution is how to deal with trasformers.