GATE COURSE MODULE 11
GATE & PYTHON

Johann Petrak and Mehmet Bakir

Last updated: 2022-06-24 for GateNLP v1.0.8a1 or later
  • Online slides: https://gatenlp.github.io/python-gatenlp/training/module11-python.slides.html
  • Slides License: CC BY-NC-SA 3.0

GATE & PYTHON¶

This tutorial covers Python tools related to GATE:

  1. Python GateNLP: Python package for NLP similar to Java GATE
  2. Python GateNLP GateWorker: run Java/GATE from Python
  3. GATE Python Plugin: Java GATE plugin to process GATE documents with Python and Python GateNLP
  4. Format BDOC Plugin: Java GATE plugin for support of loading/saving documents in JSON/YAML/MsgPack format

Python GateNLP¶

Aims:

  • NLP framework written in pure Python.
  • Similar concepts as Java GATE: documents, document features, annotation sets, annotations, ...
  • But "pythonic" API, try to make basic things very simple (e.g. loading/saving of documents)
  • Does NOT try to be a full multilingual NLP processing package, rather COMBINE:
    • Use existing tools and solutions: Spacy, Stanford Stanza
    • Add own tools and improvements where needed

Python GateNLP: status¶

  • Current release: 1.0.x
  • All 1.0.x: get community feedback:
    • how to improve API, abstractions, conventions, find bugs
    • what is most important to still get added?
    • API may slightly change, parameter names may get consolidated
  • Planned 1.1.x releases and onwards: stable API

Python GateNLP: Info and Feedback¶

  • Documentation: https://gatenlp.github.io/python-gatenlp/
  • Sources: https://github.com/GateNLP/python-gatenlp
  • Report a bug, request a feature with issue tracker: https://github.com/GateNLP/python-gatenlp/issues
  • Discuss, ask:
    • discussions forum at https://github.com/GateNLP/python-gatenlp/discussions
    • GATE mailing list https://groups.io/g/gate-users
  • Developers Chat: https://gitter.im/GateNLP/python-gatenlp

Preparation: Install Python¶

  • see also https://gatenlp.github.io/python-gatenlp/installation.html
  • Recommended:
    • Anaconda / Miniconda
    • Miniforge

Preparation: install Miniconda (Linux)¶

  • Download the Python 3.8 (or later) installer (64-bit) for your OS
  • Run the installer
  • respond "yes" to "running conda init?"
  • start a new command line
  • Create environment: conda create -y -n gatenlp python=3.9
  • activate environment: conda activate gatenlp

Preparation: install Miniconda (Windows)¶

  • Download the Python 3.8 (or later) installer (64-bit) for your OS
  • Run the installer, install for "just me", register as default Python,
  • start the "Anaconda Prompt" or "Anaconda Powershell Prompt"
  • Create environment: conda create -y -n gatenlp python=3.9
  • activate environment: conda activate gatenlp

Install gatenlp¶

To install most recent release and install all dependencies (without [all] only minimum dependencies are installed!):

pip install -U gatenlp[all]

Also install support for jupyter notebookd and for showing the slides:

pip install jupyter notebook ipython ipykernel RISE

Create kernel for the conda environment:

python -m ipykernel install --user --name gatenlp --display-name "Python-gatenlp"

See also GateNLP documentation: installation

Java GATE¶

  • Java is installed
    • e.g. Adoptium
    • java on the path and registered!
  • GATE 9.0 of later is installed
    • needed later: where (which directory) is it installed in? (GATE_HOME)
    • Windows: right-click icon, "Properties", "Target": directory that contains "gate.exe"

Follow along¶

  • Online slides
  • Download the handouts zip file and extract the directory
  • Within the directory you can either:
    • to follow in the original notebook: run jupyter notebook module11-python.ipynb
    • to explore in a new Notebook: run jupyter notebook, choose New -> Python-gatenlp
    • to explore interactively: run ipython and enter python code

If kernel error in Jupyter, try something like (Anaconda bug, apparently):

python C:\Users\USERNAME\miniconda3\envs\gatenlp\Scripts\pywin32_postinstall.py -install

Python GateNLP: Main Concepts¶

  • A document represents some text and
    • any number of named annotation sets
    • any number of features
  • An annotation set can have
    • any number of annotations
  • Annotations describe a span of a document and have
    • any number of features
    • an annotation type
    • from and to offsets that describe the span of the annotation
  • An Annotator is something that processes a document (and usually adds or changes annotations)

Documents¶

In [2]:
# Import gatenlp to check gatenlp version:
import gatenlp
print("GateNLP version:", gatenlp.__version__)
from gatenlp import Document
GateNLP version: 1.0.8.dev3

Create a document from some text/string and print it:

In [3]:
doc1 = Document("This is a small test document")
print(doc1)
Document(This is a small test document,features=Features({}),anns=[])

Documents¶

In a notebook, documents are visualized using the html-viewer when a document is the last value of a cell or when display(doc1) or when document.show() is used:

In [4]:
# from IPython.display import display
doc1
Out[4]:

Documents¶

The show() method can be used to influence and parametrize the viewer

In [5]:
doc1.show(doc_style="color: blue; font-weight: bold;")

Documents: load¶

  • to load documents use Document.load(some_location, ...)
  • document format is auto-detected from the extension or specified using the fmt parameter
  • one standard format for saving/loading GateNLP is "bdocjs" (a JSON serialization)
  • some_location can be file or URL
In [6]:
doc2 = Document.load('./data/document-testing.txt')
doc2
Out[6]:

Documents: save (JSON)¶

  • use thedocument.save(location, ...)
  • format is inferred from the extension or specified using fmt parameter
  • Formats: bdocjs (JSON, default), bdocym (YAML, slow), bdocmp (MessagePack, compact)
In [7]:
doc1.save("myfirstdocument.bdocjs")

with open("myfirstdocument.bdocjs", "rt", encoding="utf-8") as infp:
    print(infp.read())
{"annotation_sets": {}, "text": "This is a small test document", "features": {}, "offset_type": "p", "name": ""}

Document: save (YAML)¶

In [8]:
doc1.save("myfirstdocument.bdocym")  # use YAML serialization

with open("myfirstdocument.bdocym", "rt", encoding="utf-8") as infp:
    print(infp.read())
annotation_sets: {}
features: {}
name: ''
offset_type: p
text: This is a small test document

In [9]:
# Can also "save" to memory/string, here the format is needed!
doc1.save_mem(fmt="bdocjs")
Out[9]:
'{"annotation_sets": {}, "text": "This is a small test document", "features": {}, "offset_type": "p", "name": ""}'

Document features¶

  • Documents can have arbitrary features (similar to Python dictionaries)
  • key/name (string) maps to some value
  • value should be JSON serializable
  • name starting with single underscore: "private value"
  • name starting with double underscore: "private/transient value" (not saved by default, not shown in viewer)
In [10]:
import datetime
doc1.features["loading_date"] = str(datetime.datetime.now())
doc1.features["purpose"] = "Testing gatenlp."
doc1.features["numeric_value"] = 22
doc1.features["dict_of_objects"] = {"dict_key": "dict_value", "a_list": [1,2,3,4,5]}
doc1.features["_tmp1"] = "some value"
doc1.features["__tmp2"] = 12345
doc1
Out[10]:

Features: API¶

  • inherits from UserDict
  • same API as dict
  • but Document is aware of what changes are made (needed for updating a ChangeLog as we will see later!)
In [11]:
print("1:", doc1.features["purpose"])
print("2:", doc1.features.get("doesntexist"))
print("3:", doc1.features.get("doesntexist", "NA!"))
1: Testing gatenlp.
2: None
3: NA!

Features: API¶

In [12]:
for name, value in doc1.features.items():
    print(f"{name}: {value}")
loading_date: 2022-07-02 21:48:56.801623
purpose: Testing gatenlp.
numeric_value: 22
dict_of_objects: {'dict_key': 'dict_value', 'a_list': [1, 2, 3, 4, 5]}
_tmp1: some value
__tmp2: 12345

Features: serialization¶

Lets check how the document with features is serialized to "bdocjs" (JSON) format:

In [13]:
import pprint, json

js_str = doc1.save_mem(fmt="bdocjs")
js = json.loads(js_str)
pprint.pprint(js)
{'annotation_sets': {},
 'features': {'_tmp1': 'some value',
              'dict_of_objects': {'a_list': [1, 2, 3, 4, 5],
                                  'dict_key': 'dict_value'},
              'loading_date': '2022-07-02 21:48:56.801623',
              'numeric_value': 22,
              'purpose': 'Testing gatenlp.'},
 'name': '',
 'offset_type': 'p',
 'text': 'This is a small test document'}

Annotations & Annotation Sets & Spans¶

  • Span: a range of offsets
  • Annotation: information about a range of offsets, has
    • annotation type
    • features
    • unique integer annotation id
  • Annotation set: named collection of annotations
    • "set": only one annotation per set with the same annotation id
    • but ordered by insertion order or offset
    • "default" annotation set has name "" (empty string)

Adding annotations¶

  • first get the annotation set we want to add the annotation to
  • then create the annotation using the add method of the set
In [15]:
# create and get an annotation set with the name "Set1"
annset = doc1.annset("Set1")
#Now, add an annotation, this method returns the newly created annotation
annset.add(0,4,"AnnType1")
Out[15]:
Annotation(0,4,AnnType1,features=Features({}),id=0)

Annotations¶

  • The annotation covers the characters 0, 1, 2, and 3, a text of length 4 (to - from = len)
  • the "to" offset is the offset after the last covered character
  • in Python ALL unicode code points are represented by 1 character
    • In Java: UTF-16 code units
    • -> Offsets different between Java and Python!

Annotations¶

In [16]:
# add a few more
annset.add(0, 4, "Token", {"id": "token1'"})
annset.add(5, 7, "Token", {"id": "token2'"})
annset.add(8, 9, "Token", {"id": "token3'"})
annset.add(10, 15, "Token", {"id": "token4'"})
annset.add(16, 20, "Token", {"id": "token5"})
annset.add(21, 29, "Token", {"id": "token6"})
annset.add(0, 29, "Sentence", {"what": "The first 'sentence' annotation"});
for ann in annset:
    print(ann)
Annotation(0,4,AnnType1,features=Features({}),id=0)
Annotation(0,4,Token,features=Features({'id': "token1'"}),id=1)
Annotation(0,29,Sentence,features=Features({'what': "The first 'sentence' annotation"}),id=7)
Annotation(5,7,Token,features=Features({'id': "token2'"}),id=2)
Annotation(8,9,Token,features=Features({'id': "token3'"}),id=3)
Annotation(10,15,Token,features=Features({'id': "token4'"}),id=4)
Annotation(16,20,Token,features=Features({'id': 'token5'}),id=5)
Annotation(21,29,Token,features=Features({'id': 'token6'}),id=6)

Annotations: document viewer¶

In [17]:
doc1.show(preselect=[("Set1", ["AnnType1", "Sentence"])])
  • show all annotations for a type by clicking the type name
  • clicking annotation shows annotation features instead of document features
  • clicking "Document" shows the document features again
  • when multiple annotations overlap, need to select first which to view

Annotations/sets: remove¶

In [18]:
ann0 = annset.get(0)    # get by annotation id
print("Annotation id=0:", ann0)
annset.remove(ann0)     # remove the annotation with the annotation id of ann1
ann1 = annset.get(1)
print("Annotation id=1:", ann1)
annset.remove(1)   # remove the annotation with the given id
annset.remove([2,3,4])  # remove a whole list of annotations
print("After some anns removed ", annset)
annset.clear()
print("After set cleared: ", annset)
doc1.remove_annset("Set1")
Annotation id=0: Annotation(0,4,AnnType1,features=Features({}),id=0)
Annotation id=1: Annotation(0,4,Token,features=Features({'id': "token1'"}),id=1)
After some anns removed  AnnotationSet([Annotation(0,29,Sentence,features=Features({'what': "The first 'sentence' annotation"}),id=7), Annotation(16,20,Token,features=Features({'id': 'token5'}),id=5), Annotation(21,29,Token,features=Features({'id': 'token6'}),id=6)])
After set cleared:  AnnotationSet([])

Annotation Relations¶

  • Annotations can overlap arbitrarily
  • Annotation API has methods to check how they relate to each other
    • overlap, within, covering, before, after, rightoverlapping, startingat, endingwith, coextensive ...
  • Annotation API implements ordering by start offset and annotation id

Annotation Relations

  • Ann1 overlaps with all others, covers all but Ann2 and Ann4
  • Ann5 is directly before Ann3, is before Ann6
  • Ann10 starts at Ann1, Ann12 ends with Ann1, Ann3 and Ann9 are coextensive

Annotation Relations¶

Annotation Relations

Let's load and view an example document to demonstrate this:

In [20]:
doc3 = Document.load("data/ann-relations.bdocjs")
doc3.show(htmlid="view1")

Annotation Relations API¶

Annotation Relations

In [21]:
# make a variable for each annotation type
for anntype in list(doc3.annset("set1").type_names):
    vars()[anntype.lower()] = doc3.annset("set1").with_type(anntype).for_idx(0)
In [22]:
print("Ann2 isoverlapping Ann1:", ann2.isoverlapping(ann1))
print("Ann2 isbefore Ann3:", ann2.isbefore(ann3))
print("Ann3 isafter Ann2:", ann3.isafter(ann2))
print("Ann1 iscovering Ann5:", ann1.iscovering(ann5))
print("Ann3 iscoextensive Ann9:", ann3.iscoextensive(ann9))
print("Ann6 iswithin Ann1:", ann6.iswithin(ann1))
print("Ann4 isrightoverlapping Ann1:", ann4.isrightoverlapping(ann1))
Ann2 isoverlapping Ann1: True
Ann2 isbefore Ann3: True
Ann3 isafter Ann2: True
Ann1 iscovering Ann5: True
Ann3 iscoextensive Ann9: True
Ann6 iswithin Ann1: True
Ann4 isrightoverlapping Ann1: True

Spans¶

  • Objects that describe offset ranges
  • similar API for relations
  • can get from annotations, use when only the span of an annotation is needed
In [23]:
from gatenlp import Span
span1 = Span(3,4)
span2 = ann2.span
span3 = doc3.annset("set1").span
span4 = Span(ann5)
print([f"span{i}: {s}" for i, s in enumerate([span1, span2, span3, span4])])
    
['span0: Span(3,4)', 'span1: Span(0,6)', 'span2: Span(0,45)', 'span3: Span(12,18)']

AnnotationSet: retrieve by relation¶

  • get all annotations that overlap/are before/start at/... an annotation/span/annotation set
  • returns a new annotation set
  • returned set is detached: not part of document, changes do set not affect document
  • returned set is initially immutable: set cannot be changed
  • but annotations are mutable and still the same as in the set!
  • possible to "detach" annotations by (deep)copying them
In [24]:
set1 = doc3.annset("set1") # "attached" set
print("Within Ann1: ", [a.type for a in set1.within(ann1)])
print("Coextensive with Ann3:", [a.type for a in set1.coextensive(ann3)])
print("Coextensive with span of Ann3:", [a.type for a in set1.coextensive(ann3.span)])
Within Ann1:  ['Ann10', 'Ann5', 'Ann3', 'Ann7', 'Ann9', 'Ann11', 'Ann6', 'Ann8', 'Ann12']
Coextensive with Ann3: ['Ann9']
Coextensive with span of Ann3: ['Ann3', 'Ann9']

AnnotationSet: detached / immutable¶

In [25]:
print("Size of set1:", len(set1))
subset1 = set1.within(ann1)
print("Size of subset1:", len(subset1))
Size of set1: 12
Size of subset1: 9
In [26]:
# try to add an annotation to subset1:
try:
    subset1.add(2,3,"ANewOne")
except Exception as ex:
    print("Got exception:", ex)
Got exception: Cannot add an annotation to an immutable annotation set

AnnotationSet: detached / immutable¶

In [27]:
# make the set mutable and try again
subset1.immutable = False
subset1.add(2,3,"ANewOne")
print("Size of set1:", len(set1))
print("Size of subset1:", len(subset1))
print("Is set1 detached:", set1.isdetached())
print("Is subset1 detached:", subset1.isdetached())
Size of set1: 12
Size of subset1: 10
Is set1 detached: False
Is subset1 detached: True
  • annotation only got added to subset1, NOT the original set
  • detached sets cannot get attached again
  • annotations in the detached set are the same as in the document, so changing their features will affect the document!
  • detached set can also detach its annotations using subset1.clone_anns()

Document loading/saving¶

Supported formats:

  • bdocjs, bdocym, bdocmp: load/save (aliasing: only bdocym)
  • GATE xml: load (but only basic data types, no aliasing)
  • HTML: load and create annotations for HTML entities
  • plain text: load / save
  • tweet: load v1 format, WIP!
  • pickle: load/save
  • html-ann-viewer: save (also used for displaying in jupyter)

Document: load HTML¶

In [28]:
# lets load and view the main GateNLP documentation page:
doc4 = Document.load("https://gatenlp.github.io/python-gatenlp/", fmt="html")
doc4
Out[28]:

Document: view sets/types¶

Use: doc.show(annspec=["set1", ("set2", "type1"), ("set3", ["type1", "type2"])]

In [29]:
doc4.show(annspec=[("Original markups", ["h1","h2","a","li"])])

Document: save html-ann-viewer¶

In [30]:
doc4.save("gatenlp-doc.html", fmt="html-ann-viewer", notebook=False, stretch_height=True)
from IPython.display import IFrame
IFrame("gatenlp-doc.html", 900,400)
Out[30]:

Exchange Documents with Java GATE¶

  • Python GateNLP can read Java GATE XML format
  • GATE plugin Format_Bdoc provides support for loading/saving formats bdocjs, bdocym and bdocmp in Java GATE
  • Offsets differ between GATE and GateNLP:
    • Java: offsets refer to UTF-16 encoding, possibly a surrogate pair of UTF-16 characters
    • Python: offsets refer to Unicode code points
    • bdocjs/bdocym/bdocmp automatically convert the offsets on either side
    • field offset_type is either p or j

Corpus¶

  • a list-like collection of a fixed number of documents which can be retrieved and stored by index:
    get: doc = corpus[2] set: corpus[3] = doc
  • on retrieval, the index gets stored in a document feature
  • implements store(doc) to save a document to the index stored in the document feature
  • some implementations: append(doc) to add a new document to the corpus
  • some implementations: store/retrieve None
    • on retrieveal: None indicates absence of document
    • on storing: None indicates that document should get removed or should not get updated

ListCorpus¶

  • wrap a Python list-like data structure
  • but provide the store method
In [31]:
from gatenlp.corpora import ListCorpus
texts = ["this is text one", "here is text two", "and this is text three"]
docs = [Document(t) for t in texts]
lcorp = ListCorpus(docs)
doc1 = lcorp[1]
print(doc1.features)
lcorp.store(doc1)
Features({'__idx_139956274540624': 1})

DirFilesCorpus¶

  • all (recursive) files in a directory with some specific extension
  • specify some specific format or infer from file extension
  • stores the relative file path as a document feature
In [32]:
from gatenlp.corpora import DirFilesCorpus
corp1 = DirFilesCorpus("data/dir1")  # get all the matching filenames from the directory
print("Number of documents:", len(corp1))
doc1 = corp1[2]  # actually read the document from the directory
print("Text for idx=2:", doc1.text)
print("Features for idx=2:", doc1.features)
doc1.annset().add(0,len(doc1.text), "Document", dict(what="test document"))
# this writes the document back to the file:
corp1.store(doc1)
# could also have used: corp1[2] = doc1
Number of documents: 4
Text for idx=2: This is another document for testing which mentions John Smith.
Features for idx=2: Features({'__idx_139955830010000': 2})

Corpus Viewer¶

In [33]:
from gatenlp.visualization import CorpusViewer
cviewer = CorpusViewer(corp1)
cviewer.show()

Other Corpus Classes¶

  • NumberedDirFilesCorpus: create a directory tree where the path represents digits of a large number
    • e.g. 000/002/341.bdoc for element number 2341 of 600000000 total
  • EveryNthCorpus: wrap a corpus and access only elements $k*i + o$ for $i = 0..\lfloor(n/k)\rfloor$
    • $k$: every that many elements
    • $o$: start with this element ($o < k$)
    • e.g.: get elements 3, 7, 11, 15 from a corpus with 17 elements
    • useful for processing files in a DirFilesCorpus with multiple processes
  • ShuffledCorpus: random re-ordering of the elements in the wrapped corpus
  • CachedCorpus: store retrieved elements from a (slow) base corpus in a (fast) cache corpus
  • Still work in progress

Source, Destination¶

  • Document Source: something that can be iterated over to get one Document after the other
    • unknown size
    • a Corpus may also function as a Source
  • Document Destination: something that has append(doc) to add Document instances
    • unknown final size
    • also has close() to end writing
    • may implement the with documentdestination as dest: pattern
    • an appendable Corpus may also function as a Destination

Source, Destination examples¶

  • BdocjsLinesFileSource/Destination: one line of bdocjs serialization per document
  • TsvFileSource: one column in a TSV file contains the text, other columns can be stored in features
  • PandasDfSource: similar to TSV source, but for a Pandas data frame
  • Still work in progress: improvements/more to come!

TsvFileSource¶

In [34]:
from gatenlp.corpora import TsvFileSource
tsvsrc1 = TsvFileSource("data/mytsvfile.tsv", text_col="text", feature_cols=dict(src="source",year="year"))
for doc in tsvsrc1:
    print(doc)
Document(This is the text of the first row. It has several sentences.,features=Features({'src': 'source1', 'year': '2005'}),anns=[])
Document(Text of the second row.,features=Features({'src': 'source1', 'year': '2006'}),anns=[])
Document(Another text, this time of the third row. ,features=Features({'src': 'source2', 'year': '2001'}),anns=[])
Document(And here another, from the fourth row.,features=Features({'src': 'source3', 'year': '2013'}),anns=[])

PandasDfSource¶

In [35]:
from gatenlp.corpora import PandasDfSource
try:  # this requires Pandas!
    import pandas as pd, csv
    df = pd.read_csv("data/mytsvfile.tsv", sep="\t", quotechar=None, index_col=None, quoting=csv.QUOTE_NONE)
    pdsrc1 = PandasDfSource(df, text_col="text", data_cols=["source", "year"])
    for doc in pdsrc1:
        print(doc)
except:
    print("Pandas not installed")
Document(This is the text of the first row. It has several sentences.,features=Features({'__data': {'source': 'source1', 'year': 2005}}),anns=[])
Document(Text of the second row.,features=Features({'__data': {'source': 'source1', 'year': 2006}}),anns=[])
Document(Another text, this time of the third row. ,features=Features({'__data': {'source': 'source2', 'year': 2001}}),anns=[])
Document(And here another, from the fourth row.,features=Features({'__data': {'source': 'source3', 'year': 2013}}),anns=[])

Conll-U Source¶

  • Read in one of the many multilingual corpora from https://universaldependencies.org/
  • create documents from k sentences, paragraphs conll documents
  • use original text hints or space hints, if available
  • Example: first few lines of ar-ud-train.conllu
In [36]:
from gatenlp.corpora.conll import ConllUFileSource
src = ConllUFileSource("data/ar-tiny.conllu", group_by="doc", group_by_n=1)
corp = list(src)
print(len(corp))
3

Conll-U Source¶

In [37]:
corp[0].show(doc_style="direction: rtl; font-size: 1.5em; line-height: 1.5;")

Annotators, Executors¶

  • Annotator: a callable that accepts a document to process and either:
    • returns the same or a different document (most common situation)
    • returns None: something went wrong or the document should get filtered
    • returns a list of zero to n documents: filter, error, split documents
    • may be just a function, but usually a subclass of Annotator
    • standard methods for handling over-a-corpus results
  • Pipeline: a special annotator that recursively runs other annotators in sequence
  • Executor: a class that runs an annotator
    • on a corpus
    • on a source and optional destination
    • takes care of handling None, lists of returned documents

Example 1/3¶

In [38]:
from gatenlp.corpora import ListCorpus
from gatenlp.processing.pipeline import Pipeline 
from gatenlp.processing.annotator import AnnotatorFunction
from gatenlp.processing.executor import SerialCorpusExecutor

texts = ["Some text.", "Another text.", "Also some text here.", "And this is also some text."]
docs = [Document(t) for t in texts]
corp = ListCorpus(docs)

def annfunc1(doc):
    doc.annset().add(0,3,"Ann1")
    return doc
def annfunc2(doc):
    doc.annset("set1").add(1,4,"Type1")
    return doc
ann1 = AnnotatorFunction(annfunc1)
ann2 = AnnotatorFunction(annfunc2)
pipeline = Pipeline()
pipeline.add(ann1, name="FirstAnnotator")
pipeline.add(ann2, name="SecondAnnotator")

Example 2/3¶

In [39]:
exe = SerialCorpusExecutor(pipeline, corpus=corp)
exe()
corp[2]
Out[39]:

Example 3/3¶

In [40]:
# use corp as source and create another ListCorpus as destination
corpnew = ListCorpus([])
exe2 = SerialCorpusExecutor(pipeline, source=corp, destination=corpnew)
exe2()
print("Length of corpnew:", len(corpnew))
print(f"in={exe2.n_in}, out={exe2.n_out}, none={exe2.n_none}, ok={exe2.n_ok}, err={exe2.n_err}")
corpnew[2]
Length of corpnew: 4
in=4, out=4, none=0, ok=4, err=0
Out[40]:

Spacy Annotator¶

  • Use a SpaCy pipeline to annotate a document
  • convert spacy tokens, entities etc into Annotations, convert token attributes into annotation features
  • makes it much easier to add own annotations and features, no need to keep vocab files around
  • but possibly not as optimized/fast as Spacy

Preparation:

  • make sure spacy dependency is installed for your gatenlp environment:
    pip install -U spacy (not necessary if gatenlp[all] was used)
  • make sure the model for the language is installed:
    English: python -m spacy download en_core_web_sm
  • To use in notebook, need to restart kernel after installation!

Spacy Annotator¶

In [41]:
import spacy
print("Spacy version:", spacy.__version__)
from gatenlp.lib_spacy import AnnSpacy

nlp = spacy.load("en_core_web_sm")
annotator = AnnSpacy(pipeline=nlp, outsetname="Spacy")
doc2.annset("Spacy").clear()   # avoid annotation duplication when running several times
doc2 = annotator(doc2)
/home/johann/software/anaconda/envs/gatenlp-37/lib/python3.7/site-packages/torch/cuda/__init__.py:80: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:112.)
  return torch._C._cuda_getDeviceCount() > 0
Spacy version: 3.3.1
/home/johann/software/anaconda/envs/gatenlp-37/lib/python3.7/site-packages/spacy/util.py:837: UserWarning: [W095] Model 'en_core_web_sm' (3.2.0) was trained with spaCy v3.2 and may not be 100% compatible with the current version (3.3.1). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)

Spacy Annotator¶

In [42]:
# Adapt size of viewer
from IPython.core.display import display, HTML
display(HTML("<style>#view2-wrapper { font-size: 80% !important; } #view2-row1 {height: 15em; min-height: 5em;}</style>"))


doc2.show(htmlid="view2")