doc.set_ents uses tokens but Span uses char

[yw2903]

I'm trying to remove trailing special character at the end of entities. However, doc.set_ents uses token but Span uses character count. What's a good way to resolve this conflict?

from spacy.language import Language
from spacy.tokens import Span
import re
nlp = spacy.load("en_core_web_lg")
@Language.component("remove_special")

def remove_spcials(doc):
    for ent in doc.ents:
        text = ent.text
        print(ent.text, ent.start_char, ent.end_char)
        tmp = re.search("[#|.|,|%|@|\s]$", text)
        if tmp != None:
            doc.set_ents([Span(doc, ent.start_char, ent.end_char - 1, ent.label)])
        else:
            doc.set_ents([Span(doc, ent.start_char, ent.end_char, ent.label)])

nlp.add_pipe("remove_special", name="remove_special", last=True)
print(nlp.pipe_names)  
doc = nlp("Today is Feb 8th 2021.")

[svlandeg]

Hi! This is a good question for the discussion forum - so I'll move it there. You might get a message that this thread was closed/locked, but we can still continue the conversation on the open thread there, and it should automatically forward you.

[svlandeg]

You can call ent.start and ent.end instead of ent.start_char and ent.end_char to obtain the token indices of the entity instead of the char indices.

Also, please note that your custom component "remove_specials" should return the doc at the end of its processing.

Finally, I'm not sure this function will work as you intend it when you apply it on a doc with multiple entities in doc.ents, because doc.set_ents always overwrites the entire set of entities. Instead, you probably want to build up the list of new entities and call set_ents once on the doc right before returning it.

[yw2903]

I see. Thanks for the advice. However, using ent.start, ent.end, I'm unable to remove 1 or more trailing or middle special characters unless it's one token itself. Pleas see example below:

[svlandeg]

I'm not sure I understand what exactly you're trying to achieve here? The screenshot isn't very helpful to me. Can you illustrate your use-case with a code snippet, the output you're getting, and clarify what you were hoping to achieve?

Either way, named entities as predicted by the NER in spaCy, and stored in doc.ents, always need to align to token boundaries. If you feel like you need to define entities that do not align with token boundaries, perhaps you'd need to adjust the tokenizer instead?

[Cbhihe]

EDIT: (Thanks to @adrianeboyd for pointing out an error in my original response)

I am certain that you are long past this question, but your pattern in tmp = re.search("[#|.|,|%|@|\s]$", text) could be improved at a small cost.
[123] means to represent one of "1,2 or 3", so you don't need the boolean | to signify "or" inside the brackets... That syntax (with the boolean symbol |) is generally reserved for regex sub-expression groups, such as: (red|blue)? which in this ultra simple case would match zero or one instance of EITHER "red" OR "blue".
Finally it's a good habit to used Python’s raw string notation for regular expression patterns, by having rprecedes the actual regex pattern, as in tmp = re.search(r"[#.,%@\s]$", text).
HTH however belatedly.

[adrianeboyd]

This is not quite correct: the . does not need to be escaped inside []. See: https://docs.python.org/3/library/re.html#regular-expression-syntax