Named entity extraction task aims to extract phrases from plain text that correpond to entities. Polyglot recognizes 3 categories of entities:
I-LOC): cities, countries, regions, continents, neighborhoods, administrative divisions ...I-ORG): sports teams, newspapers, banks, universities, schools, non-profits, companies, ...I-PER): politicians, scientists, artists, atheletes ...The models were trained on datasets extracted automatically from Wikipedia. Polyglot currently supports 40 major languages.
from polyglot.downloader import downloader
print(downloader.supported_languages_table("ner2", 3))
1. Polish 2. Turkish 3. Russian 4. Indonesian 5. Czech 6. Arabic 7. Korean 8. Catalan; Valencian 9. Italian 10. Thai 11. Romanian, Moldavian, ... 12. Tagalog 13. Danish 14. Finnish 15. German 16. Persian 17. Dutch 18. Chinese 19. French 20. Portuguese 21. Slovak 22. Hebrew (modern) 23. Malay 24. Slovene 25. Bulgarian 26. Hindi 27. Japanese 28. Hungarian 29. Croatian 30. Ukrainian 31. Serbian 32. Lithuanian 33. Norwegian 34. Latvian 35. Swedish 36. English 37. Greek, Modern 38. Spanish; Castilian 39. Vietnamese 40. Estonian
%%bash
polyglot download embeddings2.en ner2.en
[polyglot_data] Downloading package embeddings2.en to [polyglot_data] /home/rmyeid/polyglot_data... [polyglot_data] Package embeddings2.en is already up-to-date! [polyglot_data] Downloading package ner2.en to [polyglot_data] /home/rmyeid/polyglot_data... [polyglot_data] Package ner2.en is already up-to-date!
Entities inside a text object or a sentence are represented as chunks. Each chunk identifies the start and the end indices of the word subsequence within the text.
from polyglot.text import Text
blob = """The Israeli Prime Minister Benjamin Netanyahu has warned that Iran poses a "threat to the entire world"."""
text = Text(blob)
# We can also specify language of that text by using
# text = Text(blob, hint_language_code='en')
We can query all entities mentioned in a text.
text.entities
[I-ORG([u'Israeli']), I-PER([u'Benjamin', u'Netanyahu']), I-LOC([u'Iran'])]
Or, we can query entites per sentence
for sent in text.sentences:
print(sent, "\n")
for entity in sent.entities:
print(entity.tag, entity)
The Israeli Prime Minister Benjamin Netanyahu has warned that Iran poses a "threat to the entire world". I-ORG [u'Israeli'] I-PER [u'Benjamin', u'Netanyahu'] I-LOC [u'Iran']
By doing more careful inspection of the second entity Benjamin Netanyahu, we can locate the position of the entity within the sentence.
benjamin = sent.entities[1]
sent.words[benjamin.start: benjamin.end]
WordList([u'Benjamin', u'Netanyahu'])
!polyglot --lang en tokenize --input testdata/cricket.txt | polyglot --lang en ner | tail -n 20
, O which O was O equalled O five O days O ago O by O South I-LOC Africa I-LOC in O their O victory O over O West I-ORG Indies I-ORG in O Sydney I-LOC . O
This work is a direct implementation of the research being described in the Polyglot-NER: Multilingual Named Entity Recognition paper. The author of this library strongly encourage you to cite the following paper if you are using this software.
@article{polyglotner,
author = {Al-Rfou, Rami and Kulkarni, Vivek and Perozzi, Bryan and Skiena, Steven},
title = {{Polyglot-NER}: Massive Multilingual Named Entity Recognition},
journal = {{Proceedings of the 2015 {SIAM} International Conference on Data Mining, Vancouver, British Columbia, Canada, April 30 - May 2, 2015}},
month = {April},
year = {2015},
publisher = {SIAM}
}