|
Lingua::DE::Sentence - Perl extension for tokenizing german texts into their sentences. |
Lingua::DE::Sentence - Perl extension for tokenizing german texts into their sentences.
use Lingua::DE::Sentence;
my $sentences = get_sentences($text);
foreach (@$sentences) {
print $nr++, "\t$_";
}
or
use Lingua::DE::Sentence;
my ($sentences, $positions) = get_sentences($text);
for (my $i=0; $i < scalar(@$sentences); $i++) {
print "\n", $nr++, "\t",
$positions->[$i]->[0], "-", $positions->[$i]->[1],
"\t", $sentences->[$i];
}
The Lingua::DE::Sentence module contains the function get_sentences,
which splits text into its constituent sentences.
The result can be either the list of sentences in the text or
the list of sentences plus and a list of their absolute positions in the text
It's based on a regular expression to find possible endings of sentences and
many little rules to avoid exceptions like acronyms or numbers.
There is a large list of known abbrevations and a not so large list of known file extensions, which ones are used to differences acronyms and filenames from endings of sentences. They can be extented or exchanged if needed.
get_sentences by default.
You can further export the following methods:
get_sentences, get_acronyms, set_acronyms, add_acronyms,
get_file_extensions, set_file_extensions, add_file_extensions.
Basically, I use a ``big'' regular expression to find possible sentence endings. This regular expression find punctations (.?!) or sequences of punctations like ?? or !?, perhaps followed by quotationmarks or brackets like ``'), but never by comma. An empty line is interpreted as sentence end, too. Of course, the end of text also.
Then, found possibilities of sentence endings are checked for exceptions. To do this, I take 2 substrings, the first from the last sentence endings to the momentan position, the second starts at the momentan positions and has a length of 100 chars. So I can test the environment without any slow substitution and without using $`, ... . Before I check, I cut leading spaces, or any other stuff from the beginning of the sentence and throw it away. I use some heuristics:
Another form of abbreviations are known acronyms, I've listed ca. 370 ones. I hope, that's enough for the most cases.
Last I look, wether the word before the dot ends with a lot of consonants. Or the word has only consonants or only vocals as letters. So I'm able, to interprete ``Dtschl.'' in the right way.
Allthough these are many rules, they are implemented to run fast. There are no substitutions, no $`, ... .
Sentences like 'Spieler A schoss den Ball zu Spieler B.' are misinterpreted. B. is always an acronym. Similary are sentences wich ends on small numbers.
Many abbreviations and file extensions still misses, feel free to contact me.
If a sentence starts with the incorrect quotes >>quote<<, the '>>' characters are removed. It's not really a bug, it's a feature. The module intends, that these are quotings from email like
Andrea Holstein wrote: > ... > ... > >
You should use the right form of quoting: <<quote>>.
There are texts with such a form of quoting: ,,quote''. Well, the commata are removed, too.
This module tries to use a german locale setting. It tries to set the locale on a POSIX OS to de_DE. Neither on a non POSIX OS, neither you have installed german language locales, the module won't function.
One of the greatest bugs is surely my bad English. Sorry.
Andrea Holstein <andrea_holsten@yahoo.de>
Lingua::EN::Sentence
Text::Sentence
Copyright (c) 2001 Andrea Holstein. All rights reserved.
This library is free software.
You can redistribute it and/or modify it under the same terms as Perl itself.
|
Lingua::DE::Sentence - Perl extension for tokenizing german texts into their sentences. |