/home/cpanrun/depot/main/contrib-patched/perl/CPAN/src/Array-Suffix/blib/html/site/lib/Array/Suffix.html

DESCRIPTION</PRE> <P> <H2><A NAME=``1. introduction''>1. Introduction</A></H2> <P>The Array::Suffix module is a module that allows for the retrieval of variable length ngrams. An ngram is defined as a sequence of 'n' tokens that occur within a window of at leaste 'n' tokens in the text. What constitutes as a 'token' can be defined by the user.</P> <P> <H2><A NAME=``2. ngrams''>2. Ngrams</A></H2> <P>An ngram is a sequence of n tokens. The tokens in the ngrams are delimited by the diamond symbol, ``<>''. Therefore ``to<>be<>'' is a bigram whose tokens consist of ``to'' and ``be''. Similarly, ``or<>not<>to<>'' is a trigram whose tokens consist of ``or'', ``not'', and ``to''.</P> <P>Given a piece of text, Ngrams are usually formed of contiguous tokens. For example, if we take the phrase:</P> <PRE> to be or not to be</PRE> <P>The bigrams for this phrase would be:</P> <PRE> to<>be<> be<>or<> or<>not<></PRE> <P>The trigrams for this phrase would be:</P> <PRE> to<>be<>or<> be<>or<>not<> or<>not<>to<> not<>to<>be<></PRE> <P> <H2><A NAME=``3. tokens''>3. Tokens</A></H2> <P>We define a token as a contiguous sequence of characters that match one of a set of regular expressions. These regular expressions may be user-provided, or, if not provided, are assumed to be the following two regular expressions:</P> <PRE> \w+ -> this matches a contiguous sequence of alpha-numeric characters</PRE> <PRE> [\.,;:\?!] -> this matches a single punctuation mark</PRE> <P>For example, assume the following is a line of text:</P> <P>``to be or not to be, that is the question!''</P> <P>Then, using the above regular expressions, we get the following tokens:</P> <PRE> to be or not to be , that is the question !</PRE> <P>If we assume that the user provides the following regular expression:</P> <PRE> [a-zA-Z]+ -> this matches a contiguous sequence of alphabetic characters</PRE> <P>Then, we get the following tokens:</P> <PRE> to be or not to be that is the question</PRE> <P> <H2><A NAME=``4. usage''>4. Usage</A></H2> <PRE> use Array::Suffix;</PRE> <P> <H3><A NAME=``array::suffix requirements''>Array::Suffix Requirements</A></H3> <PRE> use Array::Suffix;</PRE> <PRE> # create an instance of Array::Suffix my $sarray = Array::Suffix->new();</PRE> <PRE> # create the files needed and specify which # file you would like to get the ngrams from $sarray->create_files("my_file.txt");</PRE> <PRE> # get the ngrams $sarray->get_ngrams();</PRE> <P> <H3><A NAME=``array::suffix functions''>Array::Suffix Functions</A></H3> <OL> <LI><STRONG><A NAME=``item_create_files''><CODE>create_files(@FILE)</CODE></A></STRONG><BR>

<PRE> Takes an array of files in which the ngrams are to be obtained from. This function will creates the files that are required for the suffix array to be created. These files are defined as the name of the first file entered in the FILE array and timestamped.</PRE> <PRE> 1. vocabulary file : converts tokens to integers prefix: 2. snt file : integer representation of corpus 3. sntngram file : integer representation of the ngrams and their frequency counts 4. ngram file : ngrams and their frequencies</PRE> <LI><STRONG><A NAME=``item_get_ngrams''><CODE>get_ngrams()</CODE></A></STRONG><BR>

<PRE> Obtains ngrams of size two and their frequencies storing them in the given ngram file.</PRE> <LI><STRONG><A NAME=``item_create_stop_list''><CODE>create_stop_list(FILE)</CODE></A></STRONG><BR>

<PRE> Removes n-grams containing at least one (in OR mode) stop word or all stop words (in AND mode). The default is OR mode. Each stop word should be a regular expression in this FILE and should be on a line of its own. These should be valid Perl regular expressions, which means that any occurrence of the forward slash '/' within the regular expression must be 'escaped'.</PRE> <LI><STRONG><A NAME=``item_set_stop_mode''><CODE>set_stop_mode(MODE)</CODE></A></STRONG><BR>

<PRE>

    OR mode removes n-grams containing at least 
    one stop word and AND mode removes n-grams 
    that consists of entirely of stop words. 
    Default:  AND</PRE>
<LI><STRONG><A NAME="item_set_token_file"><CODE>set_token_file(FILE)</CODE></A></STRONG><BR>

<PRE> Each regular expression in this FILE should be on a line of its own, and should be delimited by the forward slash '/'. These should be valid Perl regular expressions, which means that any occurrence of the forward slash '/' within the regular expression must be 'escaped'. </PRE> <PRE>

    NOTE: This function should be called before the 
    function that creates the main files ie before 
    create_files(FILE).</PRE>
<PRE>

set_nontoken_file(FILE)</PRE> <PRE>

    The set_nontoken_file function can be used when there 
    are predictable sequences of characters that you know 
    should not be included as tokens.</PRE>
<PRE>
    NOTE: This function should be called before the 
    function that creates the main files ie before 
    create_files(FILE).</PRE>
<LI><STRONG><A NAME="item_set_remove"><CODE>set_remove()</CODE></A></STRONG><BR>

<PRE> Ignores Ignores n-grams that occur less than N times. Ignored n-grams are not counted and so do not affect counts and frequencies.</PRE> <PRE> NOTE: Should be set before you retrieve the ngrams, ie before you call the get_ngrams() function. </PRE> <PRE>

set_marginals()</PRE> <PRE> The marginal frequencies consist of the frequencies of the individual tokens in their respective positions in the n-gram.</PRE> <PRE> NOTE: Should be set before you retrieve the ngrams, ie before you call the get_ngrams() function.</PRE> <LI><STRONG><A NAME=``item_set_newline''><CODE>set_newline()</CODE></A></STRONG><BR>
<PRE> Prevents n-grams from spanning across the new-line character </PRE> <PRE>

set_frequency(N)</PRE> <PRE> Does not display n-grams that occur less than N times</PRE> <PRE> NOTE: Should be set before you retrieve the ngrams, ie before you call the get_ngrams() function.</PRE> <LI><STRONG><A NAME=``item_set_min_ngram_size''><CODE>set_min_ngram_size(N)</CODE></A></STRONG><BR>
<PRE>

    Finds n-grams greater than or equal to size N.
    Default: 2</PRE>
<PRE>
    NOTE:  Should be set before you retrieve the ngrams, 
    ie before you call the get_ngrams() function.</PRE>
<LI><STRONG><A NAME="item_set_max_ngram_size"><CODE>set_max_ngram_size(N)</CODE></A></STRONG><BR>

<PRE> Finds n-grams less than or equal to size N Default: 2</PRE> <PRE> NOTE: Should be set before you retrieve the ngrams, ie before you call the get_ngrams() function.</PRE> <LI><STRONG><A NAME=``item_set_ngram_size''><CODE>set_ngram_size(N)</CODE></A></STRONG><BR>

<PRE>

    Finds ngrams equal to size N
    Default : 2</PRE>
<PRE>
    NOTE:  Should be set before you retrieve the ngrams, 
    ie before you call the get_ngrams() function.</PRE>
<LI><STRONG><A NAME="item_set_destination_file"><CODE>set_destination_file(FILE)</CODE></A></STRONG><BR>

<PRE>

    Prints the ngrams to FILE.</PRE>
<PRE>
    The hidden files that get erased when program is 
    completed are named: &lt;FILE&gt;.&lt;ext&gt;.</PRE>
<PRE>
    If this is not set the files will be named
    default.&lt;ext&gt;
</PRE>
<PRE>

get_ngram_count()</PRE> <PRE>

    Returns the number of n-grams.</PRE>
<LI><STRONG><A NAME="item_remove_files"><CODE>remove_files()</CODE></A></STRONG><BR>

<PRE> Removes the snt, sntngram and the vocab file. </PRE> <PRE>

AUTHOR</PRE> <P>Bridget Thomson McInnes, <A HREF=``mailto:bthomson@d.umn.edu''>bthomson@d.umn.edu</P> </OL> <P> <HR> <H1><A NAME=``bugs''>BUG/A</H1> <P>Limitations of this package are:</P> <OL> <LI><STRONG><A NAME=``item_Only_a_partial_set_of_marginal_counts_are_found_in''>Only a partial set of marginal counts are found in this pachage. The frequency of the individual tokens in the n-gram are recorded. For example, given the trigram, w1 w2 w3, the marginal counts that would be returned are: the number of times w1 occurs in position one of the ngram, the number of times that w2 occurs in the second position of an ngram, and the number of times that w3 occurs in the third position of the ngram.</A></STRONG><BR>

<LI><STRONG><A NAME=``item_The_size_of_the_corpus_that_this_package_can_retri''>The size of the corpus that this package can retrieve ngrams fromm is limited to approximatly 75 million tokens. Please note that this number may vary dependng on what options are used.</A></STRONG><BR>

</OL> <P> <HR> <H1><A NAME=``see also''>SEE ALSO</A></H1> <P> <HR> <H1><A NAME=``copyright''>COPYRIGHT</A></H1> <P>Copyright (C) 2004-2007, Bridget Thomson McInnes</P> <P>This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.</P> <P>You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.</P> <P>Note: a copy of the GNU Free Documentation License is available on the web at <A HREF=``http://www.gnu.org/copyleft/fdl.html''>http://www.gnu.org/copyleft/fdl.html and is included in this distribution as FDL.txt.</P> <P><CODE>perl(1)</CODE></P> <TABLE BORDER=0 CELLPADDING=0 CELLSPACING=0 WIDTH=100%> <TR><TD CLASS=block VALIGN=MIDDLE WIDTH=100% BGCOLOR=``#cccccc''> <FONT SIZE=+1><STRONG><P CLASS=block> /home/cpanrun/depot/main/contrib-patched/perl/CPAN/src/Array-Suffix/blib/lib/Array/Suffix.pm</P></STRONG></FONT> </TD></TR> </TABLE>

</BODY>

</HTML>

/home/cpanrun/depot/main/contrib-patched/perl/CPAN/src/Array-Suffix/blib/html/site/lib/Array/Suffix.html