NAME

DMOZ::ParseRDF - Parse the gigantic dmoz.org content file into smaller parts.

SYNOPSIS

   use DMOZ::ParseRDF;
   my $dmoz = DMOZ::ParseRDF->new({
       -rdf_gzip_stream => 1,
       -rdf_local_file  => 'content.rdf.u8.gz',
       -rdf_gzip_path   => '/bin/gzip',
       -rdf_part_stack  => {
           'Top/Arts' => '/home/dmoz/arts.part'
       }
   });

DESCRIPTION

DMOZ::ParseRDF is an object-oriented module for parsing DMOZ data into manageable sub-sections. As of January 8th 2004 the DMOZ content file is around 1.3GB in size. The data is free to download and can be used in your custom database but please make sure you read the license agreement at http://dmoz.org/license.html first.

METHODS

$class->new( [HASHREF] )

The the new manpage method creates and returns a blessed object reference. If desired you may pass in a hash reference of options or alternatively you can call the appropriate method to set a value for the corresponding attribute.

The available attributes are listed below:

rdf_gzip_steam: This attribute specifies whether the content file should be streamed through gzip and can either be 1 for yes or 0 for no.
rdf_gzip_path: In order to stream the content file through gzip the path to the gzip binary must be specified. If no path is given an attempt will be made to guess the location if (possible).
rdf_local_file: This attribute sets the location of the content file. It can either be the uncompressed content.rdf.u8 file or the gzipped version.
rdf_part_stack: This attribute must be a hash reference of categories to parse. The hash key is the full category name and the corresponding value is the file to write the parsed data to.

$object->data( STRING )

This method is used to set the location of the DMOZ content file. It is simply the full server path to the content.rdf file, either in gzip or rdf format.

$object->gzip_stream( BOOLEAN )

Call this method to specify whether to stream the file through gzip. The argument must either be 1 for yes or 0 for no.

$object->gzip( STRING )

This method sets the location of the gzip binary. It is not essential to call this method if your gzip binary is in a typical location as gzip is looked for automatically, however calling this method ensures the correct binary is used.

$object->parts( HASHREF )

This method requires a hash reference as an argument. The hash keys should be the category name to parse, such as Top/Arts and the hash key should be the corresponding output file location, for example /root/arts.part.

The hash is sorted before parsing begins to ensure that the parser can parse each category alphabetically (which is how the RDF file is formatted).

$object->parse()

Calling this method triggers the parser. It is called with no arguments.

EXAMPLES

Here are some simple examples of how to use DMOZ::ParseRDF

This example sets attributes by calling methods...

    my $dmoz = DMOZ::ParseRDF->new();
    $dmoz->data('/home/dmoz/content.rdf.u8.gz');
    $dmoz->gzip_stream(1);
    $dmoz->gzip('/bin/gzip');
    $dmoz->parts({
        'Top/Health' => '/home/dmoz/health.part'
    });
    $dmoz->parse() or die $dmoz->error();

This example sets attributes at the time of creating the object...

    my $dmoz = DMOZ::ParseRDF->new({
        -rdf_local_file  => '/home/dmoz/content.rdf.u8.gz',
        -rdf_gzip_path   => '/bin/gzip',
        -rdf_gzip_stream => 1,
        -rdf_part_stack  => {
            'Top/Health' => '/home/dmoz/health.part'
        }
    });
    $dmoz->parse() or die $dmoz->error();

PERFORMANCE

Due to the nature of this module and the tasks it performs, it hogs CPU power. Fortunately the memory usage stays quite low as the data is not stored in memory.

Just be aware that CPU power will briefly hit the roof while parsing the DMOZ content.

SUPPORT

Please email <paul@wilsonprograms.com> for support. You may also visit the Wilson Programming support forum at http://wilsonprograms.com/cgi-bin/bb/gforum.cgi

AUTHOR

DMOZ::ParseRDF was written by Paul Wilson <paul@wilsonprograms.com> in 2004

COPYRIGHT

DMOZ::ParseRDF - Parse the I dmoz.org content file into smaller parts.