After a lengthy absence (the end of school holidays, but my thesis kept me away from this blog), I will slowly resume writing articles for it.

Until now, this blog has mainly been devoted to Perl , but the approaching school year makes me want to expand the topics covered in the courses I teach, it will remain largely computer-related issues (and more generally computerized documentation), but with a more focused information sciences.

For the resumption of this blog, I'll stay in the habits of the past, namely a little note on Perl . Today, I speak of a module which is useful when you want to perform processing related to the web. It is LWP:: Online . This module allows a very simple answer to the question "Am I being connected to the web? .

Usage is as simple as the implementation of the module as it is an import function online() , which returns a boolean value.

A simple script would look to this:

 #! / Usr / bin / env perl

use strict;
use warnings;

use LWP:: Online qw (online) # we import function online () in our script

if (online ()) (
    print "We're connected \ n";
)
else (
    print "We are not connected \ n";
)

The module also provides a function offline() which returns true when we are not connected to the Web.

I was confronted with an interesting problem this morning. The problem was the following:

In terms of solutions, there were not too many choices:

1. do the job manually;
2. develop a more automated.

My heart obviously looking for the automated solution. I developed a little tool to retrieve the son of an RSS site. This tool was very simple because based on the function find_feeds() module XML:: Feed . Here is version one-liner:

 perl-MXML:: Feed-MDAT:: Dump-e 'dd (XML:: Feed-> find_feeds (shift))' http://lesoir.be

I changed my script to manage a list of links, and provide in return a list of RSS son. I also added a small management statistics to know the number of sites treated and which ones do not offer RSS son (or more precisely, sites with XML:: Feed failed to retrieve the son RSS). Here's the final script:

 #! / Usr / bin / env perl

use strict;
use warnings;

use YAML;
use Getopt:: Long;
use XML:: Feed;

my $ config = ();

GetOptions ($ config, "input = s", "output = s");

_usage die () unless _Validation ($ config);

my $ sites = YAML:: LoadFile ($ config -> (input));
my @ feeds;
my @ with_feeds;
my @ no_feeds;

foreach my $ site (@ ($ site)) (
    my @ site_feeds = XML:: Feed-> find_feeds ($ site);
    if (scalar (@ site_feeds)> 0) (
        push @ feeds, @ site_feeds;
        push @ with_feeds, $ site;
    )
    else (
        push @ no_feeds, $ site;
    )
)

YAML:: DumpFile ($ config -> (output), \

For several years now, I'm régulièremnt contacted by mail to communicate with the tools to migrate databases Winisis . Unfortunately, I have never taken the time to write a proper documentation, which is unfortunate. I will lay the foundations of this material on this blog, and I hope that I will take time in the future to develop a more complete documentation.

Before turning to the tools itself, here is the context in which they were developed. As part of a training course organized jointly by the Free University of Brussels (ULB) and the University Commission for Development (CUD) . I taught a course called "Integrated Library Management. The trainees in this training should conduct a project, and one of those students wanted to migrate Winisis Koha (software was used to illustrate my way). So I looked into the issue. The objective was to migrate the database of ISIS MARC21 .

Databases based on a scheme specific to each user (Winisis is not an ILS, but a DBMS, specializing in bibliographic data, certainly, but a DBMS, so it looks like more than to Microsoft Access Koha). The first step is to determine the structure of the database in order to establish a correspondence table to the desired MARC (MARC21, UNIMARC, or any other variant). Today, I see only this phase, and the tool developed for this. Other steps will follow in the coming days.

The structure of an ISIS database, called "Field Definition Table", then "table field definition" in the jargon ISIS is stored in a text file with the extension " .fdt . In analyzing this file, one can easily determine the structure of the database. Biblio:: Isis is the module of choice for manipulating ISIS databases, it provides a function to read the table definition of fields, and So the following code is based on this module:

 #! / Usr / bin / env perl

use strict;
use warnings;

use Getopt:: Long;
use File:: Find:: Rule;
use Data:: Dumper;
use Encode qw (from_to);
use Spreadsheet:: WriteExcel;

my $ config = ();

my% save_functions = (
    Excel => \ & _save_excel,
    dump => \ & _dump_struct,
)

my $ fdt_struct = ();

GetOptions ($ config, 'database = s', 'file = s', 'save = s');

if (not exists $ config -> (

When migrating to an information system to another, one is often tempted to engage in an operation to improve quality. Although this is a project in itself, this may be the "right time", provided that we have the resources to do so. Human Resources, of course, but we must also have enough time to be able to develop a methodology able to guarantee us a better quality.

This year some students have embarked on this adventure for their graduation work with the migration of library catalogs. And a special problem was quickly placed with the presence of duplicates in the lists of authority. Catalogs are excellent venues to measure the creativity of the human mind can not follow the rules ;-)

Thus, in a library catalog, we have a list of publishers for example, and depending on the creativity of cataloguers, but also the amount of people involved in the management of these publishers, we will find more or fewer variants :

• Ed O'Reilly
• O'Reilly;
• O'Reilly
• E. O'Reilly
• etc..

The ideal would therefore be able to find any duplicates, and replace them with the correct form. But how to find all these duplicates? Without computers, the task is tedious:

1. browse the list of authority;
2. establish a list of duplicates;
3. make the choice of either "correct";
4. make all necessary changes.

Faced with such situations, my first instinct is often to determine what I can computerize, and even automate. In this case, it would be ideal to get a list of "possible duplicates", ie, expressions that close enough for us to put a flea in his ear. In this case, the computer offers several techniques:

• an algorithm to compute the Levenshtein distance , that is to say, the number of elementary operations to move a word to a word P M, based on this algorithm, we will be able to compare each entry in the list authority with the rest of that, and keep the items which the Levenshtein distance is not important (this threshold is obviously set as the mesh size of nets), of course, this algorithm is available on CPAN: Text :: Levenshtein for a version in Perl, and Text:: LevenshteinXS for a version in C;
• other techniques derived from the previous exist, for example using the algorithm hiding behind agrep , a grep to make approximations in the investigations to strings, there is a Perl module that reproduces this behavior: String:: Approx (this is also an XS module, thus based on C).

Thus, techniques exist, "there is more than" ... do a search on CPAN, for example with the keyword "group" or "Similarity," which allowed me must see String:: Similarity:: Group based on String:: Similarity , which is based on an algorithm which significantly different but the same techniques as explained above. In short, once again CPAN save me time and allows me to develop a prototype quickly (without this module, I set up the group creation, which is certainly not complicated, but brings a lot of reflections).

Here is the prototype in question:

 #! / Usr / bin / env perl

use strict;
use warnings

It's been a while since I had not gone into a bookstore, but this passage has been been successful since it allowed me to find a few books to complete these Easter holidays with this special issue of Linux Magazine devoted to Perl: Perl Special Edition Powerhouse .

Plack is a port of WSGI (Python) and Rack (Ruby). Its objective is to provide a common environment developers web framework. It provides connectors for many web servers, but also an environment to easily write middleware. Plack is still young but is already adopted by all frameworks Perl (Catalyst, Dancer, Mojo, ...), and many middleware are also available on CPAN.

via journeesperl.fr

frankcuny Will Talk about Plack in French Perl Workshop 2010.

Some time ago now, when I began to read every day on my ebook, I quickly wanted to be able to read articles from the web on my reading lamp. After a quick tour, I had found nothing at the time (when it goes back even a little while now) and I had therefore embarked on developing a personal solution.

After some reflection, I developed the following tools:

• HTML:: Excerpt:: FromXPath; this tool to extract part of a web page based on an expression XPath in that time I had not heard of the wonderful tool developed by Arc90, j ' I called Readability ¹
• Mobigen:: Command: this tool was just a wrapper program mobigen_linux that converts a file in HTML file Mobibook it is worth noting that mobigen_linux has been replaced by kindlegen (available for free even if not free);
• HTML:: Image:: Save, this tool can save images to an HTML file, and modify it in order to match the image links to local versions;
• App:: uri2mobi; tool is exploiting these three tools in a previous line interface command.

As (bad) laziness, I never took the time to put these modules on CPAN ^2, but this does not stop me to talk about anyway.

So, in the case before us today, we'll see HTML:: Image:: Save. Using it is as simple as possible:

With a lag, I just read the note from Jean Véronis entitled "Ontologies: Perl is a planet in the Solar System" (an English version of the article is available).

And therefore, set to hand the intrinsic value of the ticket, long live Perl ;-)

In reading the latest "PragPub: The First Iteration" (No. 9, available on the website PragProg) in the article "JavaScript: It's Not Just for Browsers Any More" by Jason Huggins, I came across a reflection interesting

When we choose a technology to write an application, we do not just choose the language, we also choose the list of available libraries. If a language has many useful libraries with a vibrant community around them, it's going to be easier to write your application in less time.

Well, in my case, it made me think to Perl and CPAN, but I'm sure others will read other things:)

After some discussions recent, I plunge into the arcana of RDF. Indeed, Pearltrees offering a possibility to export favorites RDF, my natural curiosity prompted me to investigate in order to know what I could actually do with this file. In addition, @ SebDeclercq was nice enough to send me that his own safety, thus sparing me the tedious task of enriching my own Pearltrees.

So I have a file full of links, and now I want to exploit this information. How? As indicated in the ticket's blog Nicolas Cynober, we can use a tool like SPARQL to manipulate information. In SPARQL, you can submit your application written in SPARQL, and get a response in different format:

• XML, with the opportunity to learn a script XSLT to transform the document (eg XHTML);
• in JSON;
• or just text.

So it is a very interesting tool, but it must be online, and personally, I like having my tools directly available, regardless of my web access. So, the question arises of what I have on my machine directly. A trip through the CPAN, and hop, here are some promising modules:

• RDF:: Redland;
• RDF:: Trine;
• and that I need to query the file via the SPARQL: RDF:: Query.

Recently, at the turn of GitHub, I saw an interesting project: growlme. For those not familiar with Growl, it is a notification system non-intrusive. Imagine that you start burning a DVD, and during that time you enjoy to work on an article or blog post. Traditionally, the burning software you display a small window at the end of the grave, thus interrupting what you were doing. With reporting systems as Growl, a window will appear flying in the top right of the screen indicating the end of burning. Growl is only available on Mac ^1, but there are similar systems on Linux (libnotify among others).

growlme can launch a command line and be kept informed of the outcome of its execution via Growl. As I often start a lengthy process via the command line and harness myself to other tasks in the meantime, I'd like to have the equivalent on my Linux.

The result is not very long and operates Desktop::Notify is to say, the Perl interface to libnotify, IPC::Run to run the command, Getopt::Long to process parameters on the command line (note use of Getopt::Long::Configure('pass_through') to keep @ARGV and Sys::Hostname for the title of the notification. For the rest, I aped the original program.

# '/ usr / bin / env perl

use strict;
use warnings;

use Desktop:: Notify;
use IPC:: Run qw (run);
use Getopt:: Long;
use Sys:: Hostname;

my $ config = (
    message => 'Succeed!'
    fail => 'FAILED',
    title => hostname (),
);

Getopt:: Long:: Configure ( 'pass_through');
GetOptions ($ config, 'message = s', 'fail = s', 'title = s');

if (scalar (@ ARGV) == 0) (
    die "$ 0: Must provide a command to execute \ n";
)
else (
    my $ notify = Deskto

In the context of a during a course of management of digital knowledge, I speak a little information retrieval. Just the basics, but hey, it seems important to know how a search engine, especially for future specialists in information and documentation. Among these bases, we how a search engine that receives text to index. If the theory is easy to understand, this does not prevent me from adding a visual layer, and for this reason that I wrote a little to illustrate this stage of indexing.

The script was developed over a period of noon, just before giving the course, and so I did not work ergonomics and layout tool. To achieve rapid tool, I used the following tools:

• HTTP:: Server:: Simple + Template:: Declare to create a small web site from which users can copy and paste text;
• Text:: ExtractWords to extract the words.

Otherwise, the code is quite simple:

# '/ usr / bin / env perl

use strict;
use warnings;

package MyView:: Templates;
use Template:: Declare:: Tags;
use base 'Template:: Declare';

private template form => sub (
    my $ self = shift;
    my $ title = shift;

    div (
        attr (style => 'margin: auto; size: 15%',);
        form (
            attr (
                action => '/ submit',
                method => 'POST',
            );
            textarea (
                attr (
                    cols => '100 ',
                    rows => '25 ',
                    name => 'title'
                    style => 'float: left;'
                );
                $ title;
            );
            input (
                attr (
                    name => 'submit',
                    type => 'submit',
                    value => 'Submit'
                    style => 'float: left;'
                );
            );
            input (
                attr (
                    name => 'reset'
                    type => 'reset'
                    value =>

Recently on the mailing list perl4lib, a question was raised about the existence of a solution to divide a too large MARC file into several smaller files.

Like many users of MARC records, and programmers manipulating MARC records, I faced this problem and I developed a small quick solution:

# '/ usr / bin / env perl

use strict;
use warnings;

use MARC:: File:: USMARC;
use MARC:: Record;

use Getopt:: Long;

my $ config = (output => 'input');

GetOptions ($ config, 'input = s', 'chunk = s', 'output = s', 'max = s');

if (not exists $ config-> (input) and not exists $ config-> (chunk)) (
    die "Usage: $ 0 - input file - chunk-size [- output file] \ n";
Else ()
    run ($ config-> (input), $ config-> (output), $ config-> (chunk), $ config-> (max));
    
)

sub run (
    my ($ input, $ output, $ chunk, $ max) = @ _;

    my $ marcfile = MARC:: File:: USMARC-> in ($ input);
    
    my $ fh = $ output eq 'input'? create_file ($ input): create_file ($ output);
    my $ cpt = 1;
        my $ total = 0;
    while (my $ record = $ marcfile-> next) (
        $ count + +;
        
        if (defined $ max) (
            last if $ count> $ max;
        )
        if ($ cpt + +> $ chunk) (
            close $ fh;
            $ fh = $ output eq 'input'? create_file ($ input): create_file ($ output);
            $ cpt = 1;
        ) 

        print $ fh $ record-> as_usmarc;
    )   
    close $ fh;
)

create_file sub (
    my ($ output) = @ _;
    my $ cpt = 0;
    
    my $ filename = sprintf ( '% s.% 03d', $ output, $ cpt + +);
    while (-e $ filename) (
        $ filename = sprintf ( '% s.% 03d', $ output, $ cpt + +);
    )

    open my $ fh, '>', $ filename;
    return $ fh;
)

This tool is an example of a solution librarian - librarian should be able to program (if it wishes to schedule, of course). The algorithm used is far from complicated (although this is a good exercise), and it is an exploitation of existing modules (CPAN live!) For which there is documentation. In short, a fine example of laziness and spécicialisation.

Nevertheless, a better solution than what marc _split. Pl is undoubtedly the use of utility-marcdump c yaz

We wish you all the best for the new year.

Our best wishes for the new year.

Eh! Maelys, Caroline & Yann

Baby pictures are available on my Flickr photostream as usual.

The baby pictures are available on my Flickr as usual.

For several years now, an initiative of the Perl community, that can be called a tradition now organized in this period. It is an advent calendar. From the first day of December until Christmas (Advent so), an article is published to explain now a module, sometimes a technique, or an element of a culture monger ^1.

To my knowledge, there are four Advent calendars in the Perl community this year. Each focuses on a particular project in the community:

• Perl: the timing of language itself: http://www.perladvent.org
• Perl 6: the timing of future Perl 6: http://perl6advent.wordpress.com/
• Catalyst: Calendar of web frameworks written in Perl: http://www.catalystframework.org/calendar
• Plack, a rackup written in Perl: http://advent.plackperl.org/

I did a brief search to see if communities of other programming languages had taken the initiative, but I found nothing really followed:

• Ruby was one in 2006 and 2008, but obviously nothing this year;
• PHP has one in 2007, and this year;
• I have found nothing for Python;
• and I stopped there! Feel free to submit a comment for me to discover other such schedules.

Personally, I think it's an interesting initiative, first it helps to learn a language or a particular software package, and then it should also provide a sort of promotion for the purpose of the calendar .

In short, in my case, much more interesting than the timing of a famous brand of tires, for example ;-)

It's been a while since I have published notes on this blog, but this does not mean that Perl has not been helpful in the meantime. In fact, it is even quite the opposite! For now, I find it especially useful for me to make prototypes to better understand certain technologies.

Thus in GNU / Linux Magazine France the month of November 2009, there is an article (Put a sphinx in your search engine!) On a tool which I had thrown a quick glance: Sphinx. I have not had time to finish the article (I'm late on everything, I tell you!), But the principle is to provide an entry point to a database via a query. The results of this query is then indexed by an external software (Sphinx in this case). Then when we want to do a full text search on this database, we first examine the Sphinx to get the identifiers of data, we can then retrieve the data directly into the database.

I do not have databases large enough to hand that to see if the promises of performance are at the rendezvous, but according to the author of the article is true!

I therefore question the possibility to use Sphinx to create OPACs "new generation". The advantage is that simply adding Sphinx in the game can use a query language much richer than conventional OPACs.

I have used this technique for indexing by an external program to make available a database Winisis via the protocol Z39.50. I used Perl for this, with the following modules:

• KinoSearch: take over for the indexing;

Perl is really a tool that I like in my toolbox, and like many other programmers Perl, which I particularly appreciate is being able to count on the work of many others to help me solve my problems daily. Sometimes the services rendered are not of the order of support to complete a job, but simply to make an even more comfortable.

Often relying on the CPAN, I came to myself a copy on my hard drive that I keep synchronized as evenly as possible. This work is directed by the excellent minicpan (CPAN:: Mini). So here I am in the pleasant situation of always having to wear hand an archive from CPAN, which I am connected to the Internet or not. But the CPAN is not limited to this, it is also the tool that I consult to read the documentation (yes, I know perldoc is there for that), but what to do when I'm on the road? It's simple, use minicpan _webserver (CPAN:: Mini:: Webserver). The latter is based on the configuration file minicpan, and provides all via a web server. I can now do research, read the documentation, testing, and I can even start the installation from this interface. In short, a CPAN survitaminé! A try!

Most tools that I developed based on LWP, so I need a web access to run them (nice platitude!). Among these tools, some are expected to start via cron, and thus raises the question of what happens to the program when it starts and I'm not connected. One way to address the problem is to modify the program so as not to start treatment when web access is operational. Here LWP:: Online just to help me. It allows me to import a function online (), which will check if web access is present. If so, it returns a positive value. LWP:: Online checks web access by checking the presence of the copyrights on some sites like Google and Yahoo!, So this adds some latency in your program, but it remains for some interesting tools.

Recently, I found myself faced with a problem rather silly but nonetheless annoying. I use for some years now the services of a site. I had to connect to this site recently, but

1. I could not remember my password!
2. resetting the password for difficiel was no way to remember the email address used to create the account!

Damned! How?

Well, welcome to be lazy! Indeed, to enjoy the services of this site, I had written a small robot in Perl with LWP. This tool works good as always relied on a cookie. This cookie was in the proprietary format of HTTP:: Cookies, so I had to convert the format of Mozilla Firefox.

No sooner said than done. A little research on the web, and I found an article of Mongueurs explaining a conversion of cookies, but Mozilla to LWP. After some experimentation, I eventually adapted the program. Here:

# '/ usr / bin / env perl

use strict;
use warnings;

use HTTP:: Cookies;
use HTTP:: Cookies:: Mozilla;
use Getopt:: Long;

my $ config = ();

GetOptions ($ config, 'from = s', 'to = s');

_usage die () unless _valid_config ($ config);

my $ input_jar = HTTP:: Cookies-> new (file => $ config-> (from));
bless $ input_jar, 'HTTP:: Cookies:: Mozilla';

$ input_jar-> save ($ config-> (to));

_usage sub (
    return "Usage: $ 0 - from cookies.lwp - to cookies.mozilla \ n";
)

_valid_config sub (
    my $ config = shift;

    if (exists $ config-> (from) and exists $ config-> (to)) (
        return 1;   
    Else ()
        return 0;
    )
)

He then just call this program as follows: $./cookies_converter -f my_lwp_cookie.txt -t /path/to/mozilla/profile/cookies.sqlite

Then I could go on my account page to change my password and put an email address

At the turn of a tweet, I discovered a little tool to complement emacs. This project perl-completion.el. This allows a smart auto-completion, that is to say that does not limit the content of open buffers. You can obtain help for the names of the modules, but also methods exported by these modules. In short, a must!

If you want to see more, there is a screencast that you will appreciate the work done. There is also a git repository online.

Enjoy!

brian d foy. 15 August 2008