Date handling in Perl

Problem

You want to convert epoch into readable date and time.

Maybe you have a log file, which is pumping out lines prefixed with the epoch?



Solution

We use the perl function localtime to convert epoch into a date and time.

At the example tab is the code to convert that epoch, into a recognizable date and time.

localtime, can also return an array – take a look at the reference tag for more info.



Example


We quite simply supply the epoch to localtime and print it.


$ perl -e 'print localtime(1145980815)."n";'
Wed Apr 26 00:00:15 2006



Reference

[tags]Perl Date handling, Perl Coding School[/tags]



Column handling in Perl

Problem

During my first encounters with Perl many years ago, I asked how can I get a specific column.

It was so easy in AWK! 😉



Solution

Well here it is in Perl – see example tab.

BTW you can always write an awkscript and run it through a2p – very good for learning Perl! 🙂



Example


Show column 1:

perl -ane 'print $F[0]."n";'

Show column 2:

perl -ane 'print $F[1]."n";'

Show last column:

perl -ane 'print $F[$#F]."n";'

Show last but one column:

perl -ane 'print $F[($#F-1)]."n";'

So you just run your program, or cat your file, etc and pipe it through this code to get specific column.



Reference

[tags]Perl column handling, Awk to Perl, a2p, Perl Coding School[/tags]



Listen on port – client server demo

Problem

You want to listen on a port, maybe to test firewalls! 🙂

Excellent introduction to client server software.



Solution

Another use bit of socket programming, courtesy of Perl! 🙂

Extremely useful bit of Perl, which can be used with the other post on this site, to confirm network connectivity (port scanner).

This code (see example tab) will listen on a port (although be careful it is over 1024, unless you are running as root/admin).



Example



perl -MIO::Socket -e '$srv=IO::Socket::INET->
new(LocalPort=>$ARGV[0],Type=>SOCK_STREAM,Reuse=>1,Listen=>5)

or die "Failed trying to listen on $ARGV[0]n";while($cl=$srv->accept()) { while() { print }

} close($socket);' port

Can also use it as a cheap chat service! 🙂 Everything gets echo'd through – so just telnet host port and type away! 🙂

Also see my port scan code at the reference tab.



Reference

[tags]Perl Socket Programming, Network Programming, Perl Port Listen, Perl Client Server demo, Perl Coding School[/tags]



scan network port with perl

Problem

You want to test a network port, on a remote system – over TCP/IP.

Maybe this is a new setup, or you want confirmation it is working.

Perhaps the firewall rules have just been changed! 🙂



Solution

A nice small bit of Perl code that I’ve used thousands of times!

In fact I’m running it in most of my production environments,
as a check that the a process is not only running – but also responding.

Yep, you could just use telnet – but some systems have that taken off for security.

Additionally it is n’t as easy to program telnet – I know, I know – you can with expect. 🙂



Example


Here is just a one liner, but you can easily incorporate this into a script.

perl -MIO::Socket -e '$socket=IO::Socket::INET->
new(Proto=>tcp,PeerAddr=>$ARGV[0],PeerPort=>$ARGV[1]);
if([email protected]) { print "Failed: [email protected]" } else { print "Succeedn"; }' host port



Reference

[tags]Perl, Network, TCPIP, Port Scanner, Perl Coding School[/tags]



Obtain epoch time and calculate date yesterday

Problem

You want to capture the current epoch. Maybe to use in a log file, or as a filename.

Or maybe you want to calculate the date yesterday.



Solution

This piece of code is very useful for performing date calculations. You can obtain the current epoch (time in seconds since Jan 1 1970), then add 3600 for 1 hour – or 86400 for 24 hours hence.



Example


So use in a UNIX variable like this:

epoch=perl -M'English' -e 'print $BASETIME."n";'

To work out 24 hours ago, just subtract 86400.

perl -M'English' -e 'print(($BASETIME-86400)."n");'

Then to see the date yesterday:


$ perl -M'English' -e 'print(($BASETIME-86400)."n");'
1180746252
$ perl -M'English' -e 'print(localtime(1180746252)."n");'
Sat Jun 2 09:04:12 2007



Reference

[tags]Perl, epoch, date manipulation, Perl Coding School[/tags]



Calculate largest field big data file

Problem

I wanted to upload a delimited field to mysql db, but hit the problem that the file contained nearly 3000 rows and no schema on the required size of each field.

Therefore I needed to traverse the file and calculate the length of each field. Then at the end, print the largest field found for each column.



Solution

Perl to the rescue! Pretty easy in Perl, after scratching my head attempting with awk. 🙂

Anyway the code is under the example tab. I’m using the pipe symbol as a delimiter “|” – so just substitute this with your delimiter, cat your file and pipe it through this script.

In the example I show how you can manipulate the file, to produce pipe delimited fields too.



Example


This is how to run the script. Basically you just need to pipe your output through find largest.

cat yourfile | ./find_largest.pl

Here is the code.

#!/usr/bin/perl

@highest=();

while( < STDIN > ) {

@thisline=split(/|/);

for($i=0;$i<=$#thisline;$i++) {

$thislength=length($thisline[$i]);

if($thislength > $highest[$i]) { $highest[$i]=$thislength; }

}

}

print(join("|",@highest)."n");

exit(0);

__END__

So for example, if I want to find the largest fields in one of my web logs - for crunching into a db:

  • First off I only want lines starting with a space, then a number.
  • Next I need to replace all multiple spaces between fields, with 1 space.
  • Then I replace the spaces between each field, with a pipe.
  • Lastly I pump it through find_largest.pl - which gives me the largest sized field


[[email protected]]/var/log/httpd% grep "^ [0-9]" access_log.tools
| sed -e 's/ / /g' -e 's/ /|/g' |
~/Perl_Bin/find_largest.pl
|14|1|1|21|6|5|103|9|3|5



Reference

[tags]Perl, Data Analysis, Perl Coding School[/tags]



Perl TimeOut

Problem

You have a script which runs too long and you want to time it out, after a given number
of seconds.



Solution

Useful bit of code to time-out a section of your Perl script, via the alarm function.

See the example tab.



Example



#!/usr/bin/perl

eval {

   local %SIG;
   $SIG{ALRM}=
     sub{ die "timeout reached, after 20 seconds!n"; };
   alarm 20;
   print "sleeping for 60 secondsn";
   sleep 60; # This is where to put your code, between the alarms
   alarm 0;
};

alarm 0;

if([email protected]) { print "Error: [email protected]"; }

exit(0);

__END__

View screen shot demo of perl timeout

view screen shot of perl timeout demo



Reference

[tags]Perl timeout, Perl Coding School[/tags]



Perl WIN32 OLE – Outlook save text

Problem

How to use Perl to connect to MS Outlook. Then descend through given folders and save items to disk, as text files.



Solution

I wrote this some time ago, to traverse predefined outlook mail folders, saving items with given subject to text.

Requires WIN32:OLE perl module (which comes with activeperl by default).

Hardcoded is the upload and uploaded mail folders. Also a subject that contains – pattern: upload.

I used one outlook rule to move items from a specific source, with this subject into upload.



Example


Here is the code – if you have any dramas with it, leave a comment.


#!perl

use Win32::OLE qw(in with);
use Win32::OLE::Const 'Microsoft Outlook';

# Connect to a running version of Outlook
eval { $Outlook =
   Win32::OLE->GetActiveObject('Outlook.Application')
};

die "Outlook not installed" if [email protected];

# If that fails start up Outlook
unless(defined $Outlook) {
   $Outlook =
     Win32::OLE->new('Outlook.Application', 'Quit')
   or die "Opps, cannot start Outlook";
}

# This appears to return a ref to the object
$namespace = $Outlook->GetNamespace('MAPI');
$thisFolder=$namespace->Folders("Mailbox - mailboxna")->
   Folders('upload');
$toFolder=$namespace->Folders("Mailbox - mailboxna")->
   Folders('uploaded');

# Workaround to be able to extract key/value pairs
%thisHash=%{$thisFolder};
$name=$thisHash{'Name'};

# This is the number of items in designated folder
$count=$thisHash{'Items'}{'Count'};
open(LOGFH,">> ol_save_to_text.log")
   or die("cannot open log filen");

# Drop out if there are no mail items in this folder
if($count > 0) {

   print LOGFH "Count: $count for $namen";

   $filename='yourname';
   open(FH,"> $filename")
   or die ("cannot open $filenamen");

   for($i=1;$i<=$count;$i++) {
     print LOGFH "Count: $countn";

     $oItems=$thisFolder->Items(1);

     %thisItem=%{$oItems};
     $subject=$thisItem{'Subject'};

     if($subject =~ /pattern: upload/) {
       print LOGFH "$i: $subjectn";
       $body=$thisItem{'Body'};
       print FH "$body";
       $oItems->Move($toFolder);
     } else {
       $nonitem+=1;
     }

     $oItems->Move($toFolder);
   }

} else { print LOGFH "No Files to Processn"; }

close(LOGFH);
1;



Reference

Microsoft Outlook Keyboard Shortcuts – Courtesy of RNIB

[tags]MS Outlook OLE Perl, MS Outlook, OLE, Perl win32, cygwin, Perl, Perl Coding School[/tags]



Perl libcurl demo

Problem

You want to use libcurl, driven through Perl.



Solution

Beautiful for parse HTML and either extracted (screen scraping) content or performing actions based on results.

See the examples tab for this simple script, demonstrating the libcurl API for Perl.



Example



#!/usr/bin/perl

$url="http://perl.coding-school.com/"; # set your url here
$|++;

use Curl::easy;
# Init the curl session

my $curl= Curl::easy::init() or die "curl init failed!n err: $!n";

sub body_callback {
   my ($chunk,$context)[email protected]_;
   push @{$context}, $chunk;
   return length($chunk);
}

Curl::easy::setopt
   ($curl, CURLOPT_PROXY, $proxy) if($proxy);
Curl::easy::setopt
   ($curl, CURLOPT_PROXYPORT, $proxyport) if($proxyport);
Curl::easy::setopt
   ($curl, CURLOPT_SSL_VERIFYHOST, 0);
Curl::easy::setopt
   ($curl, CURLOPT_SSL_VERIFYPEER, 0);
Curl::easy::setopt
   ($curl, CURLOPT_URL, $url);
Curl::easy::setopt
   ($curl, CURLOPT_WRITEFUNCTION, &body_callback);

my @body;

Curl::easy::setopt
   ($curl, CURLOPT_FILE, @body);
Curl::easy::setopt
   ($curl, CURLOPT_ERRORBUFFER, "errbuf");
if (Curl::easy::perform
   ($curl) != 0) { print "Failed : $errbufn"; };
Curl::easy::cleanup($curl);

# Separate each line into one element in array
@lines=();

foreach (@body) { push(@lines,split('n', $_, 9999)); }

foreach (@lines) {
   # just to demonstrate it works!
   if(/icons/) { print("$_n"); }
}

exit(0);

Here is a demo screen shot of this code using perl and libcurl.



Reference

[tags]Perl libcurl demo, Perl libcurl, Perl, libcurl, curl, Perl Coding School[/tags]



Perl SHA digest

Problem

You want to generate a SHA digest for a given string.



Solution

Perl has great encryption and digesting algorithms.

In the example tab we use the Digest Perl module.



Example



perl -M'Digest::SHA1 qw(sha1_hex)'
-e '$dig=sha1_hex("my big test"); print "$dign";'

419e6139a21f51a3f2ea1a783cfe536a0dada873



Reference

[tags]Perl SHA Digest, Perl, SHA, Encryption, Perl Coding School[/tags]