SEPTEMBER 19TH, 2008
By RYAN

Ruby vs. Perl vs. Python

Sigh. I didn’t check my fileserver before I left home. It booted into a non-xVM kernel, so the code in my CentOS virtual machine isn’t accessible. Not that it’s a big deal, it just means that I’ll only be posting the code I’ve got here at work for a routine to check file modification times via SFTP. The reason d’être seems to be making sure customers aren’t trying to grab files whilst they’re being updated, but I’m not sure on that. They wanted the code, I wrote the code, now it’s doing… something.

At this point, I’m attempting to write every script in Perl, Python, and Ruby simultaneously. It shouldn’t be a surprise that I’m most productive in Perl, but I’d still like to keep up skills in Python (and build them in Ruby). For one reason or another (the fact that HP-UX systems don’t come with Python, for one; Ruby performing about 10 times as slow as Perl/Python on the 1.8 interpreter, and the 1.9 interpreter not being production ready for another [and yes, JRuby is really fast {faster than Ruby 1.9}, but the JVM startup time is a killer for a script running out of cron, and not every system in my datacenter has a JVM installed]), the Perl version is almost always the one that goes onto a production box somewhere.

It’s also worth noting that writing Ruby/Python in a procedural manner, like it’s Perl with methods tacked onto the objects, doesn’t make any sense to me. Sure, it’s kinda fun to write it that way, and blocks in Ruby are really handy (even if they’re slightly more confusing to a non-Rubyist than pointers were the first time I used them), but it feels against the spirit of it somehow. I don’t know.

That being said, this code isn’t nearly what would go into production anyway. The Python really isn’t far off. Perl’s kinda slow when it comes to objects, though, and I’d never use Class::Struct in something which weren’t totally network or I/O bound (probably iterate through arrays or hashes instead). The Ruby? Well, ahh, it’d probably look a lot like this. I’m not exactly a Rubyist at this point, though, and I’m sure it could be cleaned up a lot more without playing Perl Golf with it (hopefully by integrating Rubyisms which don’t make it God-awful unreadable, like the code at the top of this post).

As a total aside, I loathe Coding Horror just a little bit more than I hate Joel on Software. We’re talking about two guys who probably spend as much time blogging as they do working, treat their readers like idiots, preach bad practices (Joel’s company wrote its own fucking programming language rather than a DSL, and he advocates against using Exceptions, among other things. Jeff Atwood [Coding Horror] has no idea what the phrase “use the right tool for the right job” means, and would rather advocate using .NET for everything).

FWIW, I realize now why I changed to this theme. WIDE textarea. Not all the WordPress themes I’ve played with work nicely once I start fucking with the margins for it in CSS. I don’t give a damn about widget ready or what have you. All that stuff is easy enough to add by hand. I want a theme that’s not going to waste 1/3rd of the page on blankness.

WTB blogs from Dan.

So, here’s one script (the Craigslist parser will get posted tomorrow, I guess).

In Perl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174

#!/usr/bin/perl
 
###############
# SFTPCheck 
#
# 1.0
#
# Description:
# Checks update times for files on Omaha's server to troubleshoot scp problems
 
#Import what we need
use Class::Struct;
use Net::SFTP;
use Net::SFTP::Util qw{fx2txt};
use Net::SFTP::Attributes;
use strict;
use utf8;
 
#Set up  a struct we can dump data into
 
struct fileinfo => { filename => '$', 
 			  modtime => '$',
			  oldtime => '$',
			  waschanged => '$'};
 
#Set up logging
open(OUTFILE, ">> /data/testscript/ryan/sftpcheck.out");
print OUTFILE "-----------------------------------------------------------------------------------------\n";
my $now = localtime(time());
print OUTFILE "Starting at $now\n";
 
#Set up the connection parameters
my ($user, $pass, $host, $dir, $stamp, $diff) = @ARGV;
 
my @listers;
 
 
#Open it
my $sftp = undef;
$sftp = OpenSFTP($sftp, $user, $pass, $host);
 
#Set the directory path, since Net::SFTP->ls doesn't return a fully-qualified one
my $prefix = $dir . "/";
 
#Run through a loop 30 times, sleeping for one second inbetween
for(my $count = 0; $count <= 30; $count++) {
 
	#Do an ls
	@listers = &lookforfiles;
 
	foreach my $file (@listers) {
		#Set the fully qualified name so we can stat is
		my $remote = $prefix . $file;
		#Grab the modification time
		my $stat = $sftp->do_lstat($remote);
		#If it changed, set waschanged and put the new value in
		if ($file->oldtime != $stat->mtime) {
			$file->modtime($stat->mtime);
			$file->waschanged(1);
		}
		else {
			$file->modtime($stat->mtime);
		}
	}
	sleep(1);
}
 
foreach my $file (@listers) {
	if ($file->waschanged) {
		print OUTFILE $file->filename . " was changed during the check!\n";
	}
 
	#Compare it to now
	my $mtime = $file->modtime;
	my $difference = time() - $mtime;
	#If it's less than five minutes, pass it off in seconds
	if ($difference <= 300) {
		print OUTFILE $file->filename . " was updated $difference seconds ago\n";
	}
	elsif ($difference <=3600) {
		#Otherwise, minutes with two decimal places should be precise enough
		my $minutes = $difference / 60;
		printf(OUTFILE $file->filename . " was updated %.2f minutes ago\n", $minutes);
	}
	else {
		#If it's really that old, just print hours
		my $hours = $difference / 3600;
		printf(OUTFILE $file->filename . " was updated %.2f hours ago\n", $hours);
	}
}
 
sub OpenSFTP
{
	my ($sftp, $username, $pass, $host) = @_;
	my %args = ( 
		user => $username,
		password => $pass,
		debug => '1',
	);
	$username = utf8::encode($username);
	$pass = utf8::encode($pass);
	print "Trying to connect to $host as $username:$pass\n";
	$sftp = Net::SFTP->new($host, %args);
	my $status = fx2txt($sftp);
	if ($sftp) {
		return($sftp);
	}
}
 
sub lookforfiles
{
	#Figure out what the timestamp should be
	my @lookfor = &sftpstamp($stamp, $diff);
	#Check for yesterday's date too, why not.
	$lookfor[1] = $lookfor[0]--;
	my @found;
 
	#Do the LS, which doesn't support globbing, and pass it off to a regexp to find the files we need
	my @list = $sftp->ls("$dir", wanted => sub { $_[0]->{filename}});
	foreach my $day (@lookfor) {
		foreach my $file (@list) {
			my $name = $file->{filename};
			if ($name =~ /$day/) {
				my $there = 0;
				while (!$there) {
					foreach my $loop (@listers) {
						if($loop->filename($name)) {
							#If the filename is already in the array, break
							$there = 1;
						}
					}
					my $foundname = fileinfo->new();
					$foundname->filename($name);
					my $stat = $sftp->do_lstat($prefix . $name);
					$foundname->oldtime($stat->mtime);
					push(@listers, $foundname);
					$there = 1
				}
			}
		}
	}
	return(@found);
}
 
sub sftpstamp
{
    (my $stamp, my $diff) = @_;
	#Set the tzinfo so we know what the stamp should be
	my $timediff = 3600 * $diff; 
 
	#Check if we're in DST
	my $dststatus = 0;
	my $dstfile = "/data/.DST_Status";
	open(DST, $dstfile);
	$dststatus = (<DST>);
	chomp($dststatus);
	close(DST);
 
	#If so, modify the time accordingly
	if ($dststatus) {
		$timediff -= 3600;
	}
 
	#Get the time and format it
	my @StampNow = localtime(time() - $timediff);
	my $day = $StampNow[3];
	$day = "0$day" if $day < 10;
	my $month = $StampNow[4] + 1;
	$month = "0$month" if $month < 10;
 
	#Pass the appropriate value back
	$stamp = "$month$day";
	return $stamp;
}

In Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184

import base64
import getpass
import os
import platform
import socket
import sys
import time
import traceback
 
import paramiko
 
class foundfile:
    def __init__(self, filename=None, modtime=None, oldtime=None, waschanged=None):
        self.filename = filename
        self.modtime = modtime
        self.oldtime = oldtime
        self.waschanged = waschanged
 
class sftpcheck:
 
    def __init__(self, host=None, username=None, password=None, dirname=None, stamp=None, diff=None):
 
 
 
        #Set up the initial variables for login
        self.host = host
        self.username = username 
        self.password = password
        self.dirname = dirname
        self.stamp = stamp
        self.diff = diff
        self.sftp = None
        self.found = []
 
        #Set up logging
        if platform.uname()[0] == "Windows":
            self.LogPath = "\\\\filer2\\data\\testscript\\ryan\\"
            self.DSTPath = "\\\\filer2\\data\\.DST_Status"
            self.DebugPath = "\\\\filer2\\data\\testscript\\ryan\\"
        else:
            self.LogPath = "/data/testscript/ryan/"
            self.DSTPath = "/data/.DST_Status"
            self.DebugPath = "/data/testscript/ryan/"
        self.DebugFilename = "sftpdebug.log"
        self.LogFilename = "sftpcheck.log"
        self.LogFileObj = None
 
    def login(self, host, username, password):
        #Connect to the server
        t = paramiko.Transport((host, 22))
        t.connect(username=username, password=password)
        self.sftp = paramiko.SFTPClient.from_transport(t)
 
        return (self.sftp != None)
 
    def listfiles(self):
        #Pick up how it's supposed to be formatted
        lookfor = self.formattime(self.diff)
 
        #Get a list of files in the directory
        ls = self.sftp.listdir(self.dirname)
 
        #Check if any match
        for fileinfo in ls:
            if re.search(lookfor, fileinfo):
                there = False
                while there != True:
                    for record in found:
                        if record.filename == fileinfo:
                            there = True
                    newone = foundfile(filename = fileinfo)
                    mtime = self.sftp.lstat(self.dirname + fileinfo)[8]
                    newone.oldtime = mtime
                    self.found.append(newone)
                    there = True
 
        return found
 
    def statfiles(self):
        #Run through the list of files we found and stat them
        files = self.listfiles()
 
        for fileinfo in files:
            #Stat the file
            fullpath = self.dirname + fileinfo.filename
            mtime = self.sftp.lstat(fullpath)[8]
            if fileinfo.oldtime != mtime:
                fileinfo.modtime = mtime
                fileinfo.waschanged = True
            else:
                fileinfo.modtime = mtime
 
 
    def finalize(self):
        for gotem in self.found:
            mtime = gotem.modtime
            nowtime = time.time()
            if gotem.waschanged:
                self.log_write("%s was changed during the check!" % gotem.filename)
            #Figure out when the last time it was updated was
            difference = nowtime - mtime
 
            #If it's less than five minutes, seconds for logging
            if difference <= 300:
                self.log_write("%s was updated %d seconds ago" % (gotem.filename, difference))
 
            #Less than an hour?  Minutes with two decimal places
            elif difference <= 3600:
                minutes = difference / 60
                self.log_write("%s was updated %.2f minutes ago" % (gotem.filename, minutes))
 
            #Otherwise, hours with two decimal places    
            else:
                hours = minutes / 3600
                self.log_write("%s was updated %.2f hours ago" % (gotem.filename, hours))
 
    def formattime(self, diff):
        #Set timezone for PST, since I don't need to import more 
        #python libs
        timediff = 3600 * diff
 
        #Check if we're in DST
        dsthandle = open(self.DSTPath, 'r')
        dststatus = dsthandle.read().rstrip("\n")
        dsthandle.close()
 
        if dststatus == '1':
            timediff = timediff - 3600
 
        #Get the time in epoch seconds, subtract the diff
        #pass it back in a struct so we can format it
        sftptime = time.gmtime(int(time.time() - timediff))
        sftpstamp = sftptime.strftime("%m%d")
 
 
        return sftpstamp
 
    def log_open(self):
        #Set up logging
        logfile = "%s%s" % (self.LogPath, self.LogFilename)
        try:
            file = open(filename, "a")
            self.LogFileObj = file
        except IOError:
            print "Error: Cannot open log file %s!" % logfile
        else:
            self.log_write("Log file opened: %s!" % logfile)
 
    def log_write(self, message):
        #Actually write to the log
        timestamp = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
        message_text = "%s %s\n" % (timestamp, message)
        self.LogFileObj.write(message_text)
 
if __name__ == "__main__":
    #Instantiate it
    check = sftpcheck(username=args[0], password=args[1], hosts=args[2], dirname=args[3], stamp=args[4], diff=args[5])
 
    #Set up logging to troubleshoot the connection
    debugfile = "%s%s" % (check.DebugPath, check.DebugFilename)
    paramiko.util.log_to_file(debugfile)
    args = sys.argv[1:]
 
 
    #Let us know when we're starting
    check.log_open()
    check.log_write("--------------------------------------------------------")
    check.log_write("Start sftpcheck.py")
    check.log_write("Try to login")
 
    #Try to get the info
    if check.login(check.host, check.username, check.password):
        check.log_write("Successfully logged in to %s as %s" % (check.host, check.username))
        #Loop 30 times, sleeping for a second inbetween
        for i in range(1, 30):
            check.statfiles()
            time.sleep(1)
        check.finalize()
 
    #If we can't, log it can exit
    else:
        check.log_write("Login to %s as %s failed!  Check %s for more information." % (check.host, check.username, debugfile))
 
    check.log_write("Stop sftpcheck.py")

In Ruby:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147

#!/usr/bin/ruby
 
require 'net/sftp'
require 'dir'
 
class FoundFiles
  #Set up a class to hold info
  attr_accessor :filename, :modtime, :oldtime, :waschanged
end
 
class SFTPCheck
  #Initialize variables
  attr_accessor :username, :password, :host, :dir, :stamp, :diff
  @found = Array.new
  @logpath = "/data/testscript/ryan/"
  @dstpath = "/data/.DST_Status"
  @logfilename = "sftpcheck.log"
 
  def login(host, username, password)
    #Log in
    @sftp = Net::SFTP.start(host, username, :password => password)
    return @sftp
  end
 
  def formattime(diff)
    timediff = 3600 * diff
    #Read the DST status file.  Sure, Ruby has a .isdst? method in the time 
    #class, but that doesn't necessarily mesh with work
    File.open("#{dstpath}", "r") do |line|
      #Strip the newline
      dststatus = line.rstrip
    end
 
    if dststatus
      #If it's DST, change it
      timediff -= 3600
    end
 
    timenow = Time.now
    sftptime = time.at(timenow.to_i - timediff).strftime("%m%d")
    return sftptime
  end
 
  def listfiles
    lookfor = formatfile(diff)
 
    #Loop through
    ls = sftp.Dir.each_entry do |file|  
      #Ruby supports PCRE matching.  Yay!
      if file =~ /#{lookfor}/
        there = false
        #Same BS as Perl and Python.  I don't want to use a .contains method
        while there != true
          found.each do |record|
            if record.filename == file
              there = true
            end
          end
          newone = FoundFiles.new(file)
          newone.oldtime = sftp.stat(@dir + file).mtime
          #Could also be written as found.push(newone)
          @found << newone
        end
      end
    end
 
  end
 
  def statfiles
    listfiles()
 
    @found.each do |fileinfo| 
      fullpath = @dir + fileinfo.filename
      mtime = sftp.stat(fullpath).mtime
      #If it's changed, modify the object
      if fileinfo.oldtime != mtime
        fileinfo.modtime = mtime
        fileinfo.waschanged = true
      else
        fileinfo.modtime = mtime
      end
    end
  end
 
  def log_open
    #Set up logging
    logfile = @logpath + @logfilename
    @logfile = File.open(logfile, "a")
    @logfile.write("Log file opened: #{@logfile}!\n")
  end
 
  def log_write(message)
    #Actually write to the log
    @logfile.write(time.now.strftime("%Y-%m-%d %H:%M:%S") + message)
  end
 
  def finalize
    @found.each do |gotem|
      #Figure out when the last time it was updated was
      difference = gotem.mtime - time.now.to_i
      if gotem.waschanged?
        log_write("#{gotem.filename} was changed during the check!\n")
      end
 
 
      #If it's less than five minutes, seconds for logging
      if difference <= 300
        log_write("#{gotem.filename} was updated #{difference} seconds ago\n")
      #Less than an hour?
      elsif difference <= 3600
        log_write("#{gotem.filename} was updated #{(difference/60).round * 0.01} minutes ago\n")
      #Otherwise, hours.  
      else
        log_write("#{gotem.filename} was updated #{(difference/3600).round * 0.01} hours ago\n")
      end
    end
  end
 
end
 
username = ARGV[0]
password = ARGV[1]
host = ARGV[2]
dirname = ARGV[3]
stamp = ARGV[4]
diff = ARGV[5]
 
#Instantiate it
check = sftpcheck.new(username, password, host, dirname, stamp, diff)
 
#Let us know when we're starting
check.log_open
check.log_write("-------------------------------------------------------")
check.log_write("Start sftpcheck.rb")
check.log_write("Try to login")
 
#Try to get the info
if check.login(check.host, check.username, check.password)
  #Loop 30 times, sleeping for a second inbetween
  (1..30).each do |nothing|
    check.statfiles
  end
  check.finalize
else
  check.log_write("Login to #{check.host} as #{check.username} failed!\n")
end
check.log_write("Stopping sftpcheck.rb")

DISCLAIMER:
Use this code totally at your own risk. The Python and Ruby should work, but I haven’t tested them.

EDIT:
WTF. The Ruby came in significantly shorter than the Python and the Perl, plus it’s more fun to write? That makes me wish we had it installed on any of our servers here.

Tags: , , , ,

categories General

1 Comment

Other Links to this Post

RSS feed for comments on this post. TrackBack URI

Leave a comment

You must be logged in to post a comment.