Posts tagged: shitcode

0 RESPONSES comments
MAY 10TH, 2009
By RYAN

XMonad and xmobar on OpenSolaris with functional monitoring (mutt to boot)

I’ve been having carpal tunnel flareups lately, so I went looking around for things I could do to use the mouse less and the keyboard more (as counter-intuitive as that may sound, I find that holding a mouse for hours irritates it far more than any amount of typing, YMMV). Vimperator is an obvious first step, but, well, I love vi, so I already had that running.

Tiling window managers came to mind. I’ve already used wmii and dwm once upon a time, but they’re hardly state of the art (as state of the art as tiling window managers get, anyway), and hacking together a reasonable workspace status bar in dzen/Perl didn’t appeal to me. Awesome3 (the window manager) does appeal to me, but getting it running on Solaris looked like a little more work than I wanted to invest, and I’m sick of working with moving targets (Awesome3 looks like they’re almost as “break your config” happy as Rails).

Mind you, I still love Openbox, but given that 99% of my time is spent in terminals (irssi, mutt, ssh, vim, mcabber, slash’em), I see no reason why I should even bother with having window decorations and manually arranging them at all.

Had I known what I was getting into, I probably would have just used Awesome. I mean, it needed two libraries I didn’t have, and some dzen hacking. Not… this. Not that I’m unhappy with XMonad, but…

Firstly, there’s no build of GHC (Glaskow Haskell Compiler) in the OpenSolaris repositories. There’s a pre-compiled version of GHC 6.10, but only for SPARC. Pre-compiled version of GHC 6.8 for x86/amd64, but that ain’t helping me (a scary amount of stuff from Hackage, Haskell’s version of CPAN/rubygems/whathaveyou doesn’t want to run in GHC 6.8, and the recommended fix for some bugs is “upgrade to 6.10″).

No GCC 4 in the repos either, and GCC 4.1.2 is the recommended version for building GHC. So, onto the magic. Don’t even try with SunStudio. GNU-isms in the code stopped me dead.

I was bitching to Dan about how ridiculous this process was a few weeks ago. Maybe it’ll help somebody.

Firstly, either install readline from the OpenSolaris Pending repositories or compile it yourself.

Next, we need to bootstrap gcc. For that, we’ll need gmp from GNU and mpfr. Grab the precompiled version of GHC 6.8.2 while you’re at it. You should also get the newest versions of ghc and ghc-$version and ghc-$version-src-extralibs to get running later.

Complaint #1: GNU automake is braindead. It, I assume, just checks `uname` and not `isainfo`, so I can’t tell when we’re running 64 bit. Either use Solaris libtools or do the following for gmp and MPFR

./configure ABI=32 && gmake && pfexec gmake install
make distclean
configure --prefix=/usr/local/lib64 && gmake && pfexec gmake install

If you don’t specify another prefix, it’ll stomp all over the 32 bit libraries it just installed on your 64 bit box.

Complain #2: gcc is even more braindead. It’ll build a 64 bit binary but link it against 32 bit libraries, then eat itself during stage 2 bootstrap. You’d think the FSF would be smarter, but no. It just finds the wrong ELFCLASS down the line. To correct:

export LDFLAGS=-L/opt/local/lib -L/opt/local/lib/64 \
-R/usr/local/lib:/usr/local/lib/64
export LD_OPTIONS=-L/opt/local/lib -L/opt/local/lib/64 \
-R/usr/local/lib:/usr/local/lib/64
./configure && gmake -j4 && pfexec gmake install

Complaint #3: The precompiled ghc-6.8.2 we got? It sucks. The rts library is broken (check it with ldd). It would be nice to avoid this, but, well… Bootstrapping ghc from C sources and no Haskell compiler involves another goddamn system which DOES have Haskell installed, AND knowledge of what registers your CPU uses. Whoever thought up that notion of bootstrapping? Well… The “goal” is to have Haskell self-bootstrap (it currently does not, since, ironically, Haskell is too “pure” to actually be written in a “pure” language, since we need dirty things like actually doing something useful with “impure” data, like user input or stuff sucked in from a file).

cd ../ghc-6.8.2
./configure && pfexec gmake install
ghc-pkg describe rts > rts.pkg
vim rts.pkg
#add -R/usr/local/lib to the end of the ldoptions field, or ghc bombs bootstrapping the new version in stage 2
ghc-pkg update rts.pkg

Complaint #4: GHC is even stupider than GCC, if possible. Not only do we have to prepend /usr/local/bin to $PATH so GHC can find our shiny new gcc-4.1.2, we have to pass ridiculous amounts of config flags (including one which tells it where GCC is — if the $PATH OR –with-gcc is wrong, it won’t bootstrap. Don’t ask, because I don’t know why).

export PATH=/usr/local/bin:$PATH
./configure --with-gcc=/usr/local/bin/gcc --with-gmp-libraries=/usr/local/lib --with-gmp-include=/usr/local/include --with-readline-libraries=/usr/local/bin --with-readline-include=/usr/local/include
gmake -j4 && pfexec gmake install

Yay! Working GHC. Sadly, if you want to reclaim the 350MB or so the GHC 6.8 install is taking up, you have to go remove it yourself (apparently the GHC team does not believe in `make uninstall`). This means we can install cabal, which requires nothing special, other than you grabbing the tarball and installing it as normal.

Next, `cabal install xmonad xmonad-contrib` I said we were going to install xmobar, and we are, but it’s a little tricker. You see, even though Xmobar mostly reads a pipe to give us a title and workspace listing, the plugins are not optional. They also depend on libnotify, which is only present on Linux. Good job, xmobar developer! Fortunately, this is easily corrected, and xmobar (mostly) works. Caveats explained later.

You can’t just `cabal install` xmobar. No-go since hinotify will not install, and there’s not a clear explanation as to why from the output. As noted, it doesn’t really depend on it, just that the developer can’t be bothered to use Haskell’s typing system to throw messages at you when you try to use features that are not implemented. So… edit ~/.cabal/packages/hackage.haskell.org/xmobar/$version/xmobar-$version/xmonad.cabal

Take out the lines referring to hinotify. Then `cabal build && cabal install` from the directory xmonad.cabal was in. Ooh and aah, but don’t try to use, well… anything. BatteryReader, CpuReader, MemReader, Net, Swap, all broken. Thankfully, we have Dtrace and Python to replace it with, since xmobar’s PipeReader still works.

Memory usage?

#pragma D option quiet
#pragma D option bufsize=16k
 
dtrace:::BEGIN
{
}
 
profile:::tick-1sec
{
	/* RAM stats */
	this->ram_total = `physinstalled;
	this->unusable  = `physinstalled - `physmem;
	this->locked    = `pages_locked;
	this->ram_used  = `availrmem - `freemem;
	this->freemem   = `freemem;
	this->kernel    = `physmem - `pages_locked - `availrmem;
 
	this->ram_total	*= `_pagesize;  this->ram_total	/= 1048576;
	this->unusable	*= `_pagesize;  this->unusable	/= 1048576;
	this->kernel	*= `_pagesize;  this->kernel	/= 1048576;
	this->locked	*= `_pagesize;  this->locked	/= 1048576;
	this->ram_used	*= `_pagesize;  this->ram_used	/= 1048576;
	this->freemem	*= `_pagesize;  this->freemem	/= 1048576;
	printf("RAM: %2d%%\n", ((this->ram_total - this->freemem) * 100 / this->ram_total));
}

Network speeds?

#!/usr/sbin/dtrace -s
#pragma D option quiet
dtrace:::BEGIN
{
	TCP_out = 0; TCP_in = 0;
}
 
 
mib:::tcpOutDataBytes		{ TCP_out += arg0;   }
mib:::tcpInDataInorderBytes	{ TCP_in += arg0;    }
 
profile:::tick-1sec
{
	OUT_print = TCP_out/1024; IN_print = TCP_in/1024;
	printf("Out:%3d|In:%3d", OUT_print, IN_print);
	TCP_out = 0;
	TCP_in = 0;
 
}

.xmobarrc

Config { font = "-*-terminus-*-*-*-*-12-*-*-*-*-*-*-u"
       , bgColor = "#000000"
       , fgColor = "#AFAF87"
       , position = Top 
       , lowerOnStart = True
       , commands = [ Run Date "%a %b %_d %Y %H:%M:%S" "date" 10 
                    , Run Weather "KSTP" ["-t","<tempF>F","-L","64","-H","77","--normal","green","--high","red","--low","lightblue"] 36000
		    , Run PipeReader "/export/home/ryan/dtrace/net" "wireless"
		    , Run PipeReader "/export/home/ryan/dtrace/netspeed" "speed"
		    , Run PipeReader "/export/home/ryan/dtrace/psr" "cpui"
		    , Run PipeReader "/export/home/ryan/dtrace/ram" "mem"
                    , Run StdinReader
                    ]
       , sepChar = "%"
       , alignSep = "}{"
       , template = " %StdinReader% } { %cpui% | %mem% | %wireless% %speed% | %date% | %KSTP%"
       }

A script to feed those pipes. If you don’t have python2.6, pexpect on python2.4 (the Solaris/OpenSolaris default) works. Just install pexpect with easy_install, an .egg, or whatever your poison may be.

#!/usr/bin/python2.6
import math
import os
import platform
import re
import stat
import sys
import time
 
import pexpect
 
#Get the directory we're running from to create the fifos rather than the $pwd of whatever called us
path = os.path.dirname( os.path.realpath(__file__)) + "/"
 
wificonfig = ""
 
#Y'know, I haven't actually written the iwconfig thing.  It's here for posterity and possible later use.
osystem = platform.system()
if osystem == 'SunOS':
  wificonfig = 'wificonfig'
elif osystem == 'Linux':
  wificonfig = 'iwconfig'
 
 
#Kill off any old instances which may be running.  Poor man's pkill, but guaranteed to work pretty much anywhere.
pexpect.run('bash -c "ps -ef |grep mpstat |grep -v python| awk \'{print $2}\' | xargs kill -9"')
pexpect.run('bash -c "ps -ef | grep speed.d | awk \'{print $2}\' | xargs kill -9')
pexpect.run('bash -c "ps -ef | grep meminfo.d | awk \'{print $2}\' | xargs kill -9')
 
def checkfifo(path):
  #If it ain't there, make it
  if not os.path.exists(path):
    os.mkfifo(path)
    handle = open(path, "r+")
    return handle
  #If it is, just return it
  elif stat.S_ISFIFO(os.stat(path).st_mode):
    handle = open(path, "r+")
    return handle
  else:
    if os.path.isfile(path):
      #Not a FIFO, and it needs to be
      os.unlink(path)
    os.mkfifo(path)
    handle = open(path, "r+")
    return handle
 
#Set up our fifos
psrfifo = checkfifo(path + "psr")
netfifo = checkfifo(path + "net")
nspdfifo = checkfifo(path + "netspeed")
ramfifo = checkfifo(path + "ram")
 
#Fire off the processes we'll be reading from.  Using pexpect seems like overkill, but mpstat is apparently smart enough to tell when it's being read from a pipe, and it'll buffer no matter what you do.  pexpect/expect fake being interactive, so it happily runs without buffering.
mpstat = pexpect.spawn('bash -c "mpstat 1 | grep -v CPU"')
ramstats = pexpect.spawn(path + "meminfo.d")
nspeed = pexpect.spawn(path + "speed.d")
 
#Regular expressions to use later.  Since it's a long-runnign script, they may as well be compiled
mpre = re.compile(r'^\s+?(?P<cpu>\d+).*?(?P<idle>\d+)$')
solwifire = re.compile(r'.*?linkstatus: (?P<status>\w+).*essid: (?P<essid>\w+).*strength: \w+\((?P<strength>\d+)\).*', re.DOTALL)
 
def prstat():
  #Yay for awk/sed abuse, but it's concise and I'm already forking.  Basically getting a list of CPUs to check later, so this script should perform its duty no matter if you have 1 CPU or 128 (T2 users)
  psrinfo = pexpect.run('bash -c "psrinfo -v |grep MHz | awk \'{print $6,$7}\' | sed -e \'s/,//\'"').rstrip().split('\r\n')
  output = ""
  for cpu in psrinfo:
    line = prmatch(cpu)
    output = output + line
  psrfifo.write(output + "\n")
  psrfifo.flush()
 
def prmatch(cpu):
  line = mpstat.readline().rstrip()
  m = mpre.match(line)
  #Match it against our earlier regex and subtract the idle value from 100 to get the actual used percentage, which isn't wholly accurate (IOWAIT and whatnot), but it's good enough for me
  usage = 100 - int(m.group('idle'))
  #Padding the string seems stupid, and it is, but xmobar arbitrarily decides spots that it's not going to refresh even if text shows up there, leaving it (black in my case) when text slides.  Padding fixes that.  Also, if your CPU goes to 100%, you probably shouldn't have a script which reads dtrace probes running.  Just sayin'.
  return "Cpu%s: %2d%% (%s) " % (m.group('cpu'), usage, cpu)
 
def memory():
  #I haven't found any swap information from dtrace probes as easy to manipulate as thi sis
  swap = pexpect.run('bash -c "/usr/sbin/swap -l |tail -n 1 | awk \'{print $4, $5}\'"').rstrip().split(' ')
  #Ugly?  You bet.  Cast the subtract free swap from total swap blocks, divide it by free swap blocks * 100 cast to an int to give us an actual percentage, floor that, then cast THAT to an int
  usedswap = int(math.floor(((int(swap[1])-int(swap[0]))/int(swap[1])*100)))
  ram = ramstats.readline().rstrip()
  output =  "%s Swap: %2d%%" % (ram, usedswap)
  ramfifo.write(output + "\n")
  ramfifo.flush()
 
def network():
  #Filter out interface which aren't up, which are vnics, which only point to localhost to get the running interface.  I'm assuming you only have one at a time, but if you have more modify this to suit.
  iface = pexpect.run('bash -c "ifconfig -a | grep UP |grep RUNNING| grep -v IPv6 |grep -v lo | grep -v -E \':[0-9]: \' | awk \'{print $1}\' | sed -e \'s/://\'i"').rstrip()
  if osystem == 'SunOS':
    command = "wificonfig -i " + iface + " showstatus"
    status = pexpect.run(command).rstrip()
    output = ""
    if solwifire.match(status):
      #Beauty of regexes.  If it doesn't match, it's not wireless (or not connected).  It if is, give us values.
      m = solwifire.match(status)
      strength = math.floor((int(m.group('strength')) / 15.) * 100)
      output =  "%s: %s(%s) %3d%%" % (iface, m.group('status'), m.group('essid'), strength)
    else:
      #Probably not wireless
      output = iface + ":"
    netfifo.write(output + "\n") 
    netfifo.flush()
 
def netspeed():
  #Is this method really necessary?  Couldn't the dtrace probe just write to the fifo itself?  Probably, but if you (or I) want to colorize it at some point, it may as well get sucked in here.
  speed = nspeed.readline()
  nspdfifo.write(speed + "\n")
  nspdfifo.flush()
 
while 1:
 
  prstat()
  memory()
  network()
  netspeed()
  time.sleep(1)

It’s not the prettiest python. I should probably move those repetitive fifo flushes/etc to a method, but I didn’t honestly expect that I’d need to replace this much XMobar functionality. Notably, XMobar can colorize things with <span> attributes setting colors in case somebody wanted to pretty up the usages (really, to make it look more like XMobar’s [colorized] defaults for CPU/net usage). I don’t care, personally. I didn’t implement a battery monitor either, but hey, you can if you want to.

Complaint #5 (did I lose track?): SUNWmutt doesn’t have support for header caching, which is a real bitch when I have 12,000 emails. It also doesn’t support SMTP over SSL, making it pretty well worthless for Gmail. I have other accounts I use mutt for, but Gmail’s an important one.

This isn’t that tough, really. You need some kind of a database for the mutt config script, and gdbm is trivially easy to get running (normal ./configure && make && make install). On the other hand, we run into two hiccups. The configure file depends on ncurses, which is just a link to plain ol’ curses on lots of Solaris boxes. Secondly, (and I don’t really begrudge the Mutt guys for this, since there’s actually a commit to fix this, unlike GCC and GHC, whose response is “too fucking bad” [GHC actually posts the recommendation for fixing rts.pkg and the configure flags on their own site rather than FIXING THE BUILD]), configure.ac does some things wrong with libidn.

Find $with_idn, and replace the block which follows it with this (–with-idn doesn’t seem to build properly).:

if test "$with_idn" != "no" ; then
  if test "$with_idn" != "yes" ; then
    AC_CHECK_HEADERS([idn/idn-int.h],
      [AC_CHECK_HEADERS([idn/idna.h], [],
        [CPPFLAGS="$CPPFLAGS -I/usr/include/idn"])])
  fi
fi

If you don’t have or want ncurses (or it’s a symlink on your system), fix configure. `sed -i -e ’s/-lncurses/-lcurses/’ configure`.

./configure –with-regex –with-gnutls –enable-hcache –enable-smtp –enable-imap –enable-pop –enable-mailtool –with-sasl –with-idn=/usr/include/idn

Congratulations!

Next up, re-implementing htop for Solaris with dtrace probes, python, and ncurses.

Tags: , , , , , ,

categories General

4 RESPONSES comments
JANUARY 9TH, 2009
By RYAN

Palm Desktop, I stab at thee!

Firstly, I’m starting P90X tomorrow. Should be interesting. Secondly, I miss you guys :/ I’m living with somebody who asked me what the Dead Sea Scrolls are this morning, since it was on the news that they’re coming to the Science Museum.

By the way, ever planning on touching your blogs again (Sewpbox and Rattributes not included)?

So I’m migrating Heather’s Palm Desktop crap to Google Calendar (I have no idea why no tool exists to do this). Google Calendar doesn’t really like the CSV I massaged out of it (only importing about half the records), and I’m starting to see why. Half the records are fucking duplicates in every way but one. I wrote a Python script to do it for me anyway.

The long and short of it amounts to this:
If you want the easy way, export the Palm data to a .mda, import it into Yahoo Calendar, then into Google Calendar from there. Otherwise, export it to a CSV, and hit it with this script:

#!/usr/bin/ruby
#
require 'csv'
 
input = "export.csv"
output = "gcal.csv"
 
csvfile = File.open(input) {|f| f.read}
 
puts "Parsing..."
 
csv = CSV::parse(csvfile)
 
fields = csv.shift
 
puts "Writing..."
File.open(output, "w") do |f|
   f.print "Subject, Start Date, Start Time, End Date, End Time\n"
   csv.each do |line|
     startdate, starttime = Time.at(line[6].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     enddate, endtime = Time.at(line[7].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     f.print "\"#{line[11]}\",#{startdate},#{starttime},#{enddate},#{endtime}\n"
   end
end
 
puts "Done."

If you don’t feel like exporting, and are running on Windows:

#!/usr/bin/ruby
#
#
require 'win32ole'
require 'dbi'
 
class Access
   attr_accessor :mdb, :conn, :data, :fields
 
   def initialize(mdb=nil)
       @mdb = mdb
       @conn = nil
       @data = nil
       @fields = nil
   end
 
   def open
       connstring = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=#{@mdb}"
       @conn = WIN32OLE.new('ADODB.Connection')
       @conn.Open(connstring)
   end
 
   def query(sql)
       set = WIN32OLE.new('ADODB.Recordset')
       set.Open(sql, @conn)
       @fields = []
       set.Fields.each do |field|
           @fields << field.Name
       end
       @data = set.GetRows.transpose
       set.Close
   end
 
   def close
       @conn.Close
   end
end
 
output = "gcal.csv"
 
rows = Array.new
 
db = Access.new('c:\path\to\mdb')
db.open
 
db.query("SELECT * FROM Main;")
names = db.fields
rows = db.data
 
#Alternatively
DBI.connect("DBI:ODBC:driver=Microsoft Access Driver (*.mdb);"+"dbq=c:/path/to/mdb") do |dbh|
   dbh.select_all('select * from Main') {|row| rows << row}
end
 
puts "Writing..."
File.open(output, "w") do |f|
   f.print "Subject, Start Date, Start Time, End Date, End Time\n"
   rows.each do |line|
     startdate, starttime = Time.at(line[6].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     enddate, endtime = Time.at(line[7].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     f.print "\"#{line[11]}\",#{startdate},#{starttime},#{enddate},#{endtime}\n"
   end
end
 
puts "Done."

If you want the details…

Essentially, Palm’s Datebook dumps everything into an Access database. No keys or relations (granted, only 3 tables, but still), and no idea what most of the columns do. Tools for working with Jet on Linux are minimal, and I didn’t feel like going through win32ole just to get to Jet, plus this sort of thing is nicer to do in downtime at work. So, I exported it via ODBC to a Postgres database on my Solaris box. Not pretty.

access=# \d main
                 TABLE "public.main"
     COLUMN     |          Type          | Modifiers 
----------------+------------------------+-----------
 record_id      | bigint                 | NOT NULL
 STATUS         | integer                | 
 placement      | bigint                 | 
 private        | smallint               | 
 category       | character varying(20)  | 
 start_time     | bigint                 | 
 end_time       | bigint                 | 
 untimed        | smallint               | 
 time_zone      | character varying(40)  | 
 location       | character varying(255) | 
 summary        | text                   | 
 alarm_advance  | character varying(10)  | 
 alarm_unit     | character varying(10)  | 
 repeated_event | character varying(255) | 
 alarm          | smallint               | 
 note           | character varying(100) | 
access=#

Ok, so record_id seems to be some sort of key, and Heather doesn’t bother with notes or alarms, so this doesn’t seem like it’d be so bad. To figure why Google is only taking some of the records, though:

access=$ SELECT count(*) FROM main;
 count 
-------
  5094
(1 row)
access=$ SELECT count(DISTINCT record_id) FROM main;
 count 
-------
  5074
(1 row)
access=$ SELECT count(DISTINCT start_time) FROM main;
 count 
-------
  2488
(1 row)
access=$ SELECT count(DISTINCT end_time) FROM main;
 count 
-------
  2490
(1 row)
access=$ SELECT count(DISTINCT summary) FROM main;
 count 
-------
  2264
(1 row)
access=$ SELECT record_id, start_time, end_time, summary 
FROM main 
WHERE record_id IN 
    (SELECT record_id 
     FROM main 
     GROUP BY record_id 
     HAVING count(*)>1);
 record_id | start_time |  end_time  |                                 summary                                 
-----------+------------+------------+-------------------------------------------------------------------------
         0 | 1231437600 | 1231441200 | tammy 
         0 | 1231869600 | 1231873200 | nb chanber lunch
         0 | 1229642100 | 1229645700 | tammy AND joe photos st claire broiler
         0 | 1231959600 | 1231963200 | dr hunt
         0 | 1230505200 | 1230508800 | tilsen photos
         0 | 1230568200 | 1230571800 | meet gary at studio
         0 | 1230571800 | 1230584400 | bri AND kids
         0 | 1230744600 | 1230748200 | tilsen, AND sandy ORDER y membership mail
         0 | 1230681600 | 1230681600 | Dan, missy AND the kids.
         0 | 1231610400 | 1231614000 |  james j hill houseOngoing Daily 11/15/08 - 2/22/09  m-sat 10-4 sun 1-4
         0 | 1230663600 | 1230667200 | tammys house glasses shopping
         0 | 1229727600 | 1229731200 | ryan help at studio
         0 | 1231889400 | 1231893000 | 
         0 | 1231889400 | 1231903800 | EMS 
         0 | 1237161600 | 1237161600 | spring break
         0 | 1229983200 | 1229986800 | msp WITH the girls
         0 | 1241049600 | 1241049600 | DISH
         0 | 1232233200 | 1232244000 | jordan senior photos excel AND studio 
         0 | 1230055200 | 1230058800 | paige studio
         0 | 1230314400 | 1230318000 | amanda tg
         0 | 1229968800 | 1229972400 | sara AND nolan 
(21 rows)
 
access=$ SELECT record_id, start_time, end_time, summary 
FROM main 
ORDER BY start_time 
ASC LIMIT 10;
 record_id | start_time | end_time | summary 
-----------+------------+----------+---------
   7128069 |   31449600 | 31449600 | c
   7128068 |   31449600 | 31449600 | a
   7123605 |   31449600 | 31449600 | a
   7128070 |   31449600 | 31449600 | 3
   7124866 |   31449600 | 31449600 | c
   7124107 |   31449600 | 31449600 | 3
   7124145 |   31449600 | 31449600 | o
   7124141 |   31449600 | 31449600 | ;
   7128072 |   31449600 | 31449600 | ;
   7128071 |   31449600 | 31449600 | o
(10 rows)
access=$ SELECT record_id, start_time, end_time, summary FROM main ORDER BY start_time DESC LIMIT 10;
 record_id | start_time |  end_time  |           summary           
-----------+------------+------------+-----------------------------
   7127485 | 1256774400 | 1256774400 | lawerance wedding
   7125815 | 1256774400 | 1256774400 | lawerance wedding
   7128114 | 1244167200 | 1244170800 | NB senior ALL night party
   7125941 | 1242489600 | 1242493200 | nyquist edding
   7125827 | 1242489600 | 1242493200 | nyquist edding
         0 | 1241049600 | 1241049600 | DISH
   7128073 | 1238079600 | 1238083200 | books IN the woods
   7125623 | 1238079600 | 1238083200 | books IN the woods
   7125697 | 1238025600 | 1238025600 | gunflint books IN the woods
   7126175 | 1238025600 | 1238025600 | gunflint books IN the woods
(10 rows)
 
access=$

Oh, yeah! What I’ve gathered:

  • There are duplicate record_ids (which I’d hoped would have been unique).
  • There are events set to start and end at duplicate times
  • Palm, at some point, duplicated a lot of the other records, except for the record_id.
  • Times are stored in epoch seconds (oddly, Unix epoch seconds, not Windows)
  • Some of the times correlate to 1970? WTF

A working solution:

access=$ SELECT DISTINCT a.start_time, a.end_time, a.summary 
INTO holdkey 
FROM main a
WHERE EXISTS 
    ( SELECT 'x' FROM main b WHERE a.start_time = b.start_time
      AND a.end_time = b.end_time
      AND a.summary = b.summary) 
ORDER BY a.start_time DESC;
SELECT
access=$ SELECT count(*) FROM holdkey;
 count 
-------
  2597
(1 row)
access=$ DELETE FROM main 
USING holdkey 
WHERE main.start_time = holdkey.start_time 
    AND main.end_time = holdkey.end_time 
    AND main.summary = holdkey.summary;
DELETE 5085
 
access=$ SELECT record_id, start_time, end_time, summary FROM main;
 record_id | start_time |  end_time  | summary 
-----------+------------+------------+---------
   5280360 |   31536000 |   31536000 | 
   5280298 |   31536000 |   31536000 | 
   5280429 |   31536000 |   31536000 | 
   7125497 | 1193437800 | 1193437800 | 
   7128378 |   31536000 |   31536000 | 
   7128376 |   31536000 |   31536000 | 
   7128374 |   31536000 |   31536000 | 
   7127620 | 1193437800 | 1193437800 | 
         0 | 1231889400 | 1231893000 | 
(9 rows)
access=$ DROP TABLE main;
DROP TABLE
access=$ SELECT * INTO main FROM holdkey;
SELECT

That works. Of course there’s the quick and dirty way which doesn’t involve munging about with temp tables:

access=$ DELETE FROM main t1
USING main 
WHERE EXISTS 
    (SELECT * FROM main t2 
         WHERE t1.start_time = t2.start_time 
         AND t1.end_time = t2.end_time 
         AND t1.summary = t2.summary 
         AND t1.record_id < t2.record_id);
DELETE 2488
 
access=$ SELECT count(*) FROM test;
 count 
-------
  2606
(1 row)

It gives a slightly different result, but operates under the assumption that Palm’s record_id means something (it may not, for all I know). On the upside, it preserves all the columns in case they’re useful for something (doubtful). I could order by start_time and select into another table, add an index, and do the same thing, but it’s easier the quick and dirty way. There’s probably a trivial way to do this with joins, but I couldn’t think of one, and it leaves 9 records with a record_id of 0..

Here’s the code which it turns out I didn’t need, but it might be useful to somebody:

#Rips data from Palm Desktop.  Uploads it to Google Calendar
#Written with Python 2.5 (though imports should work anyway)
#
#Currently, the Access MDB Palm Datebook uses has been exported to a 
#PostgreSQL server via ODBC, so I'll be connecting to that
#
#There's code in here for getting through Access also, but I haven't tested it.
#Use at your own risk (kinda like Access).
#
#This is mostly due to the Postgres ODBC driver, and the fact that I didn't
#want to bother with quoting all the queries for Postgres to allow spaces
 
try:
    from xml.etree import ElementTree #Python 2.5, probably 2.6/3.0 also
except ImportError:
    from elementtree import ElementTree #Python <2.4
import gdata.calendar.service
import gdata.service
import atom.service
import gdata.calendar
import atom
import getopt
import sys
import string
import time
import psycopg2 #Talk to Postgres
 
class Struct:
    def __init__(self, *args, **kwargs):
        for k,v in kwargs.items():
            setattr(self, k, v)
 
class GCalMigrate:
    def __init__(self):
        self.conn = None
        self.cur = None
        self.calendar = None
        self.records = []
 
    def connect(self):
       try:
           self.conn = psycopg2.connect("dbname='whatever' user='yournamehere' host='server'")
       except:
           print "Can't connect to the database!\n"
           sys.exit()
       self.cur = conn.cursor()
       query()
 
    def accessconnect(self,mdbpath):
        import odbc
        self.conn = odbc.odbc("driver=Microsoft Access Driver (*.mdb);DBQ=%s") % mdbpath
        self.cur = conn.cursor()
        queryaccess()
 
    def queryaccess(self):
        rows = []
        self.cur.execute("SELECT Main.[Start Time], Main.[End Time], Main.[Summary] FROM Main")
        rows = cur.fetchall()
        conn.close()
        parserows(rows)
 
    def query(self):
       rows = []
       try:
           self.cur.execute("SELECT start_time, end_time, summary FROM main")
           rows = cur.fetchall()
       except:
           print "Couldn't query the database.\n"
       conn.close()
       parserows(rows)
 
    def parserows(self, rows):
        for row in rows:
            starttime = time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime(row[0]))
            endtime = time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime(row[1]))
            title = row[2]
            record = Struct(start_time=starttime, end_time=enddtime, title=title)
            self.records.append(record)
        login()
 
    def login(self, username, password):
        self.calendar = gdata.calendar.service.CalendarService()
        self.calendar.email = username
        self.calendar.password = password
        self.calendar.source = "Palm_Desktop_Migrator"
        self.calendar.ProgrammaticLogin()
        batchsubmit()
 
 
    def batchsubmit(self):
        feed = gdata.calendar.CalendarEventFeed()
 
        for record in records:
            insertme = gdata.calendar.CalendarEventEntry()
            insertme.title = atom.Title(record.title)
            insertme.content = atom.Content("")
            insertme.when.append(gdata.calendar.When(start_time=record.start_time, end_time=record.end_time))
            insertme.batch_id = gdata.BatchId(text='Palm_Migration')
 
            feed.Add_Insert(entry=insertme)
        response = self.calendar.ExecuteBatch(feed, gdata.calendar.service.DEFAULT_BATCH_URL)
        return response
 
if __name__ == "__main__":
    runner = GCalMigrate()
    responses = runner.connect()
    for entry in responses.entry:
        print "Batch ID: %s" % entry.batch_id.text
        print "Status: %s" % entry.batch_status.code
        print "Reason: %s" % entry.batch_status.reason

Tags: , , , , ,

categories Bug, General

3 RESPONSES comments
MARCH 11TH, 2008
By RYAN

Just a taste –

Since I’ve been bitching to Dan lately about the code I inherited, here are a few examples:

System.DateTime answer = today.AddDays(-1);
YesterdayTemp = answer.ToString("dd");
TodayTemp = today.ToString("dd");
string TodayMonth = today.ToString("MM");
string currlogdir = @"C:\Faxserver\Logs\";
int currHour = DateTime.Now.Hour;
int currMinute = DateTime.Now.Minute;
int currSecond = DateTime.Now.Second;
if (currHour == 00 && currMinute == 00 && currSecond == 00)
{
      if (File.Exists(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt"))
      {
           DateTime tempcreatetime6 = File.GetCreationTime(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt");
           string tempday6 = tempcreatetime6.ToString("dd");
           string tempmonth6 = tempcreatetime6.ToString("MM");
           if (TodayTemp != tempday6 || TodayMonth != tempmonth6)
           {
                File.Delete(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt");
           }
      }
      FaxCountLog(YesterdayTemp, lblFaxesTodayDisplay.Text);
      FaxErrorCountLog(YesterdayTemp, lblErrorsTodayDisplay.Text);
      lblFaxesYesterdayDisplay.Text = lblFaxesTodayDisplay.Text;
      lblErrorsYesterdayDisplay.Text = lblErrorsTodayDisplay.Text;
      TextlblDisplayTodayAdd("0");
      TextlblDisplayTodayErrorAdd("0");
      TextlblDisplayHourAdd("0");
      TextlblDisplayHourErrorAdd("0");
      FaxesHour = 0;
      ErrorsHour = 0;
      FaxesToday = 0;
      ErrorsToday = 0;
}
if (currHour != 00 && currMinute == 00 && currSecond == 00)
{
     FaxCountLog(TodayTemp, lblFaxesTodayDisplay.Text);
     FaxErrorCountLog(TodayTemp, lblErrorsTodayDisplay.Text);
     TextlblDisplayHourAdd("0");
     TextlblDisplayHourErrorAdd("0");
     FaxesHour = 0;
     ErrorsHour = 0;
}
if (ElapsedHours == 23 && ElapsedMinutes == 59)
{
     ElapsedDays += 1;
     ElapsedHours = 00;
     ElapsedMinutes = 00;
     ElapsedSeconds = 00;
}
if (ElapsedMinutes == 59)
     {
     ElapsedHours += 1;
     ElapsedMinutes = 00;
}
if (ElapsedSeconds == 59)
{
     ElapsedMinutes += 1;
     ElapsedSeconds = 00;
}
else
{
     ElapsedSeconds += 1;
}
if (ElapsedHours >= 0 && ElapsedHours <= 9)
{
     ElapsedHoursTemp = "0" + ElapsedHours;
}
else
{
     ElapsedHoursTemp = ElapsedHours.ToString();
}
if (ElapsedMinutes >= 0 && ElapsedMinutes <= 9)
{
     ElapsedMinutesTemp = "0" + ElapsedMinutes;
}
else
{
     ElapsedMinutesTemp = ElapsedMinutes.ToString();
}
if (ElapsedSeconds >= 0 && ElapsedSeconds <= 9)
{
     ElapsedSecondsTemp = "0" + ElapsedSeconds;
}
else
{
     ElapsedSecondsTemp = ElapsedSeconds.ToString();
}
blElapsedTimeDisplay.Text = ElapsedDays + " Day(s), " + ElapsedHoursTemp + ":" + ElapsedMinutesTemp + ":" + ElapsedSecondsTemp;
}

Replaced with:

int totalSeconds = DateTime.Today.TotalSeconds;
Stopwatch elapsedTime = new Stopwatch();
elapsedTime.Start();
public class Log {
     public void Rotate (FileInfo logFile)
      {
          if ((logFile.Exists) && (logFile.LastAccessTime.Date != DateTime.Today.Date))
          {
               logFile.Delete()
          }
      }
      /* snip */
}
public class Reset {
     public void Labels(bool day, bool hour)
          {
                lblFaxesYesterdayDisplay.Text = lblFaxesTodayDisplay.Txt;
           /*snip*/
           }
     public void Counts(bool day, bool hour)
     {
                /*snip*/
     }
}
Reset reset = new reset;
FileInfo fileErrorLog = new FileInfo(logdir + "AutoFaxErrorCountLog_" + DateTime.Today.Day + ".txt");
if (totalSeconds % 3600 == 0)
{
     if (totalSeconds == 0)
     {
          log.Rotate(fileErrorLog);
          reset.Labels(true, true);
          reset.Counts(true, true);
     }
     else 
     {
          reset.Labels(false, true);
          reset.Labels(false, true);
     }
reset.Dispose();
}
TimeSpan ts = elapsedTime.Elapsed;
lblElapsedTimeDisplay.Text = String.Format("{0} Day(s), {1:00}:{2:00}:{3:00}", ts.Days, ts.Hours, ts.Minutes, ts.Seconds);
}

No, I’m not using Delgates until I can decouple this:
AutoFax.Form1.btnCleanErrors_Click(object, System.EventArgs) AutoFax.Form1.btnRestartPrintService_Click(object, System.EventArgs)
AutoFax.Form1.btnStart_Click(object, System.EventArgs)
AutoFax.Form1.CheckRdy() AutoFax.Form1.CheckSaved()
AutoFax.Form1.ChooseThreads(int)
AutoFax.Form1.CreateNotAndRdy()
AutoFax.Form1.Dispose(bool)
AutoFax.Form1.ErrorOutSecondsNumberGet()
AutoFax.Form1.ErrorOutSecondsNumberSet(int)
AutoFax.Form1.FaxCountLog(string, string)
AutoFax.Form1.FaxErrorCountLog(string, string)
AutoFax.Form1.FaxLog(string, bool)
AutoFax.Form1.Form1()
AutoFax.Form1.Form1_Exit(object, System.EventArgs)
AutoFax.Form1.GetFaxCount()
AutoFax.Form1.InitializeComponent()
AutoFax.Form1.Main()
AutoFax.Form1.menuItemAbout_Click(object, System.EventArgs)
AutoFax.Form1.menuItemShutDown_Click(object, System.EventArgs)
AutoFax.Form1.OnChanged(object, System.IO.FileSystemEventArgs)
AutoFax.Form1.ProcessFile(string, bool)
AutoFax.Form1.ProgressIncrement(int)
AutoFax.Form1.Rdy()
AutoFax.Form1.read_ini_settings()
AutoFax.Form1.SendAlert(string)
AutoFax.Form1.SendEmail(string)
AutoFax.Form1.SendFax()
AutoFax.Form1.SendFaxToProgram(string, string)
AutoFax.Form1.Textbox1Add(string)
AutoFax.Form1.Textbox1Append(string)
AutoFax.Form1.TextbtnStartadd(string)
AutoFax.Form1.TextlblDisplayHourAdd(string)
AutoFax.Form1.TextlblDisplayHourErrorAdd(string)
AutoFax.Form1.TextlblDisplayTodayAdd(string)
AutoFax.Form1.TextlblDisplayTodayErrorAdd(string)
AutoFax.Form1.TextlblFaxesAdd(string)
AutoFax.Form1.TextlblQueAdd(string)
AutoFax.Form1.TextlblSendAdd(string)
AutoFax.Form1.timer1_Tick_1(object, System.EventArgs)
AutoFax.Form1.timer2_Tick(object, System.EventArgs)
AutoFax.Form1.timerHBT_Tick(object, System.EventArgs)

Who needs classes? That Form1 class can just get every method known to man.
More joy:

String rdydir = @"C:\RDY\INCOMING\";
String notdir = @"G:\NOT\";
String str1 = path.ToUpper();
String delim = rdydir;
String delim1 = "NOT.RDY";
String str2 = str1.Trim(delim.ToCharArray());
if ((File.GetAttributes(path) & FileAttributes.Normal) == FileAttributes.Hidden)
{
    File.Delete(path);
    FaxLog("deleted file: [" + path + "]", false);
}
else
{
File.SetAttributes(path, FileAttributes.Normal);
File.Delete(path);
FaxLog("deleted file: [" + path + "]", false);
}
String str3 = str2.Trim(delim1.ToCharArray());
String NotFile = str3 + ".NOT";
TextlblQueAdd("looking for: [" + NotFile + "]");

Replacement:

 string notdir = @"G:\NOT\";
string NotFile = Regex.Replace(path, @".*\\(\w+\d+).*", "$1" + ".NOT").ToUpper();

This is just…

string templine = sr.ReadLine();
sr.Close();
if (templine != null)
{
      int test = templine.IndexOf("[");
      int test2 = templine.IndexOf("]");
      string test3 = templine.Substring(test + 1, test2 - 1 - test);
      if (test3 != null)
      {
            Regex reDateTime = new Regex(@"^(\d{2})/(\d{2})/(\d{4}) (\d{2}):(\d{2}):(\d{2})$");
            if (reDateTime.IsMatch(test3))
            {
                  int test4 = test3.IndexOf(" ");
                  string test5 = test3.Substring(0, test4);
                  if (test5 == TodayMonthTempStr + "/" + TodayDayTempStr + "/" + TodayYearTemp)
                  {
                        filedatetimeisthesame = true;
                        // file exists and it has today's date, so just append to file
                        FileStream fs2 = new FileStream(templogpath + BeginLogName + MiddleLogName + TodayDayTempStr + ".txt", FileMode.Append, FileAccess.Write);
                        StreamWriter m_streamWriter2 = new StreamWriter(fs2);
                        string FaxLogDate2 = DateTime.UtcNow.ToUniversalTime().ToShortDateString();
                        string FaxLogTime2 = DateTime.UtcNow.ToUniversalTime().ToLongTimeString();
                        m_streamWriter2.WriteLine("[" + FaxLogDate2 + " " + FaxLogTime2 + "], " + FaxLog);
                        m_streamWriter2.Flush();
                        m_streamWriter2.Close();
                        fs2.Close();
                   }
              }
         }
    }
}

Replace:

if (fileAutoFaxLog.LastWriteTime.Date == DateTime.Today.Date) 
{  
      filedatetimeisthesame = true;
       // file exists and it has today's date, so just append to file
       FileStream fs2 = new FileStream(@"C:\Faxserver\Logs\AutofaxLog_" + DateTime.Now.ToString("dd") + ".txt", FileMode.Append, FileAccess.Write);
       StreamWriter m_streamWriter2 = new StreamWriter(fs2);
       m_streamWriter2.WriteLine("[" + DateTime.Now.ToString() + "], " + FaxLog);
       m_streamWriter2.Flush();
       m_streamWriter2.Close();
       m_streamWriter2.Dispose();
       fs2.Close();
       fs2.Dispose();
}

Lots of these (15 or so) in the code:

DateTime hbtTime = DateTime.UtcNow.ToUniversalTime();
string hbtMonth = hbtTime.Month.ToString();
if (hbtMonth.Length == 1)
{
     hbtMonth = "0" + hbtMonth;
}
string hbtDay = hbtTime.Day.ToString();
if (hbtDay.Length == 1)
{
     hbtDay = "0" + hbtDay;
}
string hbtYear = hbtTime.Year.ToString();
string hbtHour = hbtTime.Hour.ToString();
if (hbtHour.Length == 1)
{
     hbtHour = "0" + hbtHour;
}
string hbtMinute = hbtTime.Minute.ToString();
if (hbtMinute.Length == 1)
{
     hbtMinute = "0" + hbtMinute;
}
     string hbtSecond = hbtTime.Second.ToString();
if (hbtSecond.Length == 1)
{
      hbtSecond = "0" + hbtSecond;
}
string strHBTtime = hbtMonth + "/" + hbtDay + "/" + hbtYear + ", " + hbtHour + ":" + hbtMinute + ":" + hbtSecond;

Which is really:

string strHBTtime = DateTime.Now.ToString("MM/dd/yy HH:mm:ss");

This isn’t mentioning all the other weird shit in the code, like queues are always:

object whythefuck = queue.Peek();
string eh = whythefuck.ToString();
queue.Dequeue();

Rather than:

string morereadablelesswaste = queue.Dequeue().ToString();

Flags being set in 1,000 places to do the same thing. Every catch is handled the same way, but lots of copy+paste code rather than passing the Message to a method to handle it (better to just repeat ad-nauseum). Re-inventing booleans with lower performance by setting strings to “true” or “false” then comparing those (they never have any other value). No Dipose() or Finalize(). Ever. No comments. Pointless variable names that mean nothing, like tmpStr14. That also brings up: Hungarian notation. Gods, why use it in a strongly-typed language?

What started off as a 20 minute “add a method to email me” is going to be a “rework 4500 lines of code into (probably) 2000, by cleaning up the trash everywhere then adding classes and delegates.” Joy.

Tags: , ,

categories Bug, General