Category: Bug

4 RESPONSES comments
JANUARY 9TH, 2009
By RYAN

Palm Desktop, I stab at thee!

Firstly, I’m starting P90X tomorrow. Should be interesting. Secondly, I miss you guys :/ I’m living with somebody who asked me what the Dead Sea Scrolls are this morning, since it was on the news that they’re coming to the Science Museum.

By the way, ever planning on touching your blogs again (Sewpbox and Rattributes not included)?

So I’m migrating Heather’s Palm Desktop crap to Google Calendar (I have no idea why no tool exists to do this). Google Calendar doesn’t really like the CSV I massaged out of it (only importing about half the records), and I’m starting to see why. Half the records are fucking duplicates in every way but one. I wrote a Python script to do it for me anyway.

The long and short of it amounts to this:
If you want the easy way, export the Palm data to a .mda, import it into Yahoo Calendar, then into Google Calendar from there. Otherwise, export it to a CSV, and hit it with this script:

#!/usr/bin/ruby
#
require 'csv'
 
input = "export.csv"
output = "gcal.csv"
 
csvfile = File.open(input) {|f| f.read}
 
puts "Parsing..."
 
csv = CSV::parse(csvfile)
 
fields = csv.shift
 
puts "Writing..."
File.open(output, "w") do |f|
   f.print "Subject, Start Date, Start Time, End Date, End Time\n"
   csv.each do |line|
     startdate, starttime = Time.at(line[6].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     enddate, endtime = Time.at(line[7].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     f.print "\"#{line[11]}\",#{startdate},#{starttime},#{enddate},#{endtime}\n"
   end
end
 
puts "Done."

If you don’t feel like exporting, and are running on Windows:

#!/usr/bin/ruby
#
#
require 'win32ole'
require 'dbi'
 
class Access
   attr_accessor :mdb, :conn, :data, :fields
 
   def initialize(mdb=nil)
       @mdb = mdb
       @conn = nil
       @data = nil
       @fields = nil
   end
 
   def open
       connstring = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=#{@mdb}"
       @conn = WIN32OLE.new('ADODB.Connection')
       @conn.Open(connstring)
   end
 
   def query(sql)
       set = WIN32OLE.new('ADODB.Recordset')
       set.Open(sql, @conn)
       @fields = []
       set.Fields.each do |field|
           @fields << field.Name
       end
       @data = set.GetRows.transpose
       set.Close
   end
 
   def close
       @conn.Close
   end
end
 
output = "gcal.csv"
 
rows = Array.new
 
db = Access.new('c:\path\to\mdb')
db.open
 
db.query("SELECT * FROM Main;")
names = db.fields
rows = db.data
 
#Alternatively
DBI.connect("DBI:ODBC:driver=Microsoft Access Driver (*.mdb);"+"dbq=c:/path/to/mdb") do |dbh|
   dbh.select_all('select * from Main') {|row| rows << row}
end
 
puts "Writing..."
File.open(output, "w") do |f|
   f.print "Subject, Start Date, Start Time, End Date, End Time\n"
   rows.each do |line|
     startdate, starttime = Time.at(line[6].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     enddate, endtime = Time.at(line[7].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     f.print "\"#{line[11]}\",#{startdate},#{starttime},#{enddate},#{endtime}\n"
   end
end
 
puts "Done."

If you want the details…

Essentially, Palm’s Datebook dumps everything into an Access database. No keys or relations (granted, only 3 tables, but still), and no idea what most of the columns do. Tools for working with Jet on Linux are minimal, and I didn’t feel like going through win32ole just to get to Jet, plus this sort of thing is nicer to do in downtime at work. So, I exported it via ODBC to a Postgres database on my Solaris box. Not pretty.

access=# \d main
                 TABLE "public.main"
     COLUMN     |          Type          | Modifiers 
----------------+------------------------+-----------
 record_id      | bigint                 | NOT NULL
 STATUS         | integer                | 
 placement      | bigint                 | 
 private        | smallint               | 
 category       | character varying(20)  | 
 start_time     | bigint                 | 
 end_time       | bigint                 | 
 untimed        | smallint               | 
 time_zone      | character varying(40)  | 
 location       | character varying(255) | 
 summary        | text                   | 
 alarm_advance  | character varying(10)  | 
 alarm_unit     | character varying(10)  | 
 repeated_event | character varying(255) | 
 alarm          | smallint               | 
 note           | character varying(100) | 
access=#

Ok, so record_id seems to be some sort of key, and Heather doesn’t bother with notes or alarms, so this doesn’t seem like it’d be so bad. To figure why Google is only taking some of the records, though:

access=$ SELECT count(*) FROM main;
 count 
-------
  5094
(1 row)
access=$ SELECT count(DISTINCT record_id) FROM main;
 count 
-------
  5074
(1 row)
access=$ SELECT count(DISTINCT start_time) FROM main;
 count 
-------
  2488
(1 row)
access=$ SELECT count(DISTINCT end_time) FROM main;
 count 
-------
  2490
(1 row)
access=$ SELECT count(DISTINCT summary) FROM main;
 count 
-------
  2264
(1 row)
access=$ SELECT record_id, start_time, end_time, summary 
FROM main 
WHERE record_id IN 
    (SELECT record_id 
     FROM main 
     GROUP BY record_id 
     HAVING count(*)>1);
 record_id | start_time |  end_time  |                                 summary                                 
-----------+------------+------------+-------------------------------------------------------------------------
         0 | 1231437600 | 1231441200 | tammy 
         0 | 1231869600 | 1231873200 | nb chanber lunch
         0 | 1229642100 | 1229645700 | tammy AND joe photos st claire broiler
         0 | 1231959600 | 1231963200 | dr hunt
         0 | 1230505200 | 1230508800 | tilsen photos
         0 | 1230568200 | 1230571800 | meet gary at studio
         0 | 1230571800 | 1230584400 | bri AND kids
         0 | 1230744600 | 1230748200 | tilsen, AND sandy ORDER y membership mail
         0 | 1230681600 | 1230681600 | Dan, missy AND the kids.
         0 | 1231610400 | 1231614000 |  james j hill houseOngoing Daily 11/15/08 - 2/22/09  m-sat 10-4 sun 1-4
         0 | 1230663600 | 1230667200 | tammys house glasses shopping
         0 | 1229727600 | 1229731200 | ryan help at studio
         0 | 1231889400 | 1231893000 | 
         0 | 1231889400 | 1231903800 | EMS 
         0 | 1237161600 | 1237161600 | spring break
         0 | 1229983200 | 1229986800 | msp WITH the girls
         0 | 1241049600 | 1241049600 | DISH
         0 | 1232233200 | 1232244000 | jordan senior photos excel AND studio 
         0 | 1230055200 | 1230058800 | paige studio
         0 | 1230314400 | 1230318000 | amanda tg
         0 | 1229968800 | 1229972400 | sara AND nolan 
(21 rows)
 
access=$ SELECT record_id, start_time, end_time, summary 
FROM main 
ORDER BY start_time 
ASC LIMIT 10;
 record_id | start_time | end_time | summary 
-----------+------------+----------+---------
   7128069 |   31449600 | 31449600 | c
   7128068 |   31449600 | 31449600 | a
   7123605 |   31449600 | 31449600 | a
   7128070 |   31449600 | 31449600 | 3
   7124866 |   31449600 | 31449600 | c
   7124107 |   31449600 | 31449600 | 3
   7124145 |   31449600 | 31449600 | o
   7124141 |   31449600 | 31449600 | ;
   7128072 |   31449600 | 31449600 | ;
   7128071 |   31449600 | 31449600 | o
(10 rows)
access=$ SELECT record_id, start_time, end_time, summary FROM main ORDER BY start_time DESC LIMIT 10;
 record_id | start_time |  end_time  |           summary           
-----------+------------+------------+-----------------------------
   7127485 | 1256774400 | 1256774400 | lawerance wedding
   7125815 | 1256774400 | 1256774400 | lawerance wedding
   7128114 | 1244167200 | 1244170800 | NB senior ALL night party
   7125941 | 1242489600 | 1242493200 | nyquist edding
   7125827 | 1242489600 | 1242493200 | nyquist edding
         0 | 1241049600 | 1241049600 | DISH
   7128073 | 1238079600 | 1238083200 | books IN the woods
   7125623 | 1238079600 | 1238083200 | books IN the woods
   7125697 | 1238025600 | 1238025600 | gunflint books IN the woods
   7126175 | 1238025600 | 1238025600 | gunflint books IN the woods
(10 rows)
 
access=$

Oh, yeah! What I’ve gathered:

  • There are duplicate record_ids (which I’d hoped would have been unique).
  • There are events set to start and end at duplicate times
  • Palm, at some point, duplicated a lot of the other records, except for the record_id.
  • Times are stored in epoch seconds (oddly, Unix epoch seconds, not Windows)
  • Some of the times correlate to 1970? WTF

A working solution:

access=$ SELECT DISTINCT a.start_time, a.end_time, a.summary 
INTO holdkey 
FROM main a
WHERE EXISTS 
    ( SELECT 'x' FROM main b WHERE a.start_time = b.start_time
      AND a.end_time = b.end_time
      AND a.summary = b.summary) 
ORDER BY a.start_time DESC;
SELECT
access=$ SELECT count(*) FROM holdkey;
 count 
-------
  2597
(1 row)
access=$ DELETE FROM main 
USING holdkey 
WHERE main.start_time = holdkey.start_time 
    AND main.end_time = holdkey.end_time 
    AND main.summary = holdkey.summary;
DELETE 5085
 
access=$ SELECT record_id, start_time, end_time, summary FROM main;
 record_id | start_time |  end_time  | summary 
-----------+------------+------------+---------
   5280360 |   31536000 |   31536000 | 
   5280298 |   31536000 |   31536000 | 
   5280429 |   31536000 |   31536000 | 
   7125497 | 1193437800 | 1193437800 | 
   7128378 |   31536000 |   31536000 | 
   7128376 |   31536000 |   31536000 | 
   7128374 |   31536000 |   31536000 | 
   7127620 | 1193437800 | 1193437800 | 
         0 | 1231889400 | 1231893000 | 
(9 rows)
access=$ DROP TABLE main;
DROP TABLE
access=$ SELECT * INTO main FROM holdkey;
SELECT

That works. Of course there’s the quick and dirty way which doesn’t involve munging about with temp tables:

access=$ DELETE FROM main t1
USING main 
WHERE EXISTS 
    (SELECT * FROM main t2 
         WHERE t1.start_time = t2.start_time 
         AND t1.end_time = t2.end_time 
         AND t1.summary = t2.summary 
         AND t1.record_id < t2.record_id);
DELETE 2488
 
access=$ SELECT count(*) FROM test;
 count 
-------
  2606
(1 row)

It gives a slightly different result, but operates under the assumption that Palm’s record_id means something (it may not, for all I know). On the upside, it preserves all the columns in case they’re useful for something (doubtful). I could order by start_time and select into another table, add an index, and do the same thing, but it’s easier the quick and dirty way. There’s probably a trivial way to do this with joins, but I couldn’t think of one, and it leaves 9 records with a record_id of 0..

Here’s the code which it turns out I didn’t need, but it might be useful to somebody:

#Rips data from Palm Desktop.  Uploads it to Google Calendar
#Written with Python 2.5 (though imports should work anyway)
#
#Currently, the Access MDB Palm Datebook uses has been exported to a 
#PostgreSQL server via ODBC, so I'll be connecting to that
#
#There's code in here for getting through Access also, but I haven't tested it.
#Use at your own risk (kinda like Access).
#
#This is mostly due to the Postgres ODBC driver, and the fact that I didn't
#want to bother with quoting all the queries for Postgres to allow spaces
 
try:
    from xml.etree import ElementTree #Python 2.5, probably 2.6/3.0 also
except ImportError:
    from elementtree import ElementTree #Python <2.4
import gdata.calendar.service
import gdata.service
import atom.service
import gdata.calendar
import atom
import getopt
import sys
import string
import time
import psycopg2 #Talk to Postgres
 
class Struct:
    def __init__(self, *args, **kwargs):
        for k,v in kwargs.items():
            setattr(self, k, v)
 
class GCalMigrate:
    def __init__(self):
        self.conn = None
        self.cur = None
        self.calendar = None
        self.records = []
 
    def connect(self):
       try:
           self.conn = psycopg2.connect("dbname='whatever' user='yournamehere' host='server'")
       except:
           print "Can't connect to the database!\n"
           sys.exit()
       self.cur = conn.cursor()
       query()
 
    def accessconnect(self,mdbpath):
        import odbc
        self.conn = odbc.odbc("driver=Microsoft Access Driver (*.mdb);DBQ=%s") % mdbpath
        self.cur = conn.cursor()
        queryaccess()
 
    def queryaccess(self):
        rows = []
        self.cur.execute("SELECT Main.[Start Time], Main.[End Time], Main.[Summary] FROM Main")
        rows = cur.fetchall()
        conn.close()
        parserows(rows)
 
    def query(self):
       rows = []
       try:
           self.cur.execute("SELECT start_time, end_time, summary FROM main")
           rows = cur.fetchall()
       except:
           print "Couldn't query the database.\n"
       conn.close()
       parserows(rows)
 
    def parserows(self, rows):
        for row in rows:
            starttime = time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime(row[0]))
            endtime = time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime(row[1]))
            title = row[2]
            record = Struct(start_time=starttime, end_time=enddtime, title=title)
            self.records.append(record)
        login()
 
    def login(self, username, password):
        self.calendar = gdata.calendar.service.CalendarService()
        self.calendar.email = username
        self.calendar.password = password
        self.calendar.source = "Palm_Desktop_Migrator"
        self.calendar.ProgrammaticLogin()
        batchsubmit()
 
 
    def batchsubmit(self):
        feed = gdata.calendar.CalendarEventFeed()
 
        for record in records:
            insertme = gdata.calendar.CalendarEventEntry()
            insertme.title = atom.Title(record.title)
            insertme.content = atom.Content("")
            insertme.when.append(gdata.calendar.When(start_time=record.start_time, end_time=record.end_time))
            insertme.batch_id = gdata.BatchId(text='Palm_Migration')
 
            feed.Add_Insert(entry=insertme)
        response = self.calendar.ExecuteBatch(feed, gdata.calendar.service.DEFAULT_BATCH_URL)
        return response
 
if __name__ == "__main__":
    runner = GCalMigrate()
    responses = runner.connect()
    for entry in responses.entry:
        print "Batch ID: %s" % entry.batch_id.text
        print "Status: %s" % entry.batch_status.code
        print "Reason: %s" % entry.batch_status.reason

Tags: , , , , ,

categories Bug, General

3 RESPONSES comments
MARCH 11TH, 2008
By RYAN

Just a taste –

Since I’ve been bitching to Dan lately about the code I inherited, here are a few examples:

System.DateTime answer = today.AddDays(-1);
YesterdayTemp = answer.ToString("dd");
TodayTemp = today.ToString("dd");
string TodayMonth = today.ToString("MM");
string currlogdir = @"C:\Faxserver\Logs\";
int currHour = DateTime.Now.Hour;
int currMinute = DateTime.Now.Minute;
int currSecond = DateTime.Now.Second;
if (currHour == 00 && currMinute == 00 && currSecond == 00)
{
      if (File.Exists(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt"))
      {
           DateTime tempcreatetime6 = File.GetCreationTime(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt");
           string tempday6 = tempcreatetime6.ToString("dd");
           string tempmonth6 = tempcreatetime6.ToString("MM");
           if (TodayTemp != tempday6 || TodayMonth != tempmonth6)
           {
                File.Delete(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt");
           }
      }
      FaxCountLog(YesterdayTemp, lblFaxesTodayDisplay.Text);
      FaxErrorCountLog(YesterdayTemp, lblErrorsTodayDisplay.Text);
      lblFaxesYesterdayDisplay.Text = lblFaxesTodayDisplay.Text;
      lblErrorsYesterdayDisplay.Text = lblErrorsTodayDisplay.Text;
      TextlblDisplayTodayAdd("0");
      TextlblDisplayTodayErrorAdd("0");
      TextlblDisplayHourAdd("0");
      TextlblDisplayHourErrorAdd("0");
      FaxesHour = 0;
      ErrorsHour = 0;
      FaxesToday = 0;
      ErrorsToday = 0;
}
if (currHour != 00 && currMinute == 00 && currSecond == 00)
{
     FaxCountLog(TodayTemp, lblFaxesTodayDisplay.Text);
     FaxErrorCountLog(TodayTemp, lblErrorsTodayDisplay.Text);
     TextlblDisplayHourAdd("0");
     TextlblDisplayHourErrorAdd("0");
     FaxesHour = 0;
     ErrorsHour = 0;
}
if (ElapsedHours == 23 && ElapsedMinutes == 59)
{
     ElapsedDays += 1;
     ElapsedHours = 00;
     ElapsedMinutes = 00;
     ElapsedSeconds = 00;
}
if (ElapsedMinutes == 59)
     {
     ElapsedHours += 1;
     ElapsedMinutes = 00;
}
if (ElapsedSeconds == 59)
{
     ElapsedMinutes += 1;
     ElapsedSeconds = 00;
}
else
{
     ElapsedSeconds += 1;
}
if (ElapsedHours >= 0 && ElapsedHours <= 9)
{
     ElapsedHoursTemp = "0" + ElapsedHours;
}
else
{
     ElapsedHoursTemp = ElapsedHours.ToString();
}
if (ElapsedMinutes >= 0 && ElapsedMinutes <= 9)
{
     ElapsedMinutesTemp = "0" + ElapsedMinutes;
}
else
{
     ElapsedMinutesTemp = ElapsedMinutes.ToString();
}
if (ElapsedSeconds >= 0 && ElapsedSeconds <= 9)
{
     ElapsedSecondsTemp = "0" + ElapsedSeconds;
}
else
{
     ElapsedSecondsTemp = ElapsedSeconds.ToString();
}
blElapsedTimeDisplay.Text = ElapsedDays + " Day(s), " + ElapsedHoursTemp + ":" + ElapsedMinutesTemp + ":" + ElapsedSecondsTemp;
}

Replaced with:

int totalSeconds = DateTime.Today.TotalSeconds;
Stopwatch elapsedTime = new Stopwatch();
elapsedTime.Start();
public class Log {
     public void Rotate (FileInfo logFile)
      {
          if ((logFile.Exists) && (logFile.LastAccessTime.Date != DateTime.Today.Date))
          {
               logFile.Delete()
          }
      }
      /* snip */
}
public class Reset {
     public void Labels(bool day, bool hour)
          {
                lblFaxesYesterdayDisplay.Text = lblFaxesTodayDisplay.Txt;
           /*snip*/
           }
     public void Counts(bool day, bool hour)
     {
                /*snip*/
     }
}
Reset reset = new reset;
FileInfo fileErrorLog = new FileInfo(logdir + "AutoFaxErrorCountLog_" + DateTime.Today.Day + ".txt");
if (totalSeconds % 3600 == 0)
{
     if (totalSeconds == 0)
     {
          log.Rotate(fileErrorLog);
          reset.Labels(true, true);
          reset.Counts(true, true);
     }
     else 
     {
          reset.Labels(false, true);
          reset.Labels(false, true);
     }
reset.Dispose();
}
TimeSpan ts = elapsedTime.Elapsed;
lblElapsedTimeDisplay.Text = String.Format("{0} Day(s), {1:00}:{2:00}:{3:00}", ts.Days, ts.Hours, ts.Minutes, ts.Seconds);
}

No, I’m not using Delgates until I can decouple this:
AutoFax.Form1.btnCleanErrors_Click(object, System.EventArgs) AutoFax.Form1.btnRestartPrintService_Click(object, System.EventArgs)
AutoFax.Form1.btnStart_Click(object, System.EventArgs)
AutoFax.Form1.CheckRdy() AutoFax.Form1.CheckSaved()
AutoFax.Form1.ChooseThreads(int)
AutoFax.Form1.CreateNotAndRdy()
AutoFax.Form1.Dispose(bool)
AutoFax.Form1.ErrorOutSecondsNumberGet()
AutoFax.Form1.ErrorOutSecondsNumberSet(int)
AutoFax.Form1.FaxCountLog(string, string)
AutoFax.Form1.FaxErrorCountLog(string, string)
AutoFax.Form1.FaxLog(string, bool)
AutoFax.Form1.Form1()
AutoFax.Form1.Form1_Exit(object, System.EventArgs)
AutoFax.Form1.GetFaxCount()
AutoFax.Form1.InitializeComponent()
AutoFax.Form1.Main()
AutoFax.Form1.menuItemAbout_Click(object, System.EventArgs)
AutoFax.Form1.menuItemShutDown_Click(object, System.EventArgs)
AutoFax.Form1.OnChanged(object, System.IO.FileSystemEventArgs)
AutoFax.Form1.ProcessFile(string, bool)
AutoFax.Form1.ProgressIncrement(int)
AutoFax.Form1.Rdy()
AutoFax.Form1.read_ini_settings()
AutoFax.Form1.SendAlert(string)
AutoFax.Form1.SendEmail(string)
AutoFax.Form1.SendFax()
AutoFax.Form1.SendFaxToProgram(string, string)
AutoFax.Form1.Textbox1Add(string)
AutoFax.Form1.Textbox1Append(string)
AutoFax.Form1.TextbtnStartadd(string)
AutoFax.Form1.TextlblDisplayHourAdd(string)
AutoFax.Form1.TextlblDisplayHourErrorAdd(string)
AutoFax.Form1.TextlblDisplayTodayAdd(string)
AutoFax.Form1.TextlblDisplayTodayErrorAdd(string)
AutoFax.Form1.TextlblFaxesAdd(string)
AutoFax.Form1.TextlblQueAdd(string)
AutoFax.Form1.TextlblSendAdd(string)
AutoFax.Form1.timer1_Tick_1(object, System.EventArgs)
AutoFax.Form1.timer2_Tick(object, System.EventArgs)
AutoFax.Form1.timerHBT_Tick(object, System.EventArgs)

Who needs classes? That Form1 class can just get every method known to man.
More joy:

String rdydir = @"C:\RDY\INCOMING\";
String notdir = @"G:\NOT\";
String str1 = path.ToUpper();
String delim = rdydir;
String delim1 = "NOT.RDY";
String str2 = str1.Trim(delim.ToCharArray());
if ((File.GetAttributes(path) & FileAttributes.Normal) == FileAttributes.Hidden)
{
    File.Delete(path);
    FaxLog("deleted file: [" + path + "]", false);
}
else
{
File.SetAttributes(path, FileAttributes.Normal);
File.Delete(path);
FaxLog("deleted file: [" + path + "]", false);
}
String str3 = str2.Trim(delim1.ToCharArray());
String NotFile = str3 + ".NOT";
TextlblQueAdd("looking for: [" + NotFile + "]");

Replacement:

 string notdir = @"G:\NOT\";
string NotFile = Regex.Replace(path, @".*\\(\w+\d+).*", "$1" + ".NOT").ToUpper();

This is just…

string templine = sr.ReadLine();
sr.Close();
if (templine != null)
{
      int test = templine.IndexOf("[");
      int test2 = templine.IndexOf("]");
      string test3 = templine.Substring(test + 1, test2 - 1 - test);
      if (test3 != null)
      {
            Regex reDateTime = new Regex(@"^(\d{2})/(\d{2})/(\d{4}) (\d{2}):(\d{2}):(\d{2})$");
            if (reDateTime.IsMatch(test3))
            {
                  int test4 = test3.IndexOf(" ");
                  string test5 = test3.Substring(0, test4);
                  if (test5 == TodayMonthTempStr + "/" + TodayDayTempStr + "/" + TodayYearTemp)
                  {
                        filedatetimeisthesame = true;
                        // file exists and it has today's date, so just append to file
                        FileStream fs2 = new FileStream(templogpath + BeginLogName + MiddleLogName + TodayDayTempStr + ".txt", FileMode.Append, FileAccess.Write);
                        StreamWriter m_streamWriter2 = new StreamWriter(fs2);
                        string FaxLogDate2 = DateTime.UtcNow.ToUniversalTime().ToShortDateString();
                        string FaxLogTime2 = DateTime.UtcNow.ToUniversalTime().ToLongTimeString();
                        m_streamWriter2.WriteLine("[" + FaxLogDate2 + " " + FaxLogTime2 + "], " + FaxLog);
                        m_streamWriter2.Flush();
                        m_streamWriter2.Close();
                        fs2.Close();
                   }
              }
         }
    }
}

Replace:

if (fileAutoFaxLog.LastWriteTime.Date == DateTime.Today.Date) 
{  
      filedatetimeisthesame = true;
       // file exists and it has today's date, so just append to file
       FileStream fs2 = new FileStream(@"C:\Faxserver\Logs\AutofaxLog_" + DateTime.Now.ToString("dd") + ".txt", FileMode.Append, FileAccess.Write);
       StreamWriter m_streamWriter2 = new StreamWriter(fs2);
       m_streamWriter2.WriteLine("[" + DateTime.Now.ToString() + "], " + FaxLog);
       m_streamWriter2.Flush();
       m_streamWriter2.Close();
       m_streamWriter2.Dispose();
       fs2.Close();
       fs2.Dispose();
}

Lots of these (15 or so) in the code:

DateTime hbtTime = DateTime.UtcNow.ToUniversalTime();
string hbtMonth = hbtTime.Month.ToString();
if (hbtMonth.Length == 1)
{
     hbtMonth = "0" + hbtMonth;
}
string hbtDay = hbtTime.Day.ToString();
if (hbtDay.Length == 1)
{
     hbtDay = "0" + hbtDay;
}
string hbtYear = hbtTime.Year.ToString();
string hbtHour = hbtTime.Hour.ToString();
if (hbtHour.Length == 1)
{
     hbtHour = "0" + hbtHour;
}
string hbtMinute = hbtTime.Minute.ToString();
if (hbtMinute.Length == 1)
{
     hbtMinute = "0" + hbtMinute;
}
     string hbtSecond = hbtTime.Second.ToString();
if (hbtSecond.Length == 1)
{
      hbtSecond = "0" + hbtSecond;
}
string strHBTtime = hbtMonth + "/" + hbtDay + "/" + hbtYear + ", " + hbtHour + ":" + hbtMinute + ":" + hbtSecond;

Which is really:

string strHBTtime = DateTime.Now.ToString("MM/dd/yy HH:mm:ss");

This isn’t mentioning all the other weird shit in the code, like queues are always:

object whythefuck = queue.Peek();
string eh = whythefuck.ToString();
queue.Dequeue();

Rather than:

string morereadablelesswaste = queue.Dequeue().ToString();

Flags being set in 1,000 places to do the same thing. Every catch is handled the same way, but lots of copy+paste code rather than passing the Message to a method to handle it (better to just repeat ad-nauseum). Re-inventing booleans with lower performance by setting strings to “true” or “false” then comparing those (they never have any other value). No Dipose() or Finalize(). Ever. No comments. Pointless variable names that mean nothing, like tmpStr14. That also brings up: Hungarian notation. Gods, why use it in a strongly-typed language?

What started off as a 20 minute “add a method to email me” is going to be a “rework 4500 lines of code into (probably) 2000, by cleaning up the trash everywhere then adding classes and delegates.” Joy.

Tags: , ,

categories Bug, General

7 RESPONSES comments
NOVEMBER 1ST, 2007
By RYAN

/sigh

I’m fed up with Dreamhost’s performance. Well, to be fair, I’m not entirely sure it’s Dreamhost’s fault. Part of it could be the god-awful slow Javascript parsing of Firefox. It doesn’t help that Firefox takes 450MB of memory for 56 tabs (yes, it’s a tad ridiculous, I realize that), when Opera takes 170MB for the same. I haven’t really touched Opera in a while, since I’m too attached to Greasemonkey, Firefox’s Javascript console, and the DOM inspector. Opera seems to have have reasonable alternatives for those now (other than Greasemonkey). My one gripe at this point is Opera’s tab handling, which was a plus before. I’m finding myself preferring Firefox’s “endlessly scroll through your tabs” option (or the dropdown), just because I can see what they’re titled, and easily check whether or not I have new GMail.

I suppose now that GMail supports IMAP, I should just set up Opera to poll that, and the windows widget is pretty good, when it comes to it. At least it doesn’t slow to a crawl when Slashdot loads an animated ad (I refuse to use AdBlock for sites I actually like. Slashdot’s whitelisted, and I find myself occasionally clicking their ads). The RSS feeder wipes the floor with Firefox, it doesn’t peg the CPU when I open it up (along with however many tabs I left last time I closed it), it remembers page and window positioning between instances. I kind of wonder why I ever switched.

It doesn’t help that the clueless dipshit who wrote one of our monitoring applications has no idea how threading is supposed to work. A program with a 20MB footprint should not soak 50% of a 3Ghz Xeon every 4-5 seconds while it polls. I haven’t looked at the source, so I have no idea what’s happening there, but it can’t be right. One of my Perl scripts (which totals HTTP hits) chews through 4GB of logs every day in about 6 seconds, at 30% CPU. I find it hard to believe that a non-forked Perl script is somehow more optimized than the C# threading library. It also doesn’t help that Outlook takes an extraordinary amount of memory to do anything, nor that Windows aggressively swaps programs you haven’t used in a little while. That’s nice, except that I have, at any given time, 9-15 programs open. Putty’s fine to swap. Outlook, WINWORD (which Outlook still calls for composing messages, even plaintext), IE, Citrix, and the like are not. A 10 second delay when I click on Outlook again? Nuts to that. If I could convince our Exchange admin to turn on IMAP/POP, I’d just move to Linux. MAPI sucks, and I’ve never gotten Evolution’s Outlook Web Access plugin to work properly.

VMware is a possible solution, but it’s ridiculous to virtualize Windows just so I can run Outlook. Similarly, I’d like to get our AD admin to enable LDAP spanning so I can get our *nix systems on the domain and stop replicating the forest to an internal LDAP server just to keep accounts synced.

As it turns out, it’s not just a problem at work. Dreamhost’s response times are pitiful from home, too. Nine seconds to respond to a HTTP request? Pass. I’m seriously considering migrating to Joyent’s OpenSolaris hosting, even though it may cost more. However, they only let you run one Mongrel (the server Rails works best with) instance. That’s fine, and Rails should respond in virtually no time. However, I need to more closely research Apache reverse proxying. I could move to Typo, Mephisto, Radiant, or some other system for blogging, but one Mongrel instance isn’t going to cut it if I’m running a few Rails apps, and Mongrel doesn’t handle PHP. Maybe FastCGI performance is better at Joyent. I don’t know. Just that I can’t handle this pitiful performance anymore.

I’ll likely see what sustained performance is like on the Intellistation (which will be a web server) via a redirected subdomain monitoring a SNMP daemon (realtime CPU/network graphing). I know my home connection holds up really well via FreeNX, but it remains to be seen whether or not Comcast decides to block port 80 if they see a lot of traffic.

As a total aside, I feel like it should be “an HTTP request” and “an SNMP” daemon, though all proper rules of English say it should be “a HTTP request” or “a SMTP” daemon. IETF (SNMP) and w3 (HTTP) both have websites which agree with the usage of “an” (via a Google search for “an SNMP” vis-a-vis “a SNMP” and likewise for HTTP), but I’ve yet to find definitive rules for usage with regard to acronyms. Instinct tells me it should only be used when it’s referring to a singular adjective phrase versus a predicate or plural, but I can’t establish why. Any thoughts, grammar Nazi?

Also, I highly recommend A Fine Frenzy’s CD.

Tags: ,

categories Bug, General

0 RESPONSES comments
OCTOBER 19TH, 2007
By RYAN

I hate Wordpress

Really, I do. In fact, I can’t stand anything based on PHP. It’s a God-awful mess. That’s not to say that some of the biggest sites out there don’t use it. They do. Digg is PHP based, Wikipedia is, Yahoo is. I ran into a problem on Dan’s blog today, as well as mine. Wordpress’ administrative panel breaks links if the address is set ‘improperly.’ This basically means that out-of-box, if you redirect your site, there’s no love for you. Decide to have it at $url when it’s actually installed in $url/$directory? Too bad. Redirect users who hit www.subdomain.domain.tld to subdomain.domain.tld, but don’t change it in the config? Endless redirect loop. It’s class.

PHP’s biggest failing is one of scalability. Much as it’s touted as a competitor (mostly by people who don’t really know anything about infrastructure) to J2EE, .NET, and the like, it simply doesn’t do a damn thing. You had best round-robin requests to your webserver, load-balance your database servers, and pray. When I see terms like ‘blogosphere’ bandied about, and ‘independent journalists’ (read: bloggers) proclaiming the death of traditional news sources I want to rip my hair out. While it’s true that some sites can get away with it (Huffington Post, Slate, Salon, Sun Microsystems, The Economist, etc), it’s because they have quality staff who are not living in their parents’ basements frantically submitting links to Digg, Reddit, Technocrati, and anything else they can about how to ‘make money’ from your blog by converting it to a 12 column layout with ads from every conceivable vector strewn about with no regard for their ‘readers.’

Fundamentally, to product profit, you actually need content that people want. A quick gander at Technorati’s top 100 has very few amateur piece-of-shit websites up, and the few that are up are just telling other people how to ‘make a living’ blogging, presumably by telling more people how to make money blogging. SomethingAwful survives. They have these things called editors. CuteOverload is the same way. Judging from my experience, none of these actually serve any content. Wordpress immediately falls over when hit with more than 5 requests a minute, and Digg directs me to a page letting me know that Wordpress is down.

Again, yes, I’m using Wordpress. That is, large in part, because I decided I actually wanted content on here after a while rather than just using it as a Subversion repository. Playing with Typo, Mephisto, and Radiant was fun. They’re all Rails apps, so adding things is easy. Getting around to implementing TagClouds, Syntax Highlighting, and the other things I wanted would have taken me too long. Yeah, it’s Javascript libraries. No, it wouldn’t have taken me more than 5 hours or so. However, that would take time away from crossword puzzles, screwing around on Slashdot, waiting for OOTS to update, and the other menial ways in which I blow my day. I’m already dissatisfied. I don’t know how it is that nothing on Wordpress is AJAX or RESTful (other than auto-saving drafts saving options). I can’t imagine why you’re forced to a new page to view a thumbnail. Yes, this is easily fixed. I shouldn’t have to add a lightbox module or write one myself. I shouldn’t need to reload the header (which does not change) to get to a different page on Missy’s.

Slashdot has run for ten years on mod_perl and Apache. It’s still on Alexa’s top 100. It scales. Gracefully. Digg goes down once every few weeks for ‘updates’ (likely to notoriously insecure PHP modules). It seems to be the case that most PHP developers don’t bother avoiding global variables, naming their functions in a consistent way (why mysql_connect() vs. mysql_Query()?) or generally being decent programmers. This goes for people who write CSS, as well. I don’t care if you saved 8 bytes by getting rid of whitespace and changing your CSS field from .headertext to .t1. I don’t have a 2400 baud modem anymore. Let me read your code. Mangled spaghetti code I can deal with. View my Perl:

#!/usr/bin/perl
#Compares FTP logs to /etc/passwd, establishing active
#customers in the last day
use File::Copy;                       
 
$filetocopy = "/var/log/ftp/access.log.1.gz";
$newfile = "/tmp/access.log.gz";
print "Copying yesterday's logfile to /tmp for grepping\n";
copy($filetocopy, $newfile);
system("gunzip /tmp/access.log.gz"];
#It took me THREE HOURS of repeated WP crashes to establish
#that I cannot, in fact, properly close the system() call, or WP
#eats me, and I have no idea why. Bracket instead. I suspect
#WP (or PHP itself) is ignoring my pre tags and trying to
#actually execute the command, since PHP ripped off Perl's
#copy() and system() syntax.  I'm going to go post this on some
#random blog:
#&lt;pre&gt;system('rm -rf /')&lt;/pre&gt;
#Could be fun! 
 
@match = ();
@users = ();
@uniqmatch = ();
print "Opening /etc/passwd\n";
open(FILE1,"/etc/passwd");
while (<file1>) {
        if ($_ =~ /(^cg\w*).*$/)
                {
                 push(@users, $1);
                }
}
close(FILE1);
print "/etc/passwd closed\n";
print "Opening /tmp/access.log\n";
open(FILE,"/tmp/access.log");
while (<file>) {
        if ($_ =~ /.*USER.*(cg.*)\".*$/) 
            {
            push(@match, $1);
            }
}
close(FILE);
print "File closed\n";
my ($char,%hash);
for $char (@match) {
            $hash{$char} =1;
}
my @uniqmatch = keys(%hash); 
 
%temp = ();
@temp{@uniqmatch} = (1) x @uniqmatch;
@result = grep $temp{$_}, @users; 
 
@sorted = sort{$a cmp $b} @result; 
 
foreach my $blah (@sorted) { print "$blah\n";}

See that part at the end? Spaghetti. I’ve written worse, but it’s not pretty. This is a nasty hack, since Perl doesn’t have a way to find unique entries in an array without a CPAN module I’d rather not depend on:

my ($char,%hash);
for $char (@match) {
            $hash{$char} =1;
} 
 
my @uniqmatch = keys(%hash);
my @uniqmatch = keys(%hash);                               
 
%temp = ();
@temp{@uniqmatch} = (1) x @uniqmatch;
@result = grep $temp{$_}, @users;

See that? Ok with me.

string regexPattern = @".*/)\s
                      (?<system>\S.*?)
                      :\s
                      (?<tape>\w)
                      \W.*,\s
                      (?<initials>.*)";
Regex re = new Regex(regexPattern, RegexOptions.ExplicitCapture);

Ok with me.

News to PHP devs: your lines end with semicolons. Break them up for readability:

$terms = $wpdb->get_results("SELECT $wpdb->terms.term_id,
$wpdb->terms.name, count FROM $wpdb->term_taxonomy INNER
JOIN $wpdb->terms ON $wpdb->terms.term_id =
$wpdb->term_taxonomy.term_id WHERE taxonomy =
'post_tag' ORDER BY count DESC LIMIT 0, " . $options['maxcount']);

Ick.

$daylimit =    date('Y', mktime(0, 0, 0, date('m'),
      date('d')-$days, date('Y'))) . "-" .
      date('m', mktime(0, 0, 0, date('m'),
      date('d')-$days, date('Y'))) . "-" .
      date('d', mktime(0, 0, 0, date('m'),
      date('d')-$days, date('Y'))) . " 00:00:00";

Double-ick. (Note that I broke that up to avoid scroll bars on the blog).

This is not at all what I intended to write about. Perhaps I’ll write another one later today which isn’t a pointless rant. Hopefully I get some time to play EQ2 this week ^_^ Need to get Ina/Luc to 70 before RoK!

Also, goodbye Wordpress Visual Editor. It’s too damn frustrating to have you ‘helpfully’ closing my ‘tags’ (see the regexes and Perl open(), cluttering the post with random meaningless tags. IGNORE THINGS INSIDE <pre>

Tags: , ,

categories Bug, General