Posts tagged: WTF

Dear internet, please shut up

Honestly, the fervor over Iran? Enough. While it’s certainly an important event, it merely serves to illustrate that the people of Iran have a nominal choice over their president. That is all. There’s some feeling of solidarity with Iran’s youth (about 30% of the populace) in the US, but people seem to have deluded themselves into thinking it’s another Iranian Revolution or a harbinger of Western-style democracy. First, I guess, a little background on Iran’s government structure:

  • Supreme Leader:
  • Controls everything, really, as we’ll see in a bit. Personally has the power to declare war and peace, command army/intelligence/police forces.

  • Assembly of Experts:
  • Clerics who determine who the next Supreme Leader will be, though they capitulate to his desires in every recorded instance(see: Montazeri). Elected to 8 year terms by direct public vote, but the list of candidates is screened anyway, so it’s not like a “free election.”

  • Guardian Council:
  • Easily the most powerful force in Iranian politics. Twelve members, 6 picked directly by the Supreme Leader, 6 picked by the head of the judiciary (who is also picked by the Supreme Leader). Confirmed by Parliament, but it’s mostly for show. Functions as a combination Supreme Court and presidential veto (except Parliament cannot actually override their veto, it just gets sent back for revision). Screens candidates for every imaginable branch of government and arbitrarily excludes them (this has been happening for a while to keep reform candidates out — the 2004, 2006, and 2008 electiosn were particularly bad). May as well have total control of the Revolutionary Guard (which is not a branch of the normal army).

  • Expendiency Council:
  • Nominally there to function as a screen between the Guardian Council and parliament. In reality, just advises the Supreme Leader

  • Majlis (parliament):
  • Does the things you’d normally expect parliament to do (approve budgets, treaties, drafts legislation), only neutered by the Guardian Council. As with everything else, candidates are screened.

  • President/VPs/Ministries
  • Executive branch stuff — day-to-day running of the state. Nominally right below the Supreme Leader. In truth, Guardian Council wields more power. Candidates are elected, but are screened by Guardian Council beforehand, ensuring real reformers never even make it to the vote. Best described as a figurehead.

There is very little debate in Iran about the importance of religion. They’re not going to turn into a secular nation. There’s hardly any protest (even now) against the Supreme Leader, though some resentment against the Guardian Council for the behavior of the Basij. No matter how the protests go, it’s unlikely to be another revolution — very few revolutions succeed without support from another sovereign nation, which they are not getting. The nature of the screening pretty much ensures that it’ll be another conservative candidate even if it does succeed, but Ahmadinejad and the Guardian Council control the complete apparatus of the state. It’s likely to be brutally repressed (granted, I’ve seen very little from outside Tehran, which I consider to be an indication that it’s relatively calm there).

Mousavi’s campaign? Move control of the police from Khamenei to the president. Make government a little more transparent. Allow non-state run television. Not exactly Kucinich here, or even Obama. Maybe Ron Paul is an apt comparison.

Karroubi? Actual reform! Freedom of the press, women’s rights, nationalize the oil industry and distribute the profits to the people, etc. As you’ve maybe seen, he was not the winner.

With that out of the way (and the background may have been necessary for those thinking about Iran becoming South Korea or post-imperial Japan), can we stop the damn Twittering? Changing your timezone to Tehran? Ahh… I’m not sure what you think that’s accomplishing. Under the assumption that the Iranian government can track individual people Twittering (and really? They’ve got more important things to worry about — just that people don’t post updates after going to rallies), it’s because they control the feeds out of the country. Your timezone isn’t fooling them, unless you’re routing it through Iran, which is doubtful. Open proxies? Let’s say they can track Twitterers — that means they’re using Deep Packet Inspection and, well, proxies don’t exactly help against that. They get around blackholing IP blocks (Facebook/whatever), and the headers change a little, but the packet contents are still eminently sniffable.

Tor exit nodes are a better option, but there’s the possibility the Iranian government is running an exit node themselves as a honeypot (extremely unlikely). VPNs are much better, and not all that hard to set up, as long as you keep the address of the server relatively private. For the sake of argument, we’ll say OpenVPN — which is SSL based — running on port 443. It should be utterly indistinguishable from HTTPS traffic, since the key negotiation is also encrypted, and DPI isn’t going to help them here. For all they know, people would be conducting internet banking or one of the million other things SSL is used for.

Of course, doing this requires more effort than changing your Twitter timezone or giving yourself a green background for “solidarity.” Worse than that, the technical issues should have been obvious to a lot of people by now (or at least the contradiction in assuming they can track what IP addresses are using Twitter to individuals, yet thinking that packets which are never routed through Iran will somehow fool them). The irony of the “solidarity” is that our elections in 2000 and 2004 were filled with questionable voter purges, yet we didn’t rise up. The candidate they elected (assuming the leaked results are true) is more comparable to Reagan or one of the Bushes than Johnson or Carter — a man who the people in support of “#iranelection” would never, ever vote for, but they don’t realize that as he’s a “reform” candidate. We arrested twice as many journalists in the 4 days of the RNC in Saint Paul than Iran has in a week of street riots.

Until you can drop the hypocrisy, technical incompetence (and don’t think most of the people wanted to think about that, since they might have to stop patting themselves on the back for “helping Iranians” with a fucking website that has the same limitations as SMS), and utter lack of understanding about Iran’s politics (ahem, even if Mousavi is inducted, not much is likely to change in Iran until the Guardian Council stops culling candidates who want actual change), you can shut the fuck up and keep your self-righteousness to yourself.

XMonad and xmobar on OpenSolaris with functional monitoring (mutt to boot)

I’ve been having carpal tunnel flareups lately, so I went looking around for things I could do to use the mouse less and the keyboard more (as counter-intuitive as that may sound, I find that holding a mouse for hours irritates it far more than any amount of typing, YMMV). Vimperator is an obvious first step, but, well, I love vi, so I already had that running.

Tiling window managers came to mind. I’ve already used wmii and dwm once upon a time, but they’re hardly state of the art (as state of the art as tiling window managers get, anyway), and hacking together a reasonable workspace status bar in dzen/Perl didn’t appeal to me. Awesome3 (the window manager) does appeal to me, but getting it running on Solaris looked like a little more work than I wanted to invest, and I’m sick of working with moving targets (Awesome3 looks like they’re almost as “break your config” happy as Rails).

Mind you, I still love Openbox, but given that 99% of my time is spent in terminals (irssi, mutt, ssh, vim, mcabber, slash’em), I see no reason why I should even bother with having window decorations and manually arranging them at all.

Had I known what I was getting into, I probably would have just used Awesome. I mean, it needed two libraries I didn’t have, and some dzen hacking. Not… this. Not that I’m unhappy with XMonad, but…

Firstly, there’s no build of GHC (Glaskow Haskell Compiler) in the OpenSolaris repositories. There’s a pre-compiled version of GHC 6.10, but only for SPARC. Pre-compiled version of GHC 6.8 for x86/amd64, but that ain’t helping me (a scary amount of stuff from Hackage, Haskell’s version of CPAN/rubygems/whathaveyou doesn’t want to run in GHC 6.8, and the recommended fix for some bugs is “upgrade to 6.10″).

No GCC 4 in the repos either, and GCC 4.1.2 is the recommended version for building GHC. So, onto the magic. Don’t even try with SunStudio. GNU-isms in the code stopped me dead.

I was bitching to Dan about how ridiculous this process was a few weeks ago. Maybe it’ll help somebody.

Firstly, either install readline from the OpenSolaris Pending repositories or compile it yourself.

Next, we need to bootstrap gcc. For that, we’ll need gmp from GNU and mpfr. Grab the precompiled version of GHC 6.8.2 while you’re at it. You should also get the newest versions of ghc and ghc-$version and ghc-$version-src-extralibs to get running later.

Complaint #1: GNU automake is braindead. It, I assume, just checks `uname` and not `isainfo`, so I can’t tell when we’re running 64 bit. Either use Solaris libtools or do the following for gmp and MPFR

./configure ABI=32 && gmake && pfexec gmake install
make distclean
configure --prefix=/usr/local/lib64 && gmake && pfexec gmake install

If you don’t specify another prefix, it’ll stomp all over the 32 bit libraries it just installed on your 64 bit box.

Complain #2: gcc is even more braindead. It’ll build a 64 bit binary but link it against 32 bit libraries, then eat itself during stage 2 bootstrap. You’d think the FSF would be smarter, but no. It just finds the wrong ELFCLASS down the line. To correct:

export LDFLAGS=-L/opt/local/lib -L/opt/local/lib/64 \
-R/usr/local/lib:/usr/local/lib/64
export LD_OPTIONS=-L/opt/local/lib -L/opt/local/lib/64 \
-R/usr/local/lib:/usr/local/lib/64
./configure && gmake -j4 && pfexec gmake install

Complaint #3: The precompiled ghc-6.8.2 we got? It sucks. The rts library is broken (check it with ldd). It would be nice to avoid this, but, well… Bootstrapping ghc from C sources and no Haskell compiler involves another goddamn system which DOES have Haskell installed, AND knowledge of what registers your CPU uses. Whoever thought up that notion of bootstrapping? Well… The “goal” is to have Haskell self-bootstrap (it currently does not, since, ironically, Haskell is too “pure” to actually be written in a “pure” language, since we need dirty things like actually doing something useful with “impure” data, like user input or stuff sucked in from a file).

cd ../ghc-6.8.2
./configure && pfexec gmake install
ghc-pkg describe rts > rts.pkg
vim rts.pkg
#add -R/usr/local/lib to the end of the ldoptions field, or ghc bombs bootstrapping the new version in stage 2
ghc-pkg update rts.pkg

Complaint #4: GHC is even stupider than GCC, if possible. Not only do we have to prepend /usr/local/bin to $PATH so GHC can find our shiny new gcc-4.1.2, we have to pass ridiculous amounts of config flags (including one which tells it where GCC is — if the $PATH OR –with-gcc is wrong, it won’t bootstrap. Don’t ask, because I don’t know why).

export PATH=/usr/local/bin:$PATH
./configure --with-gcc=/usr/local/bin/gcc --with-gmp-libraries=/usr/local/lib --with-gmp-include=/usr/local/include --with-readline-libraries=/usr/local/bin --with-readline-include=/usr/local/include
gmake -j4 && pfexec gmake install

Yay! Working GHC. Sadly, if you want to reclaim the 350MB or so the GHC 6.8 install is taking up, you have to go remove it yourself (apparently the GHC team does not believe in `make uninstall`). This means we can install cabal, which requires nothing special, other than you grabbing the tarball and installing it as normal.

Next, `cabal install xmonad xmonad-contrib` I said we were going to install xmobar, and we are, but it’s a little tricker. You see, even though Xmobar mostly reads a pipe to give us a title and workspace listing, the plugins are not optional. They also depend on libnotify, which is only present on Linux. Good job, xmobar developer! Fortunately, this is easily corrected, and xmobar (mostly) works. Caveats explained later.

You can’t just `cabal install` xmobar. No-go since hinotify will not install, and there’s not a clear explanation as to why from the output. As noted, it doesn’t really depend on it, just that the developer can’t be bothered to use Haskell’s typing system to throw messages at you when you try to use features that are not implemented. So… edit ~/.cabal/packages/hackage.haskell.org/xmobar/$version/xmobar-$version/xmonad.cabal

Take out the lines referring to hinotify. Then `cabal build && cabal install` from the directory xmonad.cabal was in. Ooh and aah, but don’t try to use, well… anything. BatteryReader, CpuReader, MemReader, Net, Swap, all broken. Thankfully, we have Dtrace and Python to replace it with, since xmobar’s PipeReader still works.

Memory usage?

#pragma D option quiet
#pragma D option bufsize=16k
 
dtrace:::BEGIN
{
}
 
profile:::tick-1sec
{
	/* RAM stats */
	this->ram_total = `physinstalled;
	this->unusable  = `physinstalled - `physmem;
	this->locked    = `pages_locked;
	this->ram_used  = `availrmem - `freemem;
	this->freemem   = `freemem;
	this->kernel    = `physmem - `pages_locked - `availrmem;
 
	this->ram_total	*= `_pagesize;  this->ram_total	/= 1048576;
	this->unusable	*= `_pagesize;  this->unusable	/= 1048576;
	this->kernel	*= `_pagesize;  this->kernel	/= 1048576;
	this->locked	*= `_pagesize;  this->locked	/= 1048576;
	this->ram_used	*= `_pagesize;  this->ram_used	/= 1048576;
	this->freemem	*= `_pagesize;  this->freemem	/= 1048576;
	printf("RAM: %2d%%\n", ((this->ram_total - this->freemem) * 100 / this->ram_total));
}

Network speeds?

#!/usr/sbin/dtrace -s
#pragma D option quiet
dtrace:::BEGIN
{
	TCP_out = 0; TCP_in = 0;
}
 
 
mib:::tcpOutDataBytes		{ TCP_out += arg0;   }
mib:::tcpInDataInorderBytes	{ TCP_in += arg0;    }
 
profile:::tick-1sec
{
	OUT_print = TCP_out/1024; IN_print = TCP_in/1024;
	printf("Out:%3d|In:%3d", OUT_print, IN_print);
	TCP_out = 0;
	TCP_in = 0;
 
}

.xmobarrc

Config { font = "-*-terminus-*-*-*-*-12-*-*-*-*-*-*-u"
       , bgColor = "#000000"
       , fgColor = "#AFAF87"
       , position = Top 
       , lowerOnStart = True
       , commands = [ Run Date "%a %b %_d %Y %H:%M:%S" "date" 10 
                    , Run Weather "KSTP" ["-t","<tempF>F","-L","64","-H","77","--normal","green","--high","red","--low","lightblue"] 36000
		    , Run PipeReader "/export/home/ryan/dtrace/net" "wireless"
		    , Run PipeReader "/export/home/ryan/dtrace/netspeed" "speed"
		    , Run PipeReader "/export/home/ryan/dtrace/psr" "cpui"
		    , Run PipeReader "/export/home/ryan/dtrace/ram" "mem"
                    , Run StdinReader
                    ]
       , sepChar = "%"
       , alignSep = "}{"
       , template = " %StdinReader% } { %cpui% | %mem% | %wireless% %speed% | %date% | %KSTP%"
       }

A script to feed those pipes. If you don’t have python2.6, pexpect on python2.4 (the Solaris/OpenSolaris default) works. Just install pexpect with easy_install, an .egg, or whatever your poison may be.

#!/usr/bin/python2.6
import math
import os
import platform
import re
import stat
import sys
import time
 
import pexpect
 
#Get the directory we're running from to create the fifos rather than the $pwd of whatever called us
path = os.path.dirname( os.path.realpath(__file__)) + "/"
 
wificonfig = ""
 
#Y'know, I haven't actually written the iwconfig thing.  It's here for posterity and possible later use.
osystem = platform.system()
if osystem == 'SunOS':
  wificonfig = 'wificonfig'
elif osystem == 'Linux':
  wificonfig = 'iwconfig'
 
 
#Kill off any old instances which may be running.  Poor man's pkill, but guaranteed to work pretty much anywhere.
pexpect.run('bash -c "ps -ef |grep mpstat |grep -v python| awk \'{print $2}\' | xargs kill -9"')
pexpect.run('bash -c "ps -ef | grep speed.d | awk \'{print $2}\' | xargs kill -9')
pexpect.run('bash -c "ps -ef | grep meminfo.d | awk \'{print $2}\' | xargs kill -9')
 
def checkfifo(path):
  #If it ain't there, make it
  if not os.path.exists(path):
    os.mkfifo(path)
    handle = open(path, "r+")
    return handle
  #If it is, just return it
  elif stat.S_ISFIFO(os.stat(path).st_mode):
    handle = open(path, "r+")
    return handle
  else:
    if os.path.isfile(path):
      #Not a FIFO, and it needs to be
      os.unlink(path)
    os.mkfifo(path)
    handle = open(path, "r+")
    return handle
 
#Set up our fifos
psrfifo = checkfifo(path + "psr")
netfifo = checkfifo(path + "net")
nspdfifo = checkfifo(path + "netspeed")
ramfifo = checkfifo(path + "ram")
 
#Fire off the processes we'll be reading from.  Using pexpect seems like overkill, but mpstat is apparently smart enough to tell when it's being read from a pipe, and it'll buffer no matter what you do.  pexpect/expect fake being interactive, so it happily runs without buffering.
mpstat = pexpect.spawn('bash -c "mpstat 1 | grep -v CPU"')
ramstats = pexpect.spawn(path + "meminfo.d")
nspeed = pexpect.spawn(path + "speed.d")
 
#Regular expressions to use later.  Since it's a long-runnign script, they may as well be compiled
mpre = re.compile(r'^\s+?(?P<cpu>\d+).*?(?P<idle>\d+)$')
solwifire = re.compile(r'.*?linkstatus: (?P<status>\w+).*essid: (?P<essid>\w+).*strength: \w+\((?P<strength>\d+)\).*', re.DOTALL)
 
def prstat():
  #Yay for awk/sed abuse, but it's concise and I'm already forking.  Basically getting a list of CPUs to check later, so this script should perform its duty no matter if you have 1 CPU or 128 (T2 users)
  psrinfo = pexpect.run('bash -c "psrinfo -v |grep MHz | awk \'{print $6,$7}\' | sed -e \'s/,//\'"').rstrip().split('\r\n')
  output = ""
  for cpu in psrinfo:
    line = prmatch(cpu)
    output = output + line
  psrfifo.write(output + "\n")
  psrfifo.flush()
 
def prmatch(cpu):
  line = mpstat.readline().rstrip()
  m = mpre.match(line)
  #Match it against our earlier regex and subtract the idle value from 100 to get the actual used percentage, which isn't wholly accurate (IOWAIT and whatnot), but it's good enough for me
  usage = 100 - int(m.group('idle'))
  #Padding the string seems stupid, and it is, but xmobar arbitrarily decides spots that it's not going to refresh even if text shows up there, leaving it (black in my case) when text slides.  Padding fixes that.  Also, if your CPU goes to 100%, you probably shouldn't have a script which reads dtrace probes running.  Just sayin'.
  return "Cpu%s: %2d%% (%s) " % (m.group('cpu'), usage, cpu)
 
def memory():
  #I haven't found any swap information from dtrace probes as easy to manipulate as thi sis
  swap = pexpect.run('bash -c "/usr/sbin/swap -l |tail -n 1 | awk \'{print $4, $5}\'"').rstrip().split(' ')
  #Ugly?  You bet.  Cast the subtract free swap from total swap blocks, divide it by free swap blocks * 100 cast to an int to give us an actual percentage, floor that, then cast THAT to an int
  usedswap = int(math.floor(((int(swap[1])-int(swap[0]))/int(swap[1])*100)))
  ram = ramstats.readline().rstrip()
  output =  "%s Swap: %2d%%" % (ram, usedswap)
  ramfifo.write(output + "\n")
  ramfifo.flush()
 
def network():
  #Filter out interface which aren't up, which are vnics, which only point to localhost to get the running interface.  I'm assuming you only have one at a time, but if you have more modify this to suit.
  iface = pexpect.run('bash -c "ifconfig -a | grep UP |grep RUNNING| grep -v IPv6 |grep -v lo | grep -v -E \':[0-9]: \' | awk \'{print $1}\' | sed -e \'s/://\'i"').rstrip()
  if osystem == 'SunOS':
    command = "wificonfig -i " + iface + " showstatus"
    status = pexpect.run(command).rstrip()
    output = ""
    if solwifire.match(status):
      #Beauty of regexes.  If it doesn't match, it's not wireless (or not connected).  It if is, give us values.
      m = solwifire.match(status)
      strength = math.floor((int(m.group('strength')) / 15.) * 100)
      output =  "%s: %s(%s) %3d%%" % (iface, m.group('status'), m.group('essid'), strength)
    else:
      #Probably not wireless
      output = iface + ":"
    netfifo.write(output + "\n") 
    netfifo.flush()
 
def netspeed():
  #Is this method really necessary?  Couldn't the dtrace probe just write to the fifo itself?  Probably, but if you (or I) want to colorize it at some point, it may as well get sucked in here.
  speed = nspeed.readline()
  nspdfifo.write(speed + "\n")
  nspdfifo.flush()
 
while 1:
 
  prstat()
  memory()
  network()
  netspeed()
  time.sleep(1)

It’s not the prettiest python. I should probably move those repetitive fifo flushes/etc to a method, but I didn’t honestly expect that I’d need to replace this much XMobar functionality. Notably, XMobar can colorize things with <span> attributes setting colors in case somebody wanted to pretty up the usages (really, to make it look more like XMobar’s [colorized] defaults for CPU/net usage). I don’t care, personally. I didn’t implement a battery monitor either, but hey, you can if you want to.

Complaint #5 (did I lose track?): SUNWmutt doesn’t have support for header caching, which is a real bitch when I have 12,000 emails. It also doesn’t support SMTP over SSL, making it pretty well worthless for Gmail. I have other accounts I use mutt for, but Gmail’s an important one.

This isn’t that tough, really. You need some kind of a database for the mutt config script, and gdbm is trivially easy to get running (normal ./configure && make && make install). On the other hand, we run into two hiccups. The configure file depends on ncurses, which is just a link to plain ol’ curses on lots of Solaris boxes. Secondly, (and I don’t really begrudge the Mutt guys for this, since there’s actually a commit to fix this, unlike GCC and GHC, whose response is “too fucking bad” [GHC actually posts the recommendation for fixing rts.pkg and the configure flags on their own site rather than FIXING THE BUILD]), configure.ac does some things wrong with libidn.

Find $with_idn, and replace the block which follows it with this (–with-idn doesn’t seem to build properly).:

if test "$with_idn" != "no" ; then
  if test "$with_idn" != "yes" ; then
    AC_CHECK_HEADERS([idn/idn-int.h],
      [AC_CHECK_HEADERS([idn/idna.h], [],
        [CPPFLAGS="$CPPFLAGS -I/usr/include/idn"])])
  fi
fi

If you don’t have or want ncurses (or it’s a symlink on your system), fix configure. `sed -i -e ’s/-lncurses/-lcurses/’ configure`.

./configure –with-regex –with-gnutls –enable-hcache –enable-smtp –enable-imap –enable-pop –enable-mailtool –with-sasl –with-idn=/usr/include/idn

Congratulations!

Next up, re-implementing htop for Solaris with dtrace probes, python, and ncurses.

Palm Desktop, I stab at thee!

Firstly, I’m starting P90X tomorrow. Should be interesting. Secondly, I miss you guys :/ I’m living with somebody who asked me what the Dead Sea Scrolls are this morning, since it was on the news that they’re coming to the Science Museum.

By the way, ever planning on touching your blogs again (Sewpbox and Rattributes not included)?

So I’m migrating Heather’s Palm Desktop crap to Google Calendar (I have no idea why no tool exists to do this). Google Calendar doesn’t really like the CSV I massaged out of it (only importing about half the records), and I’m starting to see why. Half the records are fucking duplicates in every way but one. I wrote a Python script to do it for me anyway.

The long and short of it amounts to this:
If you want the easy way, export the Palm data to a .mda, import it into Yahoo Calendar, then into Google Calendar from there. Otherwise, export it to a CSV, and hit it with this script:

#!/usr/bin/ruby
#
require 'csv'
 
input = "export.csv"
output = "gcal.csv"
 
csvfile = File.open(input) {|f| f.read}
 
puts "Parsing..."
 
csv = CSV::parse(csvfile)
 
fields = csv.shift
 
puts "Writing..."
File.open(output, "w") do |f|
   f.print "Subject, Start Date, Start Time, End Date, End Time\n"
   csv.each do |line|
     startdate, starttime = Time.at(line[6].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     enddate, endtime = Time.at(line[7].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     f.print "\"#{line[11]}\",#{startdate},#{starttime},#{enddate},#{endtime}\n"
   end
end
 
puts "Done."

If you don’t feel like exporting, and are running on Windows:

#!/usr/bin/ruby
#
#
require 'win32ole'
require 'dbi'
 
class Access
   attr_accessor :mdb, :conn, :data, :fields
 
   def initialize(mdb=nil)
       @mdb = mdb
       @conn = nil
       @data = nil
       @fields = nil
   end
 
   def open
       connstring = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=#{@mdb}"
       @conn = WIN32OLE.new('ADODB.Connection')
       @conn.Open(connstring)
   end
 
   def query(sql)
       set = WIN32OLE.new('ADODB.Recordset')
       set.Open(sql, @conn)
       @fields = []
       set.Fields.each do |field|
           @fields << field.Name
       end
       @data = set.GetRows.transpose
       set.Close
   end
 
   def close
       @conn.Close
   end
end
 
output = "gcal.csv"
 
rows = Array.new
 
db = Access.new('c:\path\to\mdb')
db.open
 
db.query("SELECT * FROM Main;")
names = db.fields
rows = db.data
 
#Alternatively
DBI.connect("DBI:ODBC:driver=Microsoft Access Driver (*.mdb);"+"dbq=c:/path/to/mdb") do |dbh|
   dbh.select_all('select * from Main') {|row| rows << row}
end
 
puts "Writing..."
File.open(output, "w") do |f|
   f.print "Subject, Start Date, Start Time, End Date, End Time\n"
   rows.each do |line|
     startdate, starttime = Time.at(line[6].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     enddate, endtime = Time.at(line[7].to_i).strftime("%m/%d/%Y,%I:%M:%S %p").split(',')
     f.print "\"#{line[11]}\",#{startdate},#{starttime},#{enddate},#{endtime}\n"
   end
end
 
puts "Done."

If you want the details…

Essentially, Palm’s Datebook dumps everything into an Access database. No keys or relations (granted, only 3 tables, but still), and no idea what most of the columns do. Tools for working with Jet on Linux are minimal, and I didn’t feel like going through win32ole just to get to Jet, plus this sort of thing is nicer to do in downtime at work. So, I exported it via ODBC to a Postgres database on my Solaris box. Not pretty.

access=# \d main
                 TABLE "public.main"
     COLUMN     |          Type          | Modifiers 
----------------+------------------------+-----------
 record_id      | bigint                 | NOT NULL
 STATUS         | integer                | 
 placement      | bigint                 | 
 private        | smallint               | 
 category       | character varying(20)  | 
 start_time     | bigint                 | 
 end_time       | bigint                 | 
 untimed        | smallint               | 
 time_zone      | character varying(40)  | 
 location       | character varying(255) | 
 summary        | text                   | 
 alarm_advance  | character varying(10)  | 
 alarm_unit     | character varying(10)  | 
 repeated_event | character varying(255) | 
 alarm          | smallint               | 
 note           | character varying(100) | 
access=#

Ok, so record_id seems to be some sort of key, and Heather doesn’t bother with notes or alarms, so this doesn’t seem like it’d be so bad. To figure why Google is only taking some of the records, though:

access=$ SELECT count(*) FROM main;
 count 
-------
  5094
(1 row)
access=$ SELECT count(DISTINCT record_id) FROM main;
 count 
-------
  5074
(1 row)
access=$ SELECT count(DISTINCT start_time) FROM main;
 count 
-------
  2488
(1 row)
access=$ SELECT count(DISTINCT end_time) FROM main;
 count 
-------
  2490
(1 row)
access=$ SELECT count(DISTINCT summary) FROM main;
 count 
-------
  2264
(1 row)
access=$ SELECT record_id, start_time, end_time, summary 
FROM main 
WHERE record_id IN 
    (SELECT record_id 
     FROM main 
     GROUP BY record_id 
     HAVING count(*)>1);
 record_id | start_time |  end_time  |                                 summary                                 
-----------+------------+------------+-------------------------------------------------------------------------
         0 | 1231437600 | 1231441200 | tammy 
         0 | 1231869600 | 1231873200 | nb chanber lunch
         0 | 1229642100 | 1229645700 | tammy AND joe photos st claire broiler
         0 | 1231959600 | 1231963200 | dr hunt
         0 | 1230505200 | 1230508800 | tilsen photos
         0 | 1230568200 | 1230571800 | meet gary at studio
         0 | 1230571800 | 1230584400 | bri AND kids
         0 | 1230744600 | 1230748200 | tilsen, AND sandy ORDER y membership mail
         0 | 1230681600 | 1230681600 | Dan, missy AND the kids.
         0 | 1231610400 | 1231614000 |  james j hill houseOngoing Daily 11/15/08 - 2/22/09  m-sat 10-4 sun 1-4
         0 | 1230663600 | 1230667200 | tammys house glasses shopping
         0 | 1229727600 | 1229731200 | ryan help at studio
         0 | 1231889400 | 1231893000 | 
         0 | 1231889400 | 1231903800 | EMS 
         0 | 1237161600 | 1237161600 | spring break
         0 | 1229983200 | 1229986800 | msp WITH the girls
         0 | 1241049600 | 1241049600 | DISH
         0 | 1232233200 | 1232244000 | jordan senior photos excel AND studio 
         0 | 1230055200 | 1230058800 | paige studio
         0 | 1230314400 | 1230318000 | amanda tg
         0 | 1229968800 | 1229972400 | sara AND nolan 
(21 rows)
 
access=$ SELECT record_id, start_time, end_time, summary 
FROM main 
ORDER BY start_time 
ASC LIMIT 10;
 record_id | start_time | end_time | summary 
-----------+------------+----------+---------
   7128069 |   31449600 | 31449600 | c
   7128068 |   31449600 | 31449600 | a
   7123605 |   31449600 | 31449600 | a
   7128070 |   31449600 | 31449600 | 3
   7124866 |   31449600 | 31449600 | c
   7124107 |   31449600 | 31449600 | 3
   7124145 |   31449600 | 31449600 | o
   7124141 |   31449600 | 31449600 | ;
   7128072 |   31449600 | 31449600 | ;
   7128071 |   31449600 | 31449600 | o
(10 rows)
access=$ SELECT record_id, start_time, end_time, summary FROM main ORDER BY start_time DESC LIMIT 10;
 record_id | start_time |  end_time  |           summary           
-----------+------------+------------+-----------------------------
   7127485 | 1256774400 | 1256774400 | lawerance wedding
   7125815 | 1256774400 | 1256774400 | lawerance wedding
   7128114 | 1244167200 | 1244170800 | NB senior ALL night party
   7125941 | 1242489600 | 1242493200 | nyquist edding
   7125827 | 1242489600 | 1242493200 | nyquist edding
         0 | 1241049600 | 1241049600 | DISH
   7128073 | 1238079600 | 1238083200 | books IN the woods
   7125623 | 1238079600 | 1238083200 | books IN the woods
   7125697 | 1238025600 | 1238025600 | gunflint books IN the woods
   7126175 | 1238025600 | 1238025600 | gunflint books IN the woods
(10 rows)
 
access=$

Oh, yeah! What I’ve gathered:

  • There are duplicate record_ids (which I’d hoped would have been unique).
  • There are events set to start and end at duplicate times
  • Palm, at some point, duplicated a lot of the other records, except for the record_id.
  • Times are stored in epoch seconds (oddly, Unix epoch seconds, not Windows)
  • Some of the times correlate to 1970? WTF

A working solution:

access=$ SELECT DISTINCT a.start_time, a.end_time, a.summary 
INTO holdkey 
FROM main a
WHERE EXISTS 
    ( SELECT 'x' FROM main b WHERE a.start_time = b.start_time
      AND a.end_time = b.end_time
      AND a.summary = b.summary) 
ORDER BY a.start_time DESC;
SELECT
access=$ SELECT count(*) FROM holdkey;
 count 
-------
  2597
(1 row)
access=$ DELETE FROM main 
USING holdkey 
WHERE main.start_time = holdkey.start_time 
    AND main.end_time = holdkey.end_time 
    AND main.summary = holdkey.summary;
DELETE 5085
 
access=$ SELECT record_id, start_time, end_time, summary FROM main;
 record_id | start_time |  end_time  | summary 
-----------+------------+------------+---------
   5280360 |   31536000 |   31536000 | 
   5280298 |   31536000 |   31536000 | 
   5280429 |   31536000 |   31536000 | 
   7125497 | 1193437800 | 1193437800 | 
   7128378 |   31536000 |   31536000 | 
   7128376 |   31536000 |   31536000 | 
   7128374 |   31536000 |   31536000 | 
   7127620 | 1193437800 | 1193437800 | 
         0 | 1231889400 | 1231893000 | 
(9 rows)
access=$ DROP TABLE main;
DROP TABLE
access=$ SELECT * INTO main FROM holdkey;
SELECT

That works. Of course there’s the quick and dirty way which doesn’t involve munging about with temp tables:

access=$ DELETE FROM main t1
USING main 
WHERE EXISTS 
    (SELECT * FROM main t2 
         WHERE t1.start_time = t2.start_time 
         AND t1.end_time = t2.end_time 
         AND t1.summary = t2.summary 
         AND t1.record_id < t2.record_id);
DELETE 2488
 
access=$ SELECT count(*) FROM test;
 count 
-------
  2606
(1 row)

It gives a slightly different result, but operates under the assumption that Palm’s record_id means something (it may not, for all I know). On the upside, it preserves all the columns in case they’re useful for something (doubtful). I could order by start_time and select into another table, add an index, and do the same thing, but it’s easier the quick and dirty way. There’s probably a trivial way to do this with joins, but I couldn’t think of one, and it leaves 9 records with a record_id of 0..

Here’s the code which it turns out I didn’t need, but it might be useful to somebody:

#Rips data from Palm Desktop.  Uploads it to Google Calendar
#Written with Python 2.5 (though imports should work anyway)
#
#Currently, the Access MDB Palm Datebook uses has been exported to a 
#PostgreSQL server via ODBC, so I'll be connecting to that
#
#There's code in here for getting through Access also, but I haven't tested it.
#Use at your own risk (kinda like Access).
#
#This is mostly due to the Postgres ODBC driver, and the fact that I didn't
#want to bother with quoting all the queries for Postgres to allow spaces
 
try:
    from xml.etree import ElementTree #Python 2.5, probably 2.6/3.0 also
except ImportError:
    from elementtree import ElementTree #Python <2.4
import gdata.calendar.service
import gdata.service
import atom.service
import gdata.calendar
import atom
import getopt
import sys
import string
import time
import psycopg2 #Talk to Postgres
 
class Struct:
    def __init__(self, *args, **kwargs):
        for k,v in kwargs.items():
            setattr(self, k, v)
 
class GCalMigrate:
    def __init__(self):
        self.conn = None
        self.cur = None
        self.calendar = None
        self.records = []
 
    def connect(self):
       try:
           self.conn = psycopg2.connect("dbname='whatever' user='yournamehere' host='server'")
       except:
           print "Can't connect to the database!\n"
           sys.exit()
       self.cur = conn.cursor()
       query()
 
    def accessconnect(self,mdbpath):
        import odbc
        self.conn = odbc.odbc("driver=Microsoft Access Driver (*.mdb);DBQ=%s") % mdbpath
        self.cur = conn.cursor()
        queryaccess()
 
    def queryaccess(self):
        rows = []
        self.cur.execute("SELECT Main.[Start Time], Main.[End Time], Main.[Summary] FROM Main")
        rows = cur.fetchall()
        conn.close()
        parserows(rows)
 
    def query(self):
       rows = []
       try:
           self.cur.execute("SELECT start_time, end_time, summary FROM main")
           rows = cur.fetchall()
       except:
           print "Couldn't query the database.\n"
       conn.close()
       parserows(rows)
 
    def parserows(self, rows):
        for row in rows:
            starttime = time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime(row[0]))
            endtime = time.strftime("%Y-%m-%dT%H:%M:%S.000Z", time.gmtime(row[1]))
            title = row[2]
            record = Struct(start_time=starttime, end_time=enddtime, title=title)
            self.records.append(record)
        login()
 
    def login(self, username, password):
        self.calendar = gdata.calendar.service.CalendarService()
        self.calendar.email = username
        self.calendar.password = password
        self.calendar.source = "Palm_Desktop_Migrator"
        self.calendar.ProgrammaticLogin()
        batchsubmit()
 
 
    def batchsubmit(self):
        feed = gdata.calendar.CalendarEventFeed()
 
        for record in records:
            insertme = gdata.calendar.CalendarEventEntry()
            insertme.title = atom.Title(record.title)
            insertme.content = atom.Content("")
            insertme.when.append(gdata.calendar.When(start_time=record.start_time, end_time=record.end_time))
            insertme.batch_id = gdata.BatchId(text='Palm_Migration')
 
            feed.Add_Insert(entry=insertme)
        response = self.calendar.ExecuteBatch(feed, gdata.calendar.service.DEFAULT_BATCH_URL)
        return response
 
if __name__ == "__main__":
    runner = GCalMigrate()
    responses = runner.connect()
    for entry in responses.entry:
        print "Batch ID: %s" % entry.batch_id.text
        print "Status: %s" % entry.batch_status.code
        print "Reason: %s" % entry.batch_status.reason

Importing From MySpace

Given that I’ve deleted my Facebook account, and I’ll be deleting my MySpace account shortly (they never get used, and I don’t see the point), I decided to look into the possibility of importing posts from MySpace to Wordpress. As it turns out, the Wordpress developers are apparently really ridiculously lazy, or just don’t give a shit about importing. Somebody had written a Perl script which pulled MySpace blogs into RSS, but bringing RSS into Wordpress doesn’t get comments with it.

After looking over the formats Wordpress -could- import from, I threw out everything with [!CDATA] tags in XML (almost every format). Fortunately, the Movable Type developers don’t see a need to dump binary blobs in XML, or use XML at all. Their format is refreshingly simple. Hence, a parser that runs through every blog post on somebody’s MySpace, pulls out data that matters (title, date, post, commenters and their comments), then puts those into Movable Type format. You’ll note that I now have way more posts on here than I did before, some of those being from before this website existed.

Code:

#!/usr/bin/ruby
 
require 'open-uri'
require 'time'
 
class Comment
  attr_accessor :author, :datewritten, :comment
  def initialize (author, datewritten, comment)
      @author, @datewritten, @comment = author, datewritten, comment
  end
end
 
class Post
  attr_accessor :author, :title, :datewritten, :body, :comments
  def initialize (author, title, datewritten, body)
    @author, @title, @datewritten, @body = author, title, datewritten, body
    @comments = []
  end
  def addcomment (author, datewritten, comment)
    @comments.push(Comment.new(author, datewritten, comment))
  end
end
 
class Ripper
  def initialize  
    @pages = []
    @posts = Array.new
  end
  def get (uri)
    connection = open(uri)  
    content = connection.read
    return content
  end
 
  def parse (uri)
    content = get(uri)
    #blogContentInfo points to links to posts
    links = content.scan(/class="blogContentInfo">.*?<a href=".*?">/m)
    links.each do |link|
        #Strip out the bullshit amazon links
        unless link =~ /amazon/
            #Pull the URL out of the link
            link = (/.*<a href="(.*)">/m).match(link)[1]
            @pages.push(link)
        end
    end
    #Checking if there are any older pages with a hyperlink
    if content =~ /\[.*?<a href="(.*?)">Older<\/a>/
      #If so, call itself recursively to pull out the rest
      #Myspace breaks the URI standard.  Replace the spaces with real escapes
      parse($1.gsub(/\s/, "%20"))
    else
      #Edge case to break out of the loop for when there aren't any more older
      parsepages()
    end
  end
 
  def parsepages()
    @pages.each do |uri|
      #Replace with yourself, if you want
      author = "Ryan"
      content = get(uri)
      #Pull out the fields I want
      title = (/blogSubject">(.*?)\n/m).match(content)[1]
      body = (/blogContent">(.*?)<table/m).match(content)[1]
      datewritten = (/blogTimeStamp">(.*?)<\/p>/m).match(content)[1].gsub(/(^\s+|\n+)/, "")
      time = (/blogContentInfo"><b>.*?(\d+:\d+)/m).match(content)[1]
      datewritten = datewritten + " #{time}:00"
      #Parse the time, and force it into something Wordpress can deal with
      t = Time.parse(datewritten)
      datewritten = t.strftime("%m/%d/%Y %H:%M:%S")
      puts "Title: #{title}\n"
      #Create a new Post object
      post = Post.new(author, title, datewritten, body)
      #Pull out an array of all the comment blocks
      comments = content.scan(/id="blogComments.*?commentSpacer/m)
      #Pass off the post object along with the list of comments
      parsecomments(comments, post)
    end
  end
 
  def parsecomments(comments, post)
    comments.each do |com|
      author = (/profileLinks">(.*?)</m).match(com)[1]
      puts "Author: #{author}\n"
      #MySpace decided to make the CSS ids identical here, except that the
      #actual comment doesn't have "Posted" after the closing tag
      #Filter it as such
      comment = (/blogCommentsContent">(.*?)<\/p>/m).match(com)[1]
      datewritten = (/blogCommentsContent">Posted by.*?> on(.*?)<b/m).match(com)[1].gsub(/\n|\t|\r/, "")
      t = Time.parse(datewritten)
      #The same datetime munging as before
      datewritten = t.strftime("%m/%d/%Y %H:%M:%S")
      #Commit each commment to our post object
      post.addcomment(author, datewritten, comment)
    end
  #Push them all into our class array
  @posts.push(post)
  end
 
  def print(file)
    @posts.each do |post|
      #Using Movable Type's export syntax, so I don't need to mess with XML
      #It's documented here: http://www.sixapart.com/moveabletype/docs/mtimport#example
      #Basically, 5 hyphens separates the categories
      #Eight hyphens separate each post
      file.puts "TITLE: #{post.title}"
      file.puts "AUTHOR: #{post.author}"
      file.puts "DATE: #{post.datewritten}"
      #Change this, too, if you want
      file.puts "CATEGORY: MySpace"
      file.puts "-----"
      #Get rid of empty lines and fucking Windows ^M newlines, plus convert &nbsp; to " "
      file.puts "BODY:\n#{post.body.gsub(/^(\s+|\t+|\n+)$/, "").gsub(/\015/, "").gsub(/&nbsp;/, " ")}"
      file.puts "-----"
      post.comments.each do |com|
        #More stuff is possible here, but isn't necessary
        file.puts "COMMENT:"
        file.puts "AUTHOR: #{com.author}"
        file.puts "DATE: #{com.datewritten}"
        file.puts "#{com.comment}"
        file.puts "-----"
      end
      file.puts "--------"
    end
  end
end
 
#Instantiate it
ripper = Ripper.new
#Parse my blog (substitute whatever yours is here)
ripper.parse("http://blog.myspace.com/lykurgos")
#Output it
 
output = File.open("posts.txt", "a")
ripper.print(output)
puts "Done!\n"
#Import into Wordpress!

Maybe somebody will actually find it useful.

Tags: , ,

categories General

Parser, part deux

Fixed the nested <item> blocks. Ran into another problem where it didn’t parse nested characters and their items properly, then yet another where some Gifts of Khaine (and probably other things I haven’t seen in either list) are essentially nested worthlessness. Fixed code for it:

def parseitem(d, addto)
  added = addto.add_element(d.attributes["name"].gsub(/\s+/, '')).add_text(d.attributes["description"].gsub(/\./, ''))
  d.elements.each('link') do |ele|
    unless ele.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
      if !ele.attributes["description"].nil?
        added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text(ele.attributes["description"])
      else
        #This is necessary for some Gifts of Khaine, apparently
        if added.text == ele.attributes["name"]
          print "Found duplicated #{added.text}!\n\n"
          added.text = ''
        end
        added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text("True")
      end
    end
  end
end
 
def parsenested(process, addto)
  #Try to guess if it's a champion, character in the unit, or item
  process.elements.each('entity') do |d|
    if d.attributes["statset"] =~ /Normal/
      #It's a character, crew, or mount.  Figure out which
      if d.attributes["totalcost"] !~ /^0/
        #It's a champion or character
        adder = addto.add_element("champion")
        puts "Found champ\n"
        parse(d, adder)
      else
        #It's crew or the like
        adder = addto.add_element("crew")
        puts "Found crew\n"
        parse(d, adder)
      end
    else
      #It's an item
      puts "Found item\n"
      if addto.elements["item"].nil?
        @adder = addto.add_element("item")
      end
      parseitem(d, @adder)
    end
  end
end

Tags: , ,

categories General

Parser

Ok, boring. I didn’t spend as much time working on it tonight as I intended to, but it parses the Dwarf roster, at least, fine. It does not parse the Dark Elf roster properly (namely, it doesn’t pull the description out of items or Gifts of Khaine, and it doubles up the <item> tag for reasons I’m not sure of), but that’ll get fixed when I’m at work tomorrow.

Ruby code:

#!/usr/bin/ruby
require "rexml/document"
require "pp"
require "rexml/formatters/default"
include REXML
 
inputxml = File.read('dwarfroster.rst')
@roster = Document.new inputxml
 
@army = Document.new.add_element("army")
 
def parsenested(process, addto)
  #Try to guess if it's a champion, character in the unit, or item
  process.elements.each('entity') do |p|
    #puts p
    if p.elements["link"].has_elements?
      #Recursively run through these to figure out what the hell it is
      if p.elements["link/entity"].attributes["itemsummary"].any?
         adder = addto.add_element("item")
         puts "Found nested\n"
         parsenested(p.elements["link"], adder)
      else
        #This is really just stubbed out, since I haven't seen it
      end
    elsif p.attributes["statset"] =~ /Normal/
      #It's a character, crew, or mount.  Figure out which
      if p.attributes["totalcost"] !~ /^0/
        #It's a champion or character
        adder = addto.add_element("champion")
        puts "Found champ\n"
        parse(p, adder)
      else
        #It's crew or the like
        adder = addto.add_element("crew")
        puts "Found crew\n"
        parse(p, adder)
      end
    else
      #It's an item
      puts "Found item\n"
      if addto.elements["item"].nil?
        @adder = addto.add_element("item")
      end
      added = @adder.add_element(p.attributes["name"].gsub(/\s+/, ''))
      p.elements.each('link') do |ele|
        unless ele.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
          added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text(ele.attributes["description"])
        end
      end
    end
  end
end
 
def parse(s, addto)
    #In some cases, the basename differs (i.e. Supreme Sorc vs. High Sorc)
    #Also, it'll pick up whether there's a champion in the unit by the diff
    #of base and count
    %w[basename count base].each do |b|
      if s.attributes[b].any?
        addto.add_element(b).add_text(s.attributes[b])
      end
    end
    stats = addto.add_element("stats")
    s.elements.each('unitstat') do |a|
      #unit.fetch(:stats) { |el| unit[el] = {}}
      #I don't want blank stats
      if a.attributes["value"].any? && (a.attributes["value"] !~ /(0|-)/)
         stats.add_element(a.attributes["name"].gsub(/\s+/, '')).add_text(a.attributes["value"])
      end
    end
    s.elements.each('link') do |link|
      if link.has_elements?
        #Figure out what the hell it is
        parsenested(link, addto)
      else
        #unitatt = unit.add_element("attributes")
        #Rip out the name if it doesn't have "Helper, Worker, Points Left, or Cost"
        unless link.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
          #Get rid of the stuff in braces AB puts in
          if addto.elements["attributes"].nil?
            @unitatt = addto.add_element("attributes")
          end
          @unitatt.add_element(link.attributes["name"].gsub(/\{.*?\}/, '').gsub(/\s+/, '')).add_text('true')
        end
 
      end
    end
end
 
@roster.elements.each('document/roster') do |ele|
  info = @army.add_element("info")
  #Pick out the race, army name, total points, used points, canonical race name
  %w[race size activesize racename].each do |attr|
    info.add_element(attr).add_text(ele.attributes[attr])
  end
  #@army.push(info)
end
 
@roster.elements.each('document/squad') do |ele|
  @unit = @army.add_element("unit")
 
   #Pick out the name of the model and its cost, plus how many models
   %w[name modelcount totalcost].each do |attr|
     @unit.add_element(attr).add_text(ele.attributes[attr])
    end
    ele.elements.each('entity') do |s|
      #Parse it out
      parse(s, @unit)
    end
end
#pp @army
 
prettyprint = REXML::Formatters::Pretty.new
output = String.new
puts prettyprint.write(@army, output)

And the XML output:

<?xml version="1.0" encoding="ISO-8859-1"?>
<army>
	<info>
		<race>Dwarf</race>
		<size>1500</size>
		<activesize>1499.</activesize>
		<racename>Dwarfs</racename>
	</info>
	<unit>
		<name>Thane</name>
		<modelcount>1</modelcount>
		<totalcost>134</totalcost>
		<basename>Thane</basename>
		<count>1</count>
		<base>1</base>
		<stats>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>3+</Save>
			<St>4/8</St>
			<To>5</To>
			<UnitSt.>1</UnitSt.>
			<WS>6</WS>
			<Wo>2</Wo>
			<At>3</At>
			<BS>4</BS>
			<In>3</In>
			<ItemPts>75</ItemPts>
		</stats>
		<attributes>
			<General>true</General>
			<HandWeapon>true</HandWeapon>
			<GreatWeapon>true</GreatWeapon>
			<GromrilArmor>true</GromrilArmor>
		</attributes>
		<item>
			<RunicWeapon>
				<MasterRuneofKraggtheGrim>Allows other runes to be placed on a Great Weapon.</MasterRuneofKraggtheGrim>
				<RuneofCleaving>+1 Strength</RuneofCleaving>
			</RunicWeapon>
			<RunicArmor>
				<RuneofStone>+1 Armor Save</RuneofStone>
			</RunicArmor>
		</item>
	</unit>
	<unit>
		<name>Thane</name>
		<modelcount>1</modelcount>
		<totalcost>132</totalcost>
		<basename>Thane</basename>
		<count>1</count>
		<base>1</base>
		<stats>
			<In>3</In>
			<ItemPts>75</ItemPts>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>2+/1+</Save>
			<St>4/7</St>
			<To>5</To>
			<UnitSt.>1</UnitSt.>
			<WS>6</WS>
			<Wo>2</Wo>
			<At>3</At>
			<BS>4</BS>
		</stats>
		<attributes>
			<HandWeapon>true</HandWeapon>
			<GromrilArmor>true</GromrilArmor>
			<Shield>true</Shield>
		</attributes>
		<item>
			<RunicWeapon>
				<RuneofCleaving>+1 Strength</RuneofCleaving>
			</RunicWeapon>
			<RunicArmor>
				<RuneofStone>+1 Armor Save</RuneofStone>
			</RunicArmor>
		</item>
	</unit>
	<unit>
		<name>Thane</name>
		<modelcount>1</modelcount>
		<totalcost>95</totalcost>
		<basename>Thane</basename>
		<count>1</count>
		<base>1</base>
		<stats>
			<In>3</In>
			<ItemPts>75</ItemPts>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>3+</Save>
			<St>4</St>
			<To>5</To>
			<UnitSt.>1</UnitSt.>
			<WS>6</WS>
			<Wo>2</Wo>
			<At>3</At>
			<BS>4</BS>
		</stats>
		<attributes>
			<HandWeapon>true</HandWeapon>
			<GromrilArmor>true</GromrilArmor>
			<BattleStandardBearer>true</BattleStandardBearer>
		</attributes>
		<item>
			<RunicArmor>
				<RuneofStone>+1 Armor Save</RuneofStone>
			</RunicArmor>
		</item>
	</unit>
	<unit>
		<name>Dwarf Warriors</name>
		<modelcount>20</modelcount>
		<totalcost>205</totalcost>
		<basename>Dwarf Warriors</basename>
		<count>19</count>
		<base>20</base>
		<stats>
			<In>2</In>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>4+/3+</Save>
			<St>3</St>
			<To>4</To>
			<UnitSt.>1</UnitSt.>
			<WS>4</WS>
			<Wo>1</Wo>
			<At>1</At>
			<BS>3</BS>
		</stats>
		<champion>
			<basename>Veteran</basename>
			<count>1</count>
			<base>1</base>
			<stats>
				<In>2</In>
				<Ld>9</Ld>
				<Mv>3</Mv>
				<Save>4+/3+</Save>
				<St>3</St>
				<To>4</To>
				<UnitSt.>1</UnitSt.>
				<WS>4</WS>
				<Wo>1</Wo>
				<At>2</At>
				<BS>3</BS>
			</stats>
			<attributes>
				<HandWeapon>true</HandWeapon>
				<HeavyArmor>true</HeavyArmor>
				<Shield>true</Shield>
			</attributes>
		</champion>
		<attributes>
			<Musician>true</Musician>
			<StandardBearer>true</StandardBearer>
			<HandWeapon>true</HandWeapon>
			<HeavyArmor>true</HeavyArmor>
			<Shield>true</Shield>
		</attributes>
	</unit>
	<unit>
		<name>Quarellers</name>
		<modelcount>10</modelcount>
		<totalcost>110</totalcost>
		<basename>Quarrellers</basename>
		<count>10</count>
		<base>10</base>
		<stats>
			<In>2</In>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>6+</Save>
			<St>3</St>
			<To>4</To>
			<UnitSt.>1</UnitSt.>
			<WS>4</WS>
			<Wo>1</Wo>
			<At>1</At>
			<BS>3</BS>
		</stats>
		<attributes>
			<HandWeapon>true</HandWeapon>
			<Crossbow>true</Crossbow>
			<LightArmor>true</LightArmor>
		</attributes>
	</unit>
	<unit>
		<name>Quarellers</name>
		<modelcount>10</modelcount>
		<totalcost>110</totalcost>
		<basename>Quarrellers</basename>
		<count>10</count>
		<base>10</base>
		<stats>
			<In>2</In>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>6+</Save>
			<St>3</St>
			<To>4</To>
			<UnitSt.>1</UnitSt.>
			<WS>4</WS>
			<Wo>1</Wo>
			<At>1</At>
			<BS>3</BS>
		</stats>
		<attributes>
			<HandWeapon>true</HandWeapon>
			<Crossbow>true</Crossbow>
			<LightArmor>true</LightArmor>
		</attributes>
	</unit>
	<unit>
		<name>Ironbreakers</name>
		<modelcount>14</modelcount>
		<totalcost>237</totalcost>
		<basename>Ironbreakers</basename>
		<count>13</count>
		<base>14</base>
		<stats>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>3+/2+</Save>
			<St>4</St>
			<To>4</To>
			<UnitSt.>1</UnitSt.>
			<WS>5</WS>
			<Wo>1</Wo>
			<At>1</At>
			<BS>3</BS>
			<In>2</In>
		</stats>
		<champion>
			<basename>Ironbeard</basename>
			<count>1</count>
			<base>1</base>
			<stats>
				<In>2</In>
				<Ld>9</Ld>
				<Mv>3</Mv>
				<Save>3+/2+</Save>
				<St>4</St>
				<To>4</To>
				<UnitSt.>1</UnitSt.>
				<WS>5</WS>
				<Wo>1</Wo>
				<At>2</At>
				<BS>3</BS>
			</stats>
			<attributes>
				<HandWeapon>true</HandWeapon>
				<GromrilArmor>true</GromrilArmor>
				<Shield>true</Shield>
			</attributes>
		</champion>
		<attributes>
			<Musician>true</Musician>
			<StandardBearer>true</StandardBearer>
			<HandWeapon>true</HandWeapon>
			<GromrilArmor>true</GromrilArmor>
			<Shield>true</Shield>
		</attributes>
		<item>
			<RunicStandard>
				<RuneofStoicism>The unit counts as double its actual Unit Strength.</RuneofStoicism>
			</RunicStandard>
		</item>
	</unit>
	<unit>
		<name>Hammerers</name>
		<modelcount>18</modelcount>
		<totalcost>246</totalcost>
		<basename>Hammerers</basename>
		<count>17</count>
		<base>18</base>
		<stats>
			<Ld>9</Ld>
			<Mv>3</Mv>
			<Save>5+</Save>
			<St>4/6</St>
			<To>4</To>
			<UnitSt.>1</UnitSt.>
			<WS>5</WS>
			<Wo>1</Wo>
			<At>1</At>
			<BS>3</BS>
			<In>2</In>
		</stats>
		<champion>
			<basename>Gate Keeper</basename>
			<count>1</count>
			<base>1</base>
			<stats>
				<In>2</In>
				<Ld>9</Ld>
				<Mv>3</Mv>
				<Save>5+</Save>
				<St>4/6</St>
				<To>4</To>
				<UnitSt.>1</UnitSt.>
				<WS>5</WS>
				<Wo>1</Wo>
				<At>2</At>
				<BS>3</BS>
			</stats>
			<attributes>
				<HandWeapon>true</HandWeapon>
				<GreatWeapon>true</GreatWeapon>
				<HeavyArmor>true</HeavyArmor>
			</attributes>
		</champion>
		<attributes>
			<Musician>true</Musician>
			<StandardBearer>true</StandardBearer>
			<HandWeapon>true</HandWeapon>
			<GreatWeapon>true</GreatWeapon>
			<HeavyArmor>true</HeavyArmor>
			<Stubborn>true</Stubborn>
		</attributes>
	</unit>
	<unit>
		<name>Artillery Battery</name>
		<modelcount>4</modelcount>
		<totalcost>45</totalcost>
		<basename>Bolt Thrower</basename>
		<count>1</count>
		<base>1</base>
		<stats>
			<To>7</To>
			<UnitSt.>3</UnitSt.>
			<Wo>3</Wo>
		</stats>
		<crew>
			<basename>Crew</basename>
			<count>3</count>
			<base>3</base>
			<stats>
				<In>2</In>
				<Ld>9</Ld>
				<Mv>3</Mv>
				<Save>6+</Save>
				<St>3</St>
				<To>4</To>
				<WS>4</WS>
				<Wo>1</Wo>
				<At>1</At>
				<BS>3</BS>
			</stats>
			<attributes>
				<HandWeapon>true</HandWeapon>
				<LightArmor>true</LightArmor>
			</attributes>
		</crew>
		<attributes>
			<BoltThrower>true</BoltThrower>
		</attributes>
	</unit>
	<unit>
		<name>Artillery Battery</name>
		<modelcount>4</modelcount>
		<totalcost>45</totalcost>
		<basename>Bolt Thrower</basename>
		<count>1</count>
		<base>1</base>
		<stats>
			<To>7</To>
			<UnitSt.>3</UnitSt.>
			<Wo>3</Wo>
		</stats>
		<crew>
			<basename>Crew</basename>
			<count>3</count>
			<base>3</base>
			<stats>
				<In>2</In>
				<Ld>9</Ld>
				<Mv>3</Mv>
				<Save>6+</Save>
				<St>3</St>
				<To>4</To>
				<WS>4</WS>
				<Wo>1</Wo>
				<At>1</At>
				<BS>3</BS>
			</stats>
			<attributes>
				<HandWeapon>true</HandWeapon>
				<LightArmor>true</LightArmor>
			</attributes>
		</crew>
		<attributes>
			<BoltThrower>true</BoltThrower>
		</attributes>
	</unit>
	<unit>
		<name>Airborne Assault</name>
		<modelcount>1</modelcount>
		<totalcost>140</totalcost>
		<basename>Gyrocopter</basename>
		<count>1</count>
		<base>1</base>
		<stats>
			<In>2</In>
			<Ld>9</Ld>
			<Save>4+</Save>
			<St>4</St>
			<To>5</To>
			<UnitSt.>3</UnitSt.>
			<WS>4</WS>
			<Wo>3</Wo>
			<At>2</At>
		</stats>
		<attributes>
			<Flyer>true</Flyer>
		</attributes>
	</unit>
</army>

I’ll probably screw with the code so it outputs something more easily parsed by Dan (for the items and attributes, mainly) at the same time as I fix the Dark Elf parsing (which should also hit the Anvil of Doom problem). Right now it comes out like this:

<item>
      <SacrificialDagger/>
      <PearlofinfiniteBleakeness/>
      <BlackDragonEgg/>
</item>
<item>
        <RuneofKhaine/>
        <TouchofDeath>
          <KillingBlow/>
        </TouchofDeath>
</item>

As you can see, Touch of Death somehow added the name as a subelement, yet I don’t see any substantial differences between the DE roster and the dwarf roster. Still, it’ll get fixed tomorrow (and $deity willing, converted to .NET).

Tags: , , ,

categories General

How not to write an XML file

I’ll be honest. I don’t like XML. I don’t like SOAP (REST is far nicer in my opinion), since it manipulates the HTTP spec to do things it was never meant to do. Raw sockets and bit twiddling seem like a more logical extension, just that port 80 happens to be open on most corporate firewalls, so SOAP and CORBA have taken off. Inasmuch as I may dislike XML, though, it has its uses. Representing a datastream on for sets where CSV doesn’t really make sense, and YAML isn’t available, and it’s not that hard to deal with.

The ArmyBuilder developers seem to have squeezed SGML into an XML doctype somehow, and the roster files are littered with references I can’t quite make out. Yes, ArmyBuilder can export to XML, I guess, but it leads to XSL from hell. In some ways, I would have preferred to rip apart a binary format with a hex editor, as long as the data was formatted logically.

This, for instance:

<link id="dwWarCrew" count="1" actual="1" script="0" sequence="106" pseudo="no" totalcost="0" \ 
name="Crew" category="Equip" visible="no" sourceid="dwBoltThrw" sourceindex="1"></link>

Or this:

<ruleset context="dwSubtype" ruleset="dwDwarves" contextname="Army Subtype" rulesetname="Dwarf Army"/>

Is not formatted logically. The second record, as you can see, uses XML attributes rather than nodes for everything, which kinda defeats the point. XSL to parse ArmyBuilder’s XML output? Ahh…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
	<xsl:output method="html" version="1.0" encoding="UTF-8" indent="yes"/>
	<xsl:variable name="newlinefeed"><![CDATA[
]]></xsl:variable>
	<xsl:variable name="statCountGlobal" select="count(document/definition/stat_def)"/>
 
	<xsl:template match="/">
	<xsl:variable name="namedModelCount" select="document/composition/@model_count"/>
	<xsl:variable name="actualModelCount" select="sum(//regiment/@model_count)"/>
	<xsl:variable name="actualPoints" select="sum(//regiment/@cost[not(contains(.,'['))])"/>
	<xsl:value-of select="concat(/document/summary/@race_name,': ',$actualPoints, /document/definition/@points_abbrev,' - ',$actualModelCount,' ')"/> Models<xsl:value-of select="$newlinefeed"/>
		<xsl:for-each select="/document/composition/comp_entry">
		<xsl:variable name="groupName" select="@group_name"/>
		<xsl:if test="/document/roster/top_level/regiment[@composition = $groupName]">
			<xsl:variable name="unit" select="@group_name"/>
			<xsl:for-each select="/document/roster/top_level/regiment[@composition = $unit]">
				<xsl:apply-templates select="." mode="top_level">
					<xsl:with-param name="regDepth">
						<xsl:choose>
							<xsl:when test="position()=1"><xsl:value-of select="count(/document/roster/top_level[regiment/@composition = $unit]//regiment)"/></xsl:when>
							<xsl:otherwise>0</xsl:otherwise>
						</xsl:choose>
					</xsl:with-param>
				</xsl:apply-templates>
			</xsl:for-each>	
		</xsl:if>
		</xsl:for-each>
	</xsl:template>
	<xsl:template match="regiment" mode="top_level">
		<xsl:param name="regDepth">
			<xsl:value-of select="count(..//regiment)"/>
		</xsl:param>
		<xsl:variable name="statCountLocal" select="count(stat)"/>
		<xsl:variable name="fsib" select="preceding-sibling::node()"/>
		<xsl:variable name="composition" select="@composition"/>
		<xsl:if test="$regDepth > 0">
		<xsl:choose>
			<xsl:when test="not($composition = $fsib/@composition)">
				<xsl:value-of select="$composition"/>(<xsl:value-of select="/document/composition/comp_entry[@group_name = $composition]/@percentage"/>)<xsl:value-of select="$newlinefeed"/>
			</xsl:when>
		</xsl:choose>
		</xsl:if>
		<xsl:value-of select="concat('[',@model_count,'] ')"/>
		<xsl:variable name="itemcost">
			<xsl:if test="@cost"><xsl:value-of select="@cost"/></xsl:if>
		</xsl:variable>
		<xsl:variable name="retinuecost">
			<xsl:call-template name="getRetinueCost">
				<xsl:with-param name="retinuecostSum" select="0"/>
				<xsl:with-param name="current" select="regiment[position()=1]"/>
				<xsl:with-param name="rest" select="regiment[position()!=1]"/>
			</xsl:call-template>
		</xsl:variable>
		<xsl:variable name="transportcost">
			<xsl:if test="regiment[@stat_set = 0]/@cost"><xsl:value-of select="substring-after(substring-before(regiment[@stat_set = 0]/@cost,']'),'[')"/></xsl:if>
		</xsl:variable>
		<xsl:choose>
			<xsl:when test="$retinuecost and @model_count=1 and @composition='HQ'">
					<xsl:value-of select="concat('[',$itemcost - $retinuecost,'] ')"/>
			</xsl:when>
			<xsl:when test="$transportcost != ''">
					<xsl:value-of select="concat('[',$itemcost - $transportcost,'] ')"/>
			</xsl:when>
			<xsl:otherwise>
					<xsl:value-of select="concat('[',$itemcost,'] ')"/>
			</xsl:otherwise>
		</xsl:choose>
 
		<xsl:value-of select="substring-before(name,' (')"/>
		<xsl:if test="@model_count=1 and @composition='HQ'">
			<xsl:text disable-output-escaping="yes">(IC)</xsl:text>
		</xsl:if>
		<xsl:text disable-output-escaping="yes">: </xsl:text>
 
		<xsl:for-each select="item">
			<xsl:variable name="namedItem" select="name"/>
			<xsl:choose>
				<xsl:when test="count(../item[name=$namedItem]) > 1"><xsl:value-of select="concat($namedItem,'(x',count(../item/name[.=$namedItem]),');')"/></xsl:when>
				<xsl:otherwise><xsl:value-of select="concat(name,';')"/></xsl:otherwise>
			</xsl:choose>
		</xsl:for-each>
		<xsl:for-each select="choice"><xsl:value-of select="concat(name,';')"/></xsl:for-each>
		<xsl:value-of select="$newlinefeed"/>
 
		<xsl:for-each select=".//regiment[not(../@category = 'Wargear Item')] ">
			<xsl:apply-templates select="." mode="regiment" />
		</xsl:for-each>
		<xsl:value-of select="$newlinefeed"/>
	</xsl:template>
	<xsl:template match="regiment" mode="regiment">
		<xsl:variable name="statCountLocal" select="count(stat)"/>
		<xsl:variable name="depth">
			<xsl:choose>
				<xsl:when test="../../@stat_count=1"><xsl:value-of select="number(@depth)-1" /></xsl:when>
				<xsl:otherwise><xsl:value-of select="@depth" /></xsl:otherwise>
			</xsl:choose>
		</xsl:variable>
		<xsl:value-of select="concat('[',@model_count,'] ')"/>
 
		<xsl:variable name="itemcost">
			<xsl:if test="@cost"><xsl:value-of select="substring-after(substring-before(@cost,']'),'[')"/></xsl:if>
		</xsl:variable>
		<xsl:variable name="transportcost">
			<xsl:if test="regiment[@stat_set = 0]/@cost"><xsl:value-of select="substring-after(substring-before(regiment[@stat_set = 0]/@cost,']'),'[')"/></xsl:if>
		</xsl:variable>
		<xsl:choose>
			<xsl:when test="((../@composition='HQ' and @depth = 1) or (@depth = 0)) and (regiment[@stat_set = 0])">
					<xsl:value-of select="concat('[',$itemcost - $transportcost,'] ')"/>
			</xsl:when>
			<xsl:when test="@stat_set = 0">
					<xsl:value-of select="concat('[',$itemcost,'] ')"/>
			</xsl:when>
			<xsl:when test="../@composition = 'HQ' and ../@stat_count > 1">
					<xsl:value-of select="concat('[',$itemcost,'] ')"/>
			</xsl:when>
		</xsl:choose>
 
		<xsl:call-template name="formatName"><xsl:with-param name="strName" select="concat(name,': ')"/></xsl:call-template>
 
		<xsl:for-each select="item[not(name = preceding-sibling::item/name)]">
			<xsl:variable name="namedItem" select="name"/>
			<xsl:choose>
				<xsl:when test="count(../item[name=$namedItem]) > 1"><xsl:value-of select="concat($namedItem,'(x',count(../item[name=$namedItem]),');')"/></xsl:when>
				<xsl:otherwise><xsl:value-of select="concat(name,';')"/></xsl:otherwise>
			</xsl:choose>
		</xsl:for-each>
		<xsl:for-each select="choice"><xsl:value-of select="concat(name,';')"/></xsl:for-each>
		<xsl:value-of select="$newlinefeed"/>
 
	</xsl:template>
	<xsl:template name="getRetinueCost">
		<xsl:param name="retinuecostSum" />
		<xsl:param name="current" />
		<xsl:param name="rest" />
		<xsl:variable name="curCost">
			<xsl:choose>
				<xsl:when test="contains($current/@cost,'[')"><xsl:value-of select="substring-after(substring-before($current/@cost,']'),'[')" /></xsl:when>
				<xsl:otherwise><xsl:value-of select="$current" /></xsl:otherwise>
			</xsl:choose>
		</xsl:variable>
		<xsl:choose>
			<xsl:when test="$current">
				<xsl:call-template name="getRetinueCost">
					<xsl:with-param name="retinuecostSum" select="$retinuecostSum + $curCost"/>
					<xsl:with-param name="current" select="$rest[position()=1]"/>
					<xsl:with-param name="rest" select="$rest[position()!=1]"/>
				</xsl:call-template>
			</xsl:when>
			<xsl:otherwise>
				<xsl:value-of select="$retinuecostSum" />
			</xsl:otherwise>
		</xsl:choose>
	</xsl:template>
	<xsl:template name="halfCost">
		<xsl:param name="itemCost"/>
		<xsl:value-of select="round($itemCost div 2)" />
	</xsl:template>
	<xsl:template name="formatName">
		<xsl:param name="strName"/>
		<xsl:choose>
			<xsl:when test="contains($strName,' (')"><xsl:value-of select="substring-before($strName,' (')"/></xsl:when>
			<xsl:otherwise><xsl:value-of select="$strName"/></xsl:otherwise>
		</xsl:choose>
	</xsl:template>
	<xsl:template name="doReplaceCar">
		<xsl:param name="text"/>
		<xsl:param name="replace"/>
		<xsl:param name="by"/>
		<xsl:choose>
			<xsl:when test="contains($text, $replace)">
				<xsl:value-of select="substring-before($text, $replace)" disable-output-escaping="yes"/>
				<xsl:value-of select="$by" disable-output-escaping="yes"/>
				<xsl:call-template name="doReplaceCar">
					<xsl:with-param name="text" select="substring-after($text, $replace)"/>
					<xsl:with-param name="replace" select="$replace"/>
					<xsl:with-param name="by" select="$by"/>
				</xsl:call-template>
			</xsl:when>
			<xsl:otherwise>
				<xsl:value-of select="$text" disable-output-escaping="yes"/>
			</xsl:otherwise>
		</xsl:choose>
	</xsl:template>
</xsl:stylesheet>

No, I am not writing a stylesheet like that again, and the one to parse the roster files would be far more complicated.

The problem with the roster file, fundamentally, is that it’s too tightly linked with ArmyBuilder. That makes sense, in a way, but is still irksome. The <link> elements don’t have any nodes under them, just assloads of attributes, and it’s not easy to figure out which ones I am interested in:

<link id="HeavyArmor" count="1" actual="1" script="0" sequence="26" pseudo="no" totalcost="0" name="Heavy Armor" category="Equip" \
 abbrev="Hv" description="5+ Armor Save" equipment="yes" footnote="yes" sourceid="dwWarrVet" sourceindex="5"></link>

Versus ones I’m not interested in:

<link id="ItemCost" count="1" actual="1" script="0" sequence="28" pseudo="no" totalcost="0" name="Item Cost Worker" category="Equip"\
 visible="no" sourcetype="3" sourceid="Globals" sourceindex="1"></link>

Without passing a long hashlist of element.attribute[$thing] values, or specifically excluding anything with “Helper” or “Worker” or whatever in the name, etc. Not to mention it’s formatted as:

<document>
  <squad>
  <!-- unit name and cost is here -->
    <entity>
    <!-- unit stats are here, along with composition and whatnot -->
      <link>
      <!-- sometimes there's nothing of note in the link tags -->
         <entity>
         <!-- this might be a magic item, warmachine crew, magic banner, champion, and probably other stuff, but is not easily  \
         identified, and there may be more than one -->
            <link>
            <!-- might be info for whatever is in entity, might be a helper which I don't want -->
            </link>
            <unitstat>
            <!-- if it's crew, champion, whatever, stats would be here, but this node may not exist -->
            </unitstat>
        </entity>
      </link>
   </entity>
 </squad>
</document>

The problem with some of these is that by the mantra of whoever wrote ArmyBuilder, champions fall into the “Equip” category. There is, in fact, a “isunit” attribute, but it isn’t set to yes anywhere. Only set to “no” for items, which I can’t figure out (unless there’s some kind of magic item which qualifies as a unit you can add? I don’t know).

I’ve got a parser that works in Ruby written, but I haven’t converted it to C# yet. Also, I’ve not tested it against anything that might have more complicated schema than dwarves: mounted units, chariots, to check if it’s undead/daemon/greenskin and see if special rules apply (since not everything in the army is guaranteed to be), embedded assassins, magic, et al. Sadly, the only roster I’ve got at work is for dwarves, so I’ll have to dump some more output from ArmyBuilder and run the parser against it to see how it handles it.

Any other niche cases either of you can think of that may have specific rules? I’m going to try to stabilize the parser and get it to properly validate every army type, then move it to .NET

Also, thinking about it, I’m utterly convinced that snapping things to some kind of a grid is the only real feasible solution. Querying the object via System.Drawing or GDI might work, but I’m not sure how accurate the pixel mapping is. At any rate, for things like the Lance Formation, line of sight on skirmishers, determining base contact for champions/characters embedded, reforming the unit, and templates, a grid seems like the only way to go without doing occlusion detection (for the templates). Convert inches to millimeters, and make it 1mm x 1mm squares or something.

Tags: , ,

categories General

Two posts in one day?

To begin with, here’s a link to an “article” written by some asshat which showed up on Reddit. I clicked the link out of sheer boredom, and a glimmer of hope that the article may have something interesting. Instead, it turned out to be a deluge of pseudo-intellectual garbage which attempts to redefine morality (which has been discussed for at least two millennia by philosophers and religious scholars) by hedging it into the definition some psychologist in 1987 provided, then regurgitates paraphrased snippets from Mill and a couple sociologists while affixing everybody (other than himself, as he’s “enlightened”) to Procrustes’ bed. Worse still, it appears have been some kind of post-graduate project, which is a clear affirmation that most of the people churning their way through college have no right to be there.

That may be a subjective statement, but even the opening paragraph clearly establishes that he considers himself a breed apart:

People vote Republican because Republicans offer “moral clarity”Ëœa simple vision of good and evil that activates deep seated fears in much of the electorate. Democrats, in contrast, appeal to reason with their long-winded explorations of policy options for a complex world.

Diagnosis is a pleasure. It is a thrill to solve a mystery from scattered clues, and it is empowering to know what makes others tick. In the psychological community, where almost all of us are politically liberal, our diagnosis of conservatism gives us the additional pleasure of shared righteous anger. We can explain how Republicans exploit frames, phrases, and fears to trick Americans into supporting policies (such as the “war on terror” and repeal of the “death tax”) that damage the national interest for partisan advantage.

But with pleasure comes seduction, and with righteous pleasure comes seduction wearing a halo. Our diagnosis explains away Republican successes while convincing us and our fellow liberals that we hold the moral high ground. Our diagnosis tells us that we have nothing to learn from other ideologies, and it blinds us to what I think is one of the main reasons that so many Americans voted Republican over the last 30 years: they honestly prefer the Republican vision of a moral order to the one offered by Democrats. To see what Democrats have been missing, it helps to take off the halo, step back for a moment, and think about what morality really is.

The duty of government is not to tell us what is “right” or what is “wrong” (morals), it is to protect the intrinsic rights of its citizenry while not oppressing or extinguishing those of others, and provide them with beneficial services they cannot do themselves (or as a smaller collective). I, frankly, do not have anything to learn from an ideology which espouses intolerance, sexism, racism (even in its current anti-immigration form), abolition of the right to choose whether or not to keep a child, and encroaching government surveillance. These things are an anathema to ethical behaviour.

I began to study morality and culture at the University of Pennsylvania in 1987. A then-prevalent definition of the moral domain, from the Berkeley psychologist Elliot Turiel, said that morality refers to “prescriptive judgments of justice, rights, and welfare pertaining to how people ought to relate to each other.” But if morality is about how we treat each other, then why did so many ancient texts devote so much space to rules about menstruation, who can eat what, and who can have sex with whom? There is no rational or health-related way to explain these laws. (Why are grasshoppers kosher but most locusts are not?) The emotion of disgust seemed to me like a more promising explanatory principle. The book of Leviticus makes a lot more sense when you think of ancient lawgivers first sorting everything into two categories: “disgusts me” (gay male sex, menstruation, pigs, swarming insects) and “disgusts me less” (gay female sex, urination, cows, grasshoppers ).

Hmm… I’m going to assume “so many ancient texts” refers to the Torah (and dependent texts, such as the Bible), since Babylonian law, Hittite law, Assyrian law, Indian law, Phonetician law, Greek law, and Chinese law say no such thing. Beyond which, modern analysis of kashrut gives clear reasons for the prohibitions on eating certain foods (including, but not limited to, trichinosis, salmonella, worms and other parasites), most of which relate to what that thing ate (prohibitions against shellfish, given that crabs are bottom feeders, pigs root around in the ground, whereas locusts and cows primarily eat food which people ate by itself [crops], which is not likely to make one sick). Proscriptions against having sex with other people are not relegated to ancient law. Anti-sodomy laws are very much on the books. For the record, gay sex was only prohibited in relation to temple prostitution or pagan rites, and even then the Hebrew for it is “ritually unclean.” All the sex acts in Leviticus are “ritually unclean.” No more, no less. It was a way of separating Abrahamic worship from pagan. Control.

Do not rely on a text which went Hebrew (or Aramaic/Coptic) → Greek → Latin → German → English, losing the nuances and intent of the original.

This research led me to two conclusions. First, when gut feelings are present, dispassionate reasoning is rare. In fact, many people struggled to fabricate harmful consequences that could justify their gut-based condemnation. I often had to correct people when they said things like “it’s wrong because, um, eating dog meat would make you sick” or “it’s wrong to use the flag because, um, the rags might clog the toilet.” These obviously post-hoc rationalizations illustrate the philosopher David Hume’s dictum that reason is “the slave of the passions, and can pretend to no other office than to serve and obey them.” This is the first rule of moral psychology: feelings come first and tilt the mental playing field on which reasons and arguments compete. If people want to reach a conclusion, they can usually find a way to do so. The Democrats have historically failed to grasp this rule, choosing uninspiring and aloof candidates who thought that policy arguments were forms of persuasion.

In other countries, policy arguments (gasp!) are a form of persuasion. They do not expect their candidates to be the sort of people they can have a beer with (rather, they don’t expect them to be the dude down the street they barbecue and watch football with).

When intentionally picking an argument which plays to people’s inherent ideas of right and wrong (e.g. morals), it’s not surprising to find morality come up. This is reductio ad absurdum, not fitting for a paper on psychology. Essentially, his argument is that people will hold to preconceived notions no matter whether they benefit the rest of society (or even themselves) if it can provoke a “gut feeling.” Surprise. This assumes a lab experiment situation where these people have nothing riding on whether they’d hypothetically eat their dog if it were killed by a car, or use a flag for rags. Inasmuch as “gut feelings” come into it, people can certainly pursue dispassionate reasoning when they have a reason to. There’s little doubt that cannibalism provokes “gut feelings,” yet the players of a certain well-known soccer team wasted no time in setting the meat of their compatriots out to turn into jerky, as it would survive longer, nor are their qualms about drinking your own urine if necessary, etc.

It doesn’t just boil down to a matter of survival, though. The crew which dropped the second atomic bomb (knowing what happened to Hiroshima) and those who ordered them to do so made a cold-blooded decision to kill 50,000 people in the blink of an eye.

What’s missing from his analysis is realizing that the bolded line could just as easily be phrased as this:

Why of course the people don’t want $thing. Why should some poor slob on a farm want to risk his life (or livelihood) in a $thing when the best he can get out of it is to come back to his farm in one piece? Naturally the common people don’t want $thing; neither in Russia, nor in England, nor in America, nor in Germany. That is understood. But after all, it is the leaders of the country who determine policy, and it is always a simple matter to drag the people along, whether it is a democracy, or a fascist dictatorship, or a parliament, or a communist dictatorship. Voice or no voice the people can always be brought to the bidding of the leaders. That is easy. All you have to do is to tell them they are being attacked (or losing their national identity or whatever), and denounce the pacifists for lack of patriotism and exposing the country to danger. It works the same in any country.

Yes, paraphrasing Göring. The people are easily manipulated. It’s not that they intrinsically believe whatever it is, it’s that the candidates convince them that they’re somehow superior (helped out by shit like this) to others, morally or otherwise.

When Republicans say that Democrats “just don’t get it,” this is the “it” to which they refer. Conservative positions on gays, guns, god, and immigration must be understood as means to achieve one kind of morally ordered society. When Democrats try to explain away these positions using pop psychology they err, they alienate, and they earn the label “elitist.” But how can Democrats learn to seeËœlet alone respectËœa moral order they regard as narrow-minded, racist, and dumb?

That “moral order” is narrow-minded, racist, and dumb. Religion is a personal choice. Keep it so, as the Constitution admonishes. Homosexuality is a choice which has fuck-all to do with governing, and is objected to solely on the basis of religion (as near as I can tell, since it’s not as if there’d be a mad press of people switching to the other team if gay marriage were suddenly legal). Give them the same rights in marriage that everybody else has (common law marriage pretty my abrogates any right to objection they may have, as that amounts to “living in sin” for anywhere from 5-30 years, and it’s recognized in almost all states). Immigration? Racist. Historically, we are nowhere near the level of immigrants we had in the 1880s, 1860s, 1920s, etc as a proportion of the population. People in the US have always hated immigrants when they come, yet they assimilate after a few generations, and nobody cares. Some cultures (Jewish in particular, but also a lot of Asian) still maintain a cultural identity, but nobody seems to care, as long as they’re not speaking Spanish. There’s no way one cannot pretend the Willie Horton ad was anything but racist, nor are the outright lies about Hispanics and crime (DoJ statistics indicate that the percentage of Hispanics in jail has remained steady over the last 20 years, and foreign nationals from Latin-American countries are going down).

After graduate school I moved to the University of Chicago to work with Shweder, and while there I got a fellowship to do research in India. In September 1993 I traveled to Bhubaneswar, an ancient temple town 200 miles southwest of Calcutta. I brought with me two incompatible identities. On the one hand, I was a 29 year old liberal atheist who had spent his politically conscious life despising Republican presidents, and I was charged up by the culture wars that intensified in the 1990s. On the other hand, I wanted to be like those tolerant anthropologists I had read so much about.

Just one comment here, really. Anthropologists are not supposed to be “tolerant.” They are supposed to be outside observers. Unbiased, and unable to have their prevailing opinions swayed by the culture they are documenting. That is their purpose.

My first few weeks in Bhubaneswar were therefore filled with feelings of shock and confusion. I dined with men whose wives silently served us and then retreated to the kitchen. My hosts gave me a servant of my own and told me to stop thanking him when he served me. I watched people bathe in and cook with visibly polluted water that was held to be sacred. In short, I was immersed in a sex-segregated, hierarchically stratified, devoutly religious society, and I was committed to understanding it on its own terms, not on mine.

It only took a few weeks for my shock to disappear, not because I was a natural anthropologist but because the normal human capacity for empathy kicked in. I liked these people who were hosting me, helping me, and teaching me. And once I liked them (remember that first principle of moral psychology) it was easy to take their perspective and to consider with an open mind the virtues they thought they were enacting. Rather than automatically rejecting the men as sexist oppressors and pitying the women, children, and servants as helpless victims, I was able to see a moral world in which families, not individuals, are the basic unit of society, and the members of each extended family (including its servants) are intensely interdependent. In this world, equality and personal autonomy were not sacred values. Honoring elders, gods, and guests, and fulfilling one’s role-based duties, were more important. Looking at America from this vantage point, what I saw now seemed overly individualistic and self-focused. For example, when I boarded the plane to fly back to Chicago I heard a loud voice saying “Look, you tell him that this is the compartment over MY seat, and I have a RIGHT to use it.”

Back in the United States the culture war was going strong, but I had lost my righteous passion.

Summary: “India blew my mind, man.”
More accurately (and a tad more succinct): “I fail at anthropology.”

Essentially, his viewpoint utterly changed from living in a foreign society. While not totally expected, it’s not as if it puts him in a higher caliber than people in the US. Equality and personal autonomy are not “sacred values” in India (any more than they are in Pakistan or Saudi). India is moving towards our model, though. I wonder if he would have been swayed the same way had he spent a few months with some whackjob Mormon cult in southern Nevada. Probably.

Interdependence is not diametrically opposed to individual freedoms. India still has a caste system, for God’s sake, and they’re known for their atrocious civil rights (or lack thereof). I’d say that he’d know that if he weren’t living a privileged existence, with a strong inclination that his family likely lived the same, and they didn’t need to depend upon each other. Many people in America do. Sure, we’re known for being self-centered assholes around the world (namely, that guy on the plane). That doesn’t mean it’s endemic to US society, though.

He failed to see the men as “oppressors,” and didn’t see the women, children, and servants as victims. Servants. Not that the men are exactly oppressors, but the women in that particularly religious community in a country with a literacy rate of less than 60% aren’t exactly agitating for equal rights yet.

Back in the United States the culture war was going strong, but I had lost my righteous passion. I could never have empathized with the Christian Right directly, but once I had stood outside of my home morality, once I had tried on the moral lenses of my Indian friends and interview subjects, I was able to think about conservative ideas with a newfound clinical detachment. They want more prayer and spanking in schools, and less sex education and access to abortion? I didn’t think those steps would reduce AIDS and teen pregnancy, but I could see why the religious right wanted to “thicken up” the moral climate of schools and discourage the view that children should be as free as possible to act on their desires. Conservatives think that welfare programs and feminism increase rates of single motherhood and weaken the traditional social structures that compel men to support their own children? Hmm, that may be true, even if there are also many good effects of liberating women from dependence on men. I had escaped from my prior partisan mindset (reject first, ask rhetorical questions later), and began to think about liberal and conservative policies as manifestations of deeply conflicting but equally heartfelt visions of the good society.

So… once he put on the “moral lenses” of his “Indian friends” (presumably men) from the temple town where they still have servants, he saw conservative ideas with “clinical detachment”? Right… The last sentence I may agree with, but only with the notion that they are “visions” of the “good society”, not that the realization of their policies would lead to a good society.

He’s honestly saying that he thinks welfare programs and feminism raise the rates of single motherhood and “weaken the social structures…”. I guess those Cadillacs they were giving out to “welfare queens” must not have been a total racist fabrication by Reagan, and perpetuated for decades. I guess the US Census Data showing that the rates of single parents haven’t really gone up in decades, and it was blas&eaccent; enough for Lassie, as well as being commonplace enough for it not to be a major theme in

    The Scarlet Letter

. There’s no doubt that some people choose to have a child even though they don’t really have a partner, and they raise it themselves (or continue to do so after a breakup/whatever). Welfare and feminism hardly make up for the additional costs of doing so, and the vast majority of single parents are never on welfare.

The fact that he’d even consider such arguments shows a flawed point of view. His old mindset: “reject first, ask rhetorical questions later” seems to be the same as his new mindset, when it ought to have been “find incontrovertible facts, show them to the bullshit artists and easily swayed.”

On Turiel’s definition of morality (“justice, rights, and welfare”), Christian and Hindu communities don’t look good. They restrict people’s rights (especially sexual rights), encourage hierarchy and conformity to gender roles, and make people spend extraordinary amounts of time in prayer and ritual practices that seem to have nothing to do with “real” morality. But isn’t it unfair to impose on all cultures a definition of morality drawn from the European Enlightenment tradition? Might we do better with an approach that defines moral systems by what they do rather than by what they value?

To answer his rhetorical questions: NO. No it is not. We ought not to “impose” a personal choice onto people to begin with, Hindu/Christian values included. “Justice, rights, and welfare” is a fantastic ideal to strive for.

Here’s my alternative definition: morality is any system of interlocking values, practices, institutions, and psychological mechanisms that work together to suppress or regulate selfishness and make social life possible. It turns out that human societies have found several radically different approaches to suppressing selfishness, two of which are most relevant for understanding what Democrats don’t understand about morality.

Oh, really? I didn’t realize morality was necessary for social interaction. Tell that to Ted Bundy. Suppressing and regulating selfishness isn’t necessary for a functioning society. Keeping sociopaths out is. That which benefits the many almost always benefits the few. The converse is not true. Strive for the betterment of society as a whole (justice, rights, welfare, equality).

First, imagine society as a social contract invented for our mutual benefit. All individuals are equal, and all should be left as free as possible to move, develop talents, and form relationships as they please. The patron saint of a contractual society is John Stuart Mill, who wrote (in On Liberty) that “the only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others.” Mill’s vision appeals to many liberals and libertarians; a Millian society at its best would be a peaceful, open, and creative place where diverse individuals respect each other’s rights and band together voluntarily (as in Obama’s calls for “unity”) to help those in need or to change the laws for the common good.

I didn’t know my utilitarianism defined my political views. Thanks for the enlightenment! Mill certainly appealed to libertarians. Being libertarian, however, has little to do with being conservative or liberal. It has to do with not supporting an authoritarian government. Mill’s vision discards the need for a government, though not in quite the same way as Marx/Engels.

Psychologists have done extensive research on the moral mechanisms that are presupposed in a Millian society, and there are two that appear to be partly innate. First, people in all cultures are emotionally responsive to suffering and harm, particularly violent harm, and so nearly all cultures have norms or laws to protect individuals and to encourage care for the most vulnerable. Second, people in all cultures are emotionally responsive to issues of fairness and reciprocity, which often expand into notions of rights and justice. Philosophical efforts to justify liberal democracies and egalitarian social contracts invariably rely heavily on intuitions about fairness and reciprocity.

This is starting to sound more like a blend of philosophy, political science and anthropology, which are clearly not his strong suits. People are “emotionally responsive” to suffering and harm (particularly violent)? You don’t say. Maybe they have some weird desire to avoid it happening to them. Laws and cultures have norms and laws to protect individuals, but that’s a chicken-or-the-egg situation. Were we able to form lasting societies in the first place because these are intrinsic beliefs of people, or is it because somebody laid down the proverbial law and enforced it? All the evidence from ancient law indicates the latter. Laws are there to protect a functioning social order and economy, not individuals.

But now imagine society not as an agreement among individuals but as something that emerged organically over time as people found ways of living together, binding themselves to each other, suppressing each other’s selfishness, and punishing the deviants and free-riders who eternally threaten to undermine cooperative groups. The basic social unit is not the individual, it is the hierarchically structured family, which serves as a model for other institutions. Individuals in such societies are born into strong and constraining relationships that profoundly limit their autonomy. The patron saint of this more binding moral system is the sociologist Emile Durkheim, who warned of the dangers of anomie (normlessness), and wrote, in 1897, that “Man cannot become attached to higher aims and submit to a rule if he sees nothing above him to which he belongs. To free himself from all social pressure is to abandon himself and demoralize him.” A Durkheimian society at its best would be a stable network composed of many nested and overlapping groups that socialize, reshape, and care for individuals who, if left to their own devices, would pursue shallow, carnal, and selfish pleasures. A Durkheimian society would value self-control over self-expression, duty over rights, and loyalty to one’s groups over concerns for outgroups.

Again, this idea of “suppressing selfishness.” What the hell is that? People found means to live relatively peaceful, profitable lives through laws (and religion, another form of control). It’s all about selfishness, and even his comment about “free-riders” (I’m assuming it’s analogous to a freeloader, just that he felt it sounded less offensive). If anything, a conservative society pursues shallow and selfish pleasures (materialism, self-righteousness, etc). The basic unit in such a society is a limestone block, working to bear the weight of the capstone on top where some lucky fucker gets everything for doing nothing.

“Nested and overlapping groups” take on a life of their own, and protect their own interests (see: corporations, major churches, hate groups). Those that care for individuals (the ACLU, for instance) are held up as institutions to hate by the conservatives. This model does not support a conservative ethos.

A Durkheimian ethos can’t be supported by the two moral foundations that hold up a Millian society (harm/care and fairness/reciprocity). My recent research shows that social conservatives do indeed rely upon those two foundations, but they also value virtues related to three additional psychological systems: ingroup/loyalty (involving mechanisms that evolved during the long human history of tribalism), authority/respect (involving ancient primate mechanisms for managing social rank, tempered by the obligation of superiors to protect and provide for subordinates), and purity/sanctity (a relatively new part of the moral mind, related to the evolution of disgust, that makes us see carnality as degrading and renunciation as noble). These three systems support moralities that bind people into intensely interdependent groups that work together to reach common goals. Such moralities make it easier for individuals to forget themselves and coalesce temporarily into hives, a process that is thrilling, as anyone who has ever “lost” him or herself in a choir, protest march, or religious ritual can attest.

So, his recent (uncited) research shows that conservatives do rely on “care” and “fairness.” Maybe those “compassionate conservatives” I’ve heard so much about. Their other “pillars” are:

  • Ingroup — Hating outsiders, and doing anything you can to keep them out of your society
  • Authority — who’d have guessed it? Not that authority is bad, but blindly submitting to some “alpha male” is hardly an ideal to base a society upon.
  • Disgust — Apparently we “evolved” this, rather than subverting it from something which keeps us from doing things and going near things (spiders, feces, etc) which were likely to do harm to us in earlier parts of evolution. Now “avoiding carnality” is a pillar of conservative society, which is apparent if you look at the great moral leaders of conservative society (McCain, Limbaugh, et al).

All three smack of the same basic thing: A feeling of superiority. I’m better because I’m part of something you’re not, and I’m not going to let you in because that’d lower my social standing, or I’m better than you because I can pretend to abstain from “carnal” acts, etc.

In several large internet surveys, my collaborators Jesse Graham, Brian Nosek and I have found that people who call themselves strongly liberal endorse statements related to the harm/care and fairness/reciprocity foundations, and they largely reject statements related to ingroup/loyalty, authority/respect, and purity/sanctity. People who call themselves strongly conservative, in contrast, endorse statements related to all five foundations more or less equally. (You can test yourself at www.YourMorals.org.) We think of the moral mind as being like an audio equalizer, with five slider switches for different parts of the moral spectrum. Democrats generally use a much smaller part of the spectrum than do Republicans. The resulting music may sound beautiful to other Democrats, but it sounds thin and incomplete to many of the swing voters that left the party in the 1980s, and whom the Democrats must recapture if they want to produce a lasting political realignment.

Oh, several “large internet surveys”? I wonder why I haven’t seen them, and they’re not referenced anywhere, other than plugging some website which doesn’t have any actual data.

Regardless, why do I (as a socialist, but I’ll call myself a liberal for now) want a lasting political realignment with people who will do anything to win, filibuster protecting my rights and passing progressive legislation, spy on me, outright lie, spread racism (aforementioned Willie Horton, Cadillac-driving “welfare queens”, hatred for immigrants), and whose prevailing political message seems to be hatred. Don’t bother having any actual solutions, just criticize your opponent’s character (whether or not it’s true), etc. The link to PBS above is sickening, but the popularity of Fox News is just as bad. The hell with those people. We don’t want a “lasting political realignment.” We want their party to die, and their “ideals” (hatred) along with them.

In The Political Brain, Drew Westen points out that the Republicans have become the party of the sacred, appropriating not just the issues of God, faith, and religion, but also the sacred symbols of the nation such as the Flag and the military. The Democrats, in the process, have become the party of the profaneËœof secular life and material interests. Democrats often seem to think of voters as consumers; they rely on polls to choose a set of policy positions that will convince 51% of the electorate to buy. Most Democrats don’t understand that politics is more like religion than it is like shopping.

Appropriating is certainly the right word. It doesn’t matter if the candidate is a Catholic who served in Vietnam. They’ll call him a coward, and people will believe them because they’ve somehow been brainwashed into thinking that you “need” a conservative for national security. I wish I understood how that one happened (and I do, kind of, in a historical way, but not why people believed it then or now).

I’d rather have a party which relies on polls to pick policy the populace actually wants to get elected than one which lies and throws mud at their opponents. Regardless, I’d hardly call the primary process (which both parties have) “like shopping.” I’d say it’s “letting the people choose their candidate” (even if I do favour instant runoff voting instead).

Religion and political leadership are so intertwined across eras and cultures because they are about the same thing: performing the miracle of converting unrelated individuals into a group. Durkheim long ago said that God is really society projected up into the heavens, a collective delusion that enables collectives to exist, suppress selfishness, and endure. The three Durkheimian foundations (ingroup, authority, and purity) play a crucial role in most religions. When they are banished entirely from political life, what remains is a nation of individuals striving to maximize utility while respecting the rules. What remains is a cold but fair social contract, which can easily degenerate into a nation of shoppers.

Suppress selfishness, hah. None of these statements can be backed up. At all.

To say that a society which strives to maximizes utility while respecting the rules is a bad thing is unfathomable. What, exactly, is so awful about trying to make society better while respecting the rules? Is it better to break them for the person in authority? To try to make sure people are “pure” and keep “outsiders” out, even if they’d improve society? It can “easily degenerate into a nation of shoppers”? Because that’s happened so often. There are no comparable examples in history. Scandinavia is as close as it’s going to get for a while, and they’re not exactly “a nation of shoppers.”

The Democrats must find a way to close the sacredness gap that goes beyond occasional and strategic uses of the words “God” and “faith.” But if Durkheim is right, then sacredness is really about society and its collective concerns. God is useful but not necessary. The Democrats could close much of the gap if they simply learned to see society not just as a collection of individualsËœeach with a panoply of rights–but as an entity in itself, an entity that needs some tending and caring. Our national motto is e pluribus unum (“from many, one”). Whenever Democrats support policies that weaken the integrity and identity of the collective (such as multiculturalism, bilingualism, and immigration), they show that they care more about pluribus than unum. They widen the sacredness gap.

The Democrats “must” find a way, eh? Kind of assuming things, aren’t we? They seem to be doing fine.

Our national motto is, in fact, “E Pluribus Unum.” The point of it, you see, is that we are supposed to care just as much about “pluribus” as “unum.” That diversity is our strength. The core of our national identity. What makes America what it is. Society does need tending and caring. It needs a restoration of our rights. It needs to provide for every American, not just those with money, or who happened to be born well. It does not need Minutemen on the border, nor people crusading against being bilingual (it works just fine for Switzerland, Canada, and numerous other countries). It does not need some kind of “American” culture of fast-food restaurants and Wal-Marts (ever noticed how no matter where you are in the US, pretty much every interstate exit ramp looks the same as the last one, as far as restaurants and such go?). It needs to continue to assimilate, and grow, especially if we are to even attempt competition with the Chinese, Indians, and the Russians (the Russians most assuredly embrace multiculturalism).

I may think such competition is wasted effort at this point, but I still think this guy is wrong.

The ingroup/loyalty foundation supports virtues of patriotism and self-sacrifice that can lead to dangerous nationalism, but in moderate doses a sense that “we are all one” is a recipe for high social capital and civic well-being. A recent study by Robert Putnam (titled E Pluribus Unum) found that ethnic diversity increases anomie and social isolation by decreasing people’s sense of belonging to a shared community. Democrats should think carefully, therefore, about why they celebrate diversity. If the purpose of diversity programs is to fight racism and discrimination (worthy goals based on fairness concerns), then these goals might be better served by encouraging assimilation and a sense of shared identity.

Robert Putnam also thinks communal identity in the US is declining, which people thought in the 1920s, 1960s, and 1980s. Every time there’s a new medium of mass communication, people assume it’s the death of society. Ethnic diversity increases social isolation because a lot of people are racists. They can’t picture a shared community of people don’t have exactly the same beliefs. It’s an anachronism to assume that encouraging assimilation would decrease racism and discrimination (see: white flight). Rather, it must be reinforced that a changing national or community identity does not mean the end of it, any more than aging destroys your individual identity.

The purity/sanctity foundation is used heavily by the Christian right to condemn hedonism and sexual “deviance,” but it can also be harnessed for progressive causes. Sanctity does not have to come from God; the psychology of this system is about overcoming our lower, grasping, carnal selves in order to live in a way that is higher, nobler, and more spiritual. Many liberals criticize the crassness and ugliness that our unrestrained free-market society has created. There is a long tradition of liberal anti-materialism often linked to a reverence for nature. Environmental and animal welfare issues are easily promoted using the language of harm/care, but such appeals might be more effective when supplemented with hints of purity/sanctity.

Why must we “overcome” our “lower, grasping, carnal selves”, if I may ask? Liberals are hardly Gaius Germanicus or Nero. Our “carnal” selves are a necessary part of humanity (which is hardly discarded in Hindu beliefs, or even Abrahamic belief). Spirituality is not necessary. Protecting the environment has everything to do with protecting humanity. Our food chain. Our climate. The things man has depended on for thousands (or hundreds of thousands) of years. The end of these things would spell the collapse of society (at least, society as we know it), and possibly a death knell for the human race. Beyond which, conservatives, as a rule, treat those who treat the environment with sanctity (Greenpeace, Sierra Club) with anything from disregard to ridicule.

The authority/respect foundation will be the hardest for Democrats to use. But even as liberal bumper stickers urge us to “question authority” and assert that “dissent is patriotic,” Democrats can ask what needs this foundation serves, and then look for other ways to meet them. The authority foundation is all about maintaining social order, so any candidate seen to be “soft on crime” has disqualified himself, for many Americans, from being entrusted with the ultimate authority. Democrats would do well to read Durkheim and think about the quasi-religious importance of the criminal justice system. The miracle of turning individuals into groups can only be performed by groups that impose costs on cheaters and slackers. You can do this the authoritarian way (with strict rules and harsh penalties) or you can do it using the fairness/reciprocity foundation by stressing personal responsibility and the beneficence of the nation towards those who “work hard and play by the rules.” But if you don’t do it at allËœif you seem to tolerate or enable cheaters and slackers — then you are committing a kind of sacrilege.

Quasi-religious importance of the criminal justice system? M’kay. That’s the first time I’ve heard that one. The trick is — candidates are not police officers, prosecutors, or judges. They do not need to be “soft” or “hard” on crime, and those who have issued the most pardons (and for the most egregious crimes) have not been Democrats. There is no law against being a slacker, and “imposing costs” upon such would be against the law. Even phrases such as “slacker” betray a bias in the author.

“Work hard and play by the rules” doesn’t win for people either. Sadly, people who fuck other people over often have the power in life, and it’s always been that way. Maybe I just agree with Locke, but I think people are inherently selfish and evil. History agrees. Benevolent societies are eventually overtaken by dictators and those who can work the mob (Pericles, Caesar, Marius, Medicis, etc).

The government can do little against “slackers.” It can do something against cheaters, but that would go against the Temple of the Free Market, which conservatives cling to (as it’s most often corporate executives who are “cheaters”), and the majority of ethics scandals in the last 100 years have been… conservatives. This line of reasoning falls flat on its face.

If Democrats want to understand what makes people vote Republican, they must first understand the full spectrum of American moral concerns. They should then consider whether they can use more of that spectrum themselves. The Democrats would lose their souls if they ever abandoned their commitment to social justice, but social justice is about getting fair relationships among the parts of the nation. This often divisive struggle among the parts must be balanced by a clear and oft-repeated commitment to guarding the precious coherence of the whole. America lacks the long history, small size, ethnic homogeneity, and soccer mania that holds many other nations together, so our flag, our founding fathers, our military, and our common language take on a moral importance that many liberals find hard to fathom.

Liberals do not find it hard to understand the moral importance of the founding fathers. We find it hard to understand how conservatives can repeatedly spit on the ideals they stood for (and eloquently wrote), yet wrap themselves in the flag, as if it’s still worth something once we’ve abandoned everything it stood for. I’m going to assume that a “clear and oft-repeated commitment to guarding the precious coherence of the whole” means “talk about illegal immigrants a lot.”

Unity is not the great need of the hour, it is the eternal struggle of our immigrant nation. The three Durkheimian foundations of ingroup, authority, and purity are powerful tools in that struggle. Until Democrats understand this point, they will be vulnerable to the seductive but false belief that Americans vote for Republicans primarily because they have been duped into doing so.

I don’t think they’ve been duped. I think they want to be deluded. I think they want to hate everybody. I think they want to be told what to believe, and not think for themselves or examine the facts. They’re going to be surprised.

A more interesting title for his blog, I think, would have been “Why do hunters and fishermen vote for the people who consistently work to destroy the environment they hunt and fish in?” Admittedly, I don’t think the reasoning would be much different.

I’ll Say It First…

The new Dark Elves are broken, badly. I mean, they’re not as broken as the new High Elves, where an army comprised of 2×10 archers and the rest Swordmasters probably beats anything in the game (Empire/Dwarf gunlines aside) without even thinking about it. The problems? Well…

Firstly, I’m not happy about Executioners. They wrote up rules for Draichs (essentially flail+halberd, so they wouldn’t always strike last), then decided not to use them. The champ had the ability to add 25 pts of magic items, now removed (AFAICT). 50pt banner limit, now 25 pts. Always Strike First banner bumped from 25pts to 35pts, so they only unit that’d really benefit no longer can. The one plus is that they’re still Khainite. Oh, and they’re S4 now.

Secondly, Witch Elves. Knocked down to WS4 (though the points cost dropped a bit with it). I just don’t really see a reason to take them.

Cold One Chariots are no longer 2-for-1, so it’s a special choice for one chariot? I’ll pass. They’re now mount options for heroes, which is the only way I’d ever take one (though sorceresses cannot be in one, it seems).

Bolt throwers lost the -1 penalty for multiple shots and got bumped up to six shots, like the High Elf ones. I guess I can’t really complain about this, but I’d still rather not use them.

Shades became worthwhile (ability to take great weapons, and they’ve always been skirmishing scouts who can take repeater crossbows). Unfortunately, this makes them 20pts each, and a specials choice. I -may- end up taking some, situationally.

Spearelves are six points, seven with shields. Given the 25pt sacrifice dagger (which can cause panic tests), they may be worthwhile just to sacrifice.

Dark Riders got a points drop, not that they needed it.

Harpies are core. Flying skirmishers with two attacks apiece as core (no, they don’t count towards the min core units, but hey), and the rest of my units don’t care if they panic? Yeah… Balanced.

Corsairs are pretty much the same, except for the ability to take repeater handbows, and you now have to reroll the highest die on your flee. The handbows can be taken as multiples, but 8″ range, only function as a second hand weapon in the first round of combat? Seriously, if you’re that close, I’ll want to be charging (especially if I give one of the units the 25pt frenzy banner, which is corsair only). If not, a seven-wide frontage gives me 28 S3 shots that hit on a 5+ if you charge me. Fair.

As an aside (not that they needed it), repeater crossbows are now armor piercing like guns or WE longbows, except it’s also at long range. WTF. 28 shots against, say, Knights of the Realm ends up with 9ish hits, 4 wounds, one wound after save. It’s still suck against heavy cav, but it’d be murder against lightly armored troops, especially with the usual 10-wide crossbowmen units firing for a few turns.

Crossbowmen are 10pts per, 11 with shield. Woo.

The real WTFs?

Hydras are up to 7 attacks, with three attacks per handler (the handler attacks are also armor piercing). Seven WS4 S5 attacks plus six WS4 S3 armor piercing attacks with terror, and the hydra is T5 W7 3+ save 4+ regen? Shooting randomization automatically hits the hydra, and you can’t strike the handlers in close combat if you also touch the hydra. Plus a points drop to 175? Sure…

Black Guard are no longer 0-1 and no longer rare. They got Immune to Psychology and two attacks. This makes them, for all intents and purposes, a better anvil than hammerers (since fear causing units won’t autobreak them). Not impossible for them to soak a charge from Blood Knights if I can bait them (not hard).

Cold One Knights are equivalent to Grail Knights, only special and not 0-1. WS5, S4, 14″ charge, 2 attacks, lances, Cold Ones still cause fear (and are still S4 T4). Not Immune to Psychology, but big deal. Stupidity might play a role, but that’s always been the case. They’re probably more than worth their points now.

Witch Elves can no longer take magic items. Temple of Khaine stuff only. Black Lotus poison now makes me reroll 1s to wound, and the Rune of Khaine is +D3 attacks.

Reverse ward item is WTF. Armor of Eternal Servitude got cheaper. Manticores now have killing blow, and automagically frenzy+hate if the rider dies. They take a heroes choice to use them as a mount. Dark Elf Dreadlord (replaced Highborn) can easily get a 3+ save on the manticore + reverse ward w/4+ regen plus a magic item (Sword of Ruin to deny armor saves?). Not cool.

Cauldron of Blood is a mount for Death Hags (WE hero), making the cost of hero+CoB 200pts. No rare slot or even hero slot. It still causes terror, has crew (plus the hag), and gives them 4+ wards. Any Khainite units (Execs, Witch Elves, Assassins, maybe Cold One Knights) within 12″ are stubborn. Every magic phase, it can put one of the following effects on ANY unit within 24″ which cannot be dispelled or stopped in any way:

  • Killing Blow
  • 5+ Ward Save
  • +1 Attack

Sorceresses have access to Metal and Fire now as well. Ring of Hotek (miscast on any doubles) got the range boosted to 12″, which may make it worth taking.

Assassins are no longer characters. They’re unit upgrades, but can still take stuff from the Temple of Khaine (no magic items). Stat line is the same, but they cost 45 points less or something. Once revealed, they’re just like current assassins (including the ability to leave the unit). The problem I see with this? Lone assassin makes Grail Knights worthless.

Rune of Khaine (+D3 attacks), Hand of Khaine (killing blow), Black Lotus (reroll 1s to wound). Pop him out and stick him in front of a unit that’s going to be charged, and keep him in range of the cauldron. Grail Knights charge. He hits on 3s, rerolls failed to hit rolls (did I mention that the ENTIRE ARMY has Hatred now) on the first turn with anywhere from 5-7 attacks (3+extra hand weap + D3 attacks), reroll ones, sixes are killing blow, S4. If he even kills ONE knight and can’t get struck back against, he’s perfectly safe. Even if not, the knight needs to wound twice. He loses combat and needs to roll under a 10. I flank charge the grail knights the next turn (or let the assassin tie them up forever). Not cool.

Granted, Black Guard with Cauldron Killing Blow and the ASF banner or the Hydra banner would be incredibly nasty (or both, really), giving them anywhere from 21-36 always-strike-first always-reroll-failed-hits S4 Killing Blow attacks (with both) on a 7-wide frontage, immune to psychology. Witch Elves, similarly, could have 29-43 reroll-failed-hits poisoned killing-blow attacks with a BSB (Hydra banner) and Cauldron, plus whatever Temple Gifts the Hag and BSB had. Cold One Knights look just as bad.

This is not my idea of balance, and this says nothing of the (likely) 4 units of Dark Riders, at least two terror causing units (Cauldron, Hydra, maybe Manticore, maybe lord with the Cold One Knights on a chariot or Cold One), flying skirmishing core units, etc. 50% of my army being stubborn? Cool…

WTB more Rare Units and less specials, and I’m guessing I’ll almost never have Executioners (sadly. I love the models, but Black Guard or Witch Elves simply do it just as well with the Cauldron to give them Killing Blow, and Cold One Knights being viable again gives me S6 on the charge).

Did I mention that a L4 sorceress can generate a ridiculous amount of power dice? Not as many as Vampire Counts, but the “everybody-has-it” spell (even if I don’t take Dark) dives me D3+1 power dice (I suffer wounds if I don’t use them all). By herself, 10-14 power dice per phase is not unreasonable.

Something is not right here. It might be fun against Vampire Counts (a cheesy High Elf list probably still wouldn’t be). I have difficulty seeing Dwarves competing without going ludicrously gunline, to say nothing of Tomb Kings or Bretonnians (since, as noted, I could simply shut down lances with lone assassins, WTF).

Just a taste –

Since I’ve been bitching to Dan lately about the code I inherited, here are a few examples:

System.DateTime answer = today.AddDays(-1);
YesterdayTemp = answer.ToString("dd");
TodayTemp = today.ToString("dd");
string TodayMonth = today.ToString("MM");
string currlogdir = @"C:\Faxserver\Logs\";
int currHour = DateTime.Now.Hour;
int currMinute = DateTime.Now.Minute;
int currSecond = DateTime.Now.Second;
if (currHour == 00 && currMinute == 00 && currSecond == 00)
{
      if (File.Exists(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt"))
      {
           DateTime tempcreatetime6 = File.GetCreationTime(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt");
           string tempday6 = tempcreatetime6.ToString("dd");
           string tempmonth6 = tempcreatetime6.ToString("MM");
           if (TodayTemp != tempday6 || TodayMonth != tempmonth6)
           {
                File.Delete(currlogdir + "AutoFaxErrorCountLog_" + TodayTemp + ".txt");
           }
      }
      FaxCountLog(YesterdayTemp, lblFaxesTodayDisplay.Text);
      FaxErrorCountLog(YesterdayTemp, lblErrorsTodayDisplay.Text);
      lblFaxesYesterdayDisplay.Text = lblFaxesTodayDisplay.Text;
      lblErrorsYesterdayDisplay.Text = lblErrorsTodayDisplay.Text;
      TextlblDisplayTodayAdd("0");
      TextlblDisplayTodayErrorAdd("0");
      TextlblDisplayHourAdd("0");
      TextlblDisplayHourErrorAdd("0");
      FaxesHour = 0;
      ErrorsHour = 0;
      FaxesToday = 0;
      ErrorsToday = 0;
}
if (currHour != 00 && currMinute == 00 && currSecond == 00)
{
     FaxCountLog(TodayTemp, lblFaxesTodayDisplay.Text);
     FaxErrorCountLog(TodayTemp, lblErrorsTodayDisplay.Text);
     TextlblDisplayHourAdd("0");
     TextlblDisplayHourErrorAdd("0");
     FaxesHour = 0;
     ErrorsHour = 0;
}
if (ElapsedHours == 23 && ElapsedMinutes == 59)
{
     ElapsedDays += 1;
     ElapsedHours = 00;
     ElapsedMinutes = 00;
     ElapsedSeconds = 00;
}
if (ElapsedMinutes == 59)
     {
     ElapsedHours += 1;
     ElapsedMinutes = 00;
}
if (ElapsedSeconds == 59)
{
     ElapsedMinutes += 1;
     ElapsedSeconds = 00;
}
else
{
     ElapsedSeconds += 1;
}
if (ElapsedHours >= 0 && ElapsedHours <= 9)
{
     ElapsedHoursTemp = "0" + ElapsedHours;
}
else
{
     ElapsedHoursTemp = ElapsedHours.ToString();
}
if (ElapsedMinutes >= 0 && ElapsedMinutes <= 9)
{
     ElapsedMinutesTemp = "0" + ElapsedMinutes;
}
else
{
     ElapsedMinutesTemp = ElapsedMinutes.ToString();
}
if (ElapsedSeconds >= 0 && ElapsedSeconds <= 9)
{
     ElapsedSecondsTemp = "0" + ElapsedSeconds;
}
else
{
     ElapsedSecondsTemp = ElapsedSeconds.ToString();
}
blElapsedTimeDisplay.Text = ElapsedDays + " Day(s), " + ElapsedHoursTemp + ":" + ElapsedMinutesTemp + ":" + ElapsedSecondsTemp;
}

Replaced with:

int totalSeconds = DateTime.Today.TotalSeconds;
Stopwatch elapsedTime = new Stopwatch();
elapsedTime.Start();
public class Log {
     public void Rotate (FileInfo logFile)
      {
          if ((logFile.Exists) && (logFile.LastAccessTime.Date != DateTime.Today.Date))
          {
               logFile.Delete()
          }
      }
      /* snip */
}
public class Reset {
     public void Labels(bool day, bool hour)
          {
                lblFaxesYesterdayDisplay.Text = lblFaxesTodayDisplay.Txt;
           /*snip*/
           }
     public void Counts(bool day, bool hour)
     {
                /*snip*/
     }
}
Reset reset = new reset;
FileInfo fileErrorLog = new FileInfo(logdir + "AutoFaxErrorCountLog_" + DateTime.Today.Day + ".txt");
if (totalSeconds % 3600 == 0)
{
     if (totalSeconds == 0)
     {
          log.Rotate(fileErrorLog);
          reset.Labels(true, true);
          reset.Counts(true, true);
     }
     else 
     {
          reset.Labels(false, true);
          reset.Labels(false, true);
     }
reset.Dispose();
}
TimeSpan ts = elapsedTime.Elapsed;
lblElapsedTimeDisplay.Text = String.Format("{0} Day(s), {1:00}:{2:00}:{3:00}", ts.Days, ts.Hours, ts.Minutes, ts.Seconds);
}

No, I’m not using Delgates until I can decouple this:
AutoFax.Form1.btnCleanErrors_Click(object, System.EventArgs) AutoFax.Form1.btnRestartPrintService_Click(object, System.EventArgs)
AutoFax.Form1.btnStart_Click(object, System.EventArgs)
AutoFax.Form1.CheckRdy() AutoFax.Form1.CheckSaved()
AutoFax.Form1.ChooseThreads(int)
AutoFax.Form1.CreateNotAndRdy()
AutoFax.Form1.Dispose(bool)
AutoFax.Form1.ErrorOutSecondsNumberGet()
AutoFax.Form1.ErrorOutSecondsNumberSet(int)
AutoFax.Form1.FaxCountLog(string, string)
AutoFax.Form1.FaxErrorCountLog(string, string)
AutoFax.Form1.FaxLog(string, bool)
AutoFax.Form1.Form1()
AutoFax.Form1.Form1_Exit(object, System.EventArgs)
AutoFax.Form1.GetFaxCount()
AutoFax.Form1.InitializeComponent()
AutoFax.Form1.Main()
AutoFax.Form1.menuItemAbout_Click(object, System.EventArgs)
AutoFax.Form1.menuItemShutDown_Click(object, System.EventArgs)
AutoFax.Form1.OnChanged(object, System.IO.FileSystemEventArgs)
AutoFax.Form1.ProcessFile(string, bool)
AutoFax.Form1.ProgressIncrement(int)
AutoFax.Form1.Rdy()
AutoFax.Form1.read_ini_settings()
AutoFax.Form1.SendAlert(string)
AutoFax.Form1.SendEmail(string)
AutoFax.Form1.SendFax()
AutoFax.Form1.SendFaxToProgram(string, string)
AutoFax.Form1.Textbox1Add(string)
AutoFax.Form1.Textbox1Append(string)
AutoFax.Form1.TextbtnStartadd(string)
AutoFax.Form1.TextlblDisplayHourAdd(string)
AutoFax.Form1.TextlblDisplayHourErrorAdd(string)
AutoFax.Form1.TextlblDisplayTodayAdd(string)
AutoFax.Form1.TextlblDisplayTodayErrorAdd(string)
AutoFax.Form1.TextlblFaxesAdd(string)
AutoFax.Form1.TextlblQueAdd(string)
AutoFax.Form1.TextlblSendAdd(string)
AutoFax.Form1.timer1_Tick_1(object, System.EventArgs)
AutoFax.Form1.timer2_Tick(object, System.EventArgs)
AutoFax.Form1.timerHBT_Tick(object, System.EventArgs)

Who needs classes? That Form1 class can just get every method known to man.
More joy:

String rdydir = @"C:\RDY\INCOMING\";
String notdir = @"G:\NOT\";
String str1 = path.ToUpper();
String delim = rdydir;
String delim1 = "NOT.RDY";
String str2 = str1.Trim(delim.ToCharArray());
if ((File.GetAttributes(path) & FileAttributes.Normal) == FileAttributes.Hidden)
{
    File.Delete(path);
    FaxLog("deleted file: [" + path + "]", false);
}
else
{
File.SetAttributes(path, FileAttributes.Normal);
File.Delete(path);
FaxLog("deleted file: [" + path + "]", false);
}
String str3 = str2.Trim(delim1.ToCharArray());
String NotFile = str3 + ".NOT";
TextlblQueAdd("looking for: [" + NotFile + "]");

Replacement:

 string notdir = @"G:\NOT\";
string NotFile = Regex.Replace(path, @".*\\(\w+\d+).*", "$1" + ".NOT").ToUpper();

This is just…

string templine = sr.ReadLine();
sr.Close();
if (templine != null)
{
      int test = templine.IndexOf("[");
      int test2 = templine.IndexOf("]");
      string test3 = templine.Substring(test + 1, test2 - 1 - test);
      if (test3 != null)
      {
            Regex reDateTime = new Regex(@"^(\d{2})/(\d{2})/(\d{4}) (\d{2}):(\d{2}):(\d{2})$");
            if (reDateTime.IsMatch(test3))
            {
                  int test4 = test3.IndexOf(" ");
                  string test5 = test3.Substring(0, test4);
                  if (test5 == TodayMonthTempStr + "/" + TodayDayTempStr + "/" + TodayYearTemp)
                  {
                        filedatetimeisthesame = true;
                        // file exists and it has today's date, so just append to file
                        FileStream fs2 = new FileStream(templogpath + BeginLogName + MiddleLogName + TodayDayTempStr + ".txt", FileMode.Append, FileAccess.Write);
                        StreamWriter m_streamWriter2 = new StreamWriter(fs2);
                        string FaxLogDate2 = DateTime.UtcNow.ToUniversalTime().ToShortDateString();
                        string FaxLogTime2 = DateTime.UtcNow.ToUniversalTime().ToLongTimeString();
                        m_streamWriter2.WriteLine("[" + FaxLogDate2 + " " + FaxLogTime2 + "], " + FaxLog);
                        m_streamWriter2.Flush();
                        m_streamWriter2.Close();
                        fs2.Close();
                   }
              }
         }
    }
}

Replace:

if (fileAutoFaxLog.LastWriteTime.Date == DateTime.Today.Date) 
{  
      filedatetimeisthesame = true;
       // file exists and it has today's date, so just append to file
       FileStream fs2 = new FileStream(@"C:\Faxserver\Logs\AutofaxLog_" + DateTime.Now.ToString("dd") + ".txt", FileMode.Append, FileAccess.Write);
       StreamWriter m_streamWriter2 = new StreamWriter(fs2);
       m_streamWriter2.WriteLine("[" + DateTime.Now.ToString() + "], " + FaxLog);
       m_streamWriter2.Flush();
       m_streamWriter2.Close();
       m_streamWriter2.Dispose();
       fs2.Close();
       fs2.Dispose();
}

Lots of these (15 or so) in the code:

DateTime hbtTime = DateTime.UtcNow.ToUniversalTime();
string hbtMonth = hbtTime.Month.ToString();
if (hbtMonth.Length == 1)
{
     hbtMonth = "0" + hbtMonth;
}
string hbtDay = hbtTime.Day.ToString();
if (hbtDay.Length == 1)
{
     hbtDay = "0" + hbtDay;
}
string hbtYear = hbtTime.Year.ToString();
string hbtHour = hbtTime.Hour.ToString();
if (hbtHour.Length == 1)
{
     hbtHour = "0" + hbtHour;
}
string hbtMinute = hbtTime.Minute.ToString();
if (hbtMinute.Length == 1)
{
     hbtMinute = "0" + hbtMinute;
}
     string hbtSecond = hbtTime.Second.ToString();
if (hbtSecond.Length == 1)
{
      hbtSecond = "0" + hbtSecond;
}
string strHBTtime = hbtMonth + "/" + hbtDay + "/" + hbtYear + ", " + hbtHour + ":" + hbtMinute + ":" + hbtSecond;

Which is really:

string strHBTtime = DateTime.Now.ToString("MM/dd/yy HH:mm:ss");

This isn’t mentioning all the other weird shit in the code, like queues are always:

object whythefuck = queue.Peek();
string eh = whythefuck.ToString();
queue.Dequeue();

Rather than:

string morereadablelesswaste = queue.Dequeue().ToString();

Flags being set in 1,000 places to do the same thing. Every catch is handled the same way, but lots of copy+paste code rather than passing the Message to a method to handle it (better to just repeat ad-nauseum). Re-inventing booleans with lower performance by setting strings to “true” or “false” then comparing those (they never have any other value). No Dipose() or Finalize(). Ever. No comments. Pointless variable names that mean nothing, like tmpStr14. That also brings up: Hungarian notation. Gods, why use it in a strongly-typed language?

What started off as a 20 minute “add a method to email me” is going to be a “rework 4500 lines of code into (probably) 2000, by cleaning up the trash everywhere then adding classes and delegates.” Joy.