Oh, shit.

Well, the bailout failed. Not that I was really for it, and all estimates indicated that they needed a lot more than $700 billion dollars, but I’m still not sure how to react to this. I know the Fed just released another $630 billion into the Term Auction Facility (emergency lending) and currency swaps markets today, but it seems that the Fed’s balance sheet is just about tapped anyway. So, as it stands, the Dow is down 7%, and the S&P is down 12%. It’s not even the end of trading.

It’s entirely possible that they’ll attempt to craft a new bill which has more… populist leanings, in the hopes of getting more Democratic support (since they almost certainly won’t get the Republicans). It might pass. It might be a better bill. It might not be. It seems that, in any case, they’re going to sacrifice something that might have worked for ideology and political expediency. The Asians were already leery of lending us more money. Germany and France flat-out said no. Britain’s in the middle of their own crisis. What the fuck happens now?

The US dollar disappears as a reserve currency? Commodities markets start to be denominated in something other than US dollars? Our economy crashes and burns even more? It should be obvious (and should have been obvious) to people that we cannot march inexorably upwards. People in the US (myself included) don’t (or can’t) save any substantial amounts of money. However, at some point, things have to go down when the bubble bursts. It doesn’t look like we’re going to end up creating a new one like we did when the .com bubble burst (hello, housing), thus the incredible amounts of leveraged debt US companies are holding has to crash. We’ve already had the largest bank failure in US history in the last week, Wachovia’s gone (Citigroup, Chase, and Bank of America are now huge entities), most of the investment banks are gone, and there’s still a shitload of debt hanging around.

The Republican plan of cutting the capital gains tax is idiotic. By definition, a company taking losses doesn’t have any gains, but they’re going to fiddle while Rome burns, so to speak. Nobody seems to know what’s going to happen. There may be another bill. There may not be. Bankers are finally offing themselves. It remains to be seen how the markets do for the rest of the day, I guess, but the TED and Libor are way fucking higher than the Fed’s nominal rates on T-bills. Welcome to a new depression?

It would be nice to see government being proactive rather than reactionary. We’re going down, and it looks like a lot of the world might follow for a while (Hong Kong, Russia, Japan, and London stock exchanges are down precipitously over the last week also). Seriously. Propose legislation for a new Civilian Conservation Corps (e.g. rework the Job Corps). Bring back the regulation you abolished over the last 20 years. Take over FHA backed mortgages, and do… something with them (forced refinancing to current value, with the government pocketing any equity they may acquire until such time as they’re paid off or something). Raise the prime rate to stave off inflation (it’s clear that cutting it isn’t going to let us buy our way out of a recession through liquidity at this point). Actually try to solve the problems which are inevitably going to come up before you have to.

Failing that, torches and pitchforks, maybe.

How not to write an XML file

I’ll be honest. I don’t like XML. I don’t like SOAP (REST is far nicer in my opinion), since it manipulates the HTTP spec to do things it was never meant to do. Raw sockets and bit twiddling seem like a more logical extension, just that port 80 happens to be open on most corporate firewalls, so SOAP and CORBA have taken off. Inasmuch as I may dislike XML, though, it has its uses. Representing a datastream on for sets where CSV doesn’t really make sense, and YAML isn’t available, and it’s not that hard to deal with.

The ArmyBuilder developers seem to have squeezed SGML into an XML doctype somehow, and the roster files are littered with references I can’t quite make out. Yes, ArmyBuilder can export to XML, I guess, but it leads to XSL from hell. In some ways, I would have preferred to rip apart a binary format with a hex editor, as long as the data was formatted logically.

This, for instance:

<link id="dwWarCrew" count="1" actual="1" script="0" sequence="106" pseudo="no" totalcost="0" \ 
name="Crew" category="Equip" visible="no" sourceid="dwBoltThrw" sourceindex="1"></link>

Or this:

<ruleset context="dwSubtype" ruleset="dwDwarves" contextname="Army Subtype" rulesetname="Dwarf Army"/>

Is not formatted logically. The second record, as you can see, uses XML attributes rather than nodes for everything, which kinda defeats the point. XSL to parse ArmyBuilder’s XML output? Ahh…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
	<xsl:output method="html" version="1.0" encoding="UTF-8" indent="yes"/>
	<xsl:variable name="newlinefeed"><![CDATA[
]]></xsl:variable>
	<xsl:variable name="statCountGlobal" select="count(document/definition/stat_def)"/>
 
	<xsl:template match="/">
	<xsl:variable name="namedModelCount" select="document/composition/@model_count"/>
	<xsl:variable name="actualModelCount" select="sum(//regiment/@model_count)"/>
	<xsl:variable name="actualPoints" select="sum(//regiment/@cost[not(contains(.,'['))])"/>
	<xsl:value-of select="concat(/document/summary/@race_name,': ',$actualPoints, /document/definition/@points_abbrev,' - ',$actualModelCount,' ')"/> Models<xsl:value-of select="$newlinefeed"/>
		<xsl:for-each select="/document/composition/comp_entry">
		<xsl:variable name="groupName" select="@group_name"/>
		<xsl:if test="/document/roster/top_level/regiment[@composition = $groupName]">
			<xsl:variable name="unit" select="@group_name"/>
			<xsl:for-each select="/document/roster/top_level/regiment[@composition = $unit]">
				<xsl:apply-templates select="." mode="top_level">
					<xsl:with-param name="regDepth">
						<xsl:choose>
							<xsl:when test="position()=1"><xsl:value-of select="count(/document/roster/top_level[regiment/@composition = $unit]//regiment)"/></xsl:when>
							<xsl:otherwise>0</xsl:otherwise>
						</xsl:choose>
					</xsl:with-param>
				</xsl:apply-templates>
			</xsl:for-each>	
		</xsl:if>
		</xsl:for-each>
	</xsl:template>
	<xsl:template match="regiment" mode="top_level">
		<xsl:param name="regDepth">
			<xsl:value-of select="count(..//regiment)"/>
		</xsl:param>
		<xsl:variable name="statCountLocal" select="count(stat)"/>
		<xsl:variable name="fsib" select="preceding-sibling::node()"/>
		<xsl:variable name="composition" select="@composition"/>
		<xsl:if test="$regDepth > 0">
		<xsl:choose>
			<xsl:when test="not($composition = $fsib/@composition)">
				<xsl:value-of select="$composition"/>(<xsl:value-of select="/document/composition/comp_entry[@group_name = $composition]/@percentage"/>)<xsl:value-of select="$newlinefeed"/>
			</xsl:when>
		</xsl:choose>
		</xsl:if>
		<xsl:value-of select="concat('[',@model_count,'] ')"/>
		<xsl:variable name="itemcost">
			<xsl:if test="@cost"><xsl:value-of select="@cost"/></xsl:if>
		</xsl:variable>
		<xsl:variable name="retinuecost">
			<xsl:call-template name="getRetinueCost">
				<xsl:with-param name="retinuecostSum" select="0"/>
				<xsl:with-param name="current" select="regiment[position()=1]"/>
				<xsl:with-param name="rest" select="regiment[position()!=1]"/>
			</xsl:call-template>
		</xsl:variable>
		<xsl:variable name="transportcost">
			<xsl:if test="regiment[@stat_set = 0]/@cost"><xsl:value-of select="substring-after(substring-before(regiment[@stat_set = 0]/@cost,']'),'[')"/></xsl:if>
		</xsl:variable>
		<xsl:choose>
			<xsl:when test="$retinuecost and @model_count=1 and @composition='HQ'">
					<xsl:value-of select="concat('[',$itemcost - $retinuecost,'] ')"/>
			</xsl:when>
			<xsl:when test="$transportcost != ''">
					<xsl:value-of select="concat('[',$itemcost - $transportcost,'] ')"/>
			</xsl:when>
			<xsl:otherwise>
					<xsl:value-of select="concat('[',$itemcost,'] ')"/>
			</xsl:otherwise>
		</xsl:choose>
 
		<xsl:value-of select="substring-before(name,' (')"/>
		<xsl:if test="@model_count=1 and @composition='HQ'">
			<xsl:text disable-output-escaping="yes">(IC)</xsl:text>
		</xsl:if>
		<xsl:text disable-output-escaping="yes">: </xsl:text>
 
		<xsl:for-each select="item">
			<xsl:variable name="namedItem" select="name"/>
			<xsl:choose>
				<xsl:when test="count(../item[name=$namedItem]) > 1"><xsl:value-of select="concat($namedItem,'(x',count(../item/name[.=$namedItem]),');')"/></xsl:when>
				<xsl:otherwise><xsl:value-of select="concat(name,';')"/></xsl:otherwise>
			</xsl:choose>
		</xsl:for-each>
		<xsl:for-each select="choice"><xsl:value-of select="concat(name,';')"/></xsl:for-each>
		<xsl:value-of select="$newlinefeed"/>
 
		<xsl:for-each select=".//regiment[not(../@category = 'Wargear Item')] ">
			<xsl:apply-templates select="." mode="regiment" />
		</xsl:for-each>
		<xsl:value-of select="$newlinefeed"/>
	</xsl:template>
	<xsl:template match="regiment" mode="regiment">
		<xsl:variable name="statCountLocal" select="count(stat)"/>
		<xsl:variable name="depth">
			<xsl:choose>
				<xsl:when test="../../@stat_count=1"><xsl:value-of select="number(@depth)-1" /></xsl:when>
				<xsl:otherwise><xsl:value-of select="@depth" /></xsl:otherwise>
			</xsl:choose>
		</xsl:variable>
		<xsl:value-of select="concat('[',@model_count,'] ')"/>
 
		<xsl:variable name="itemcost">
			<xsl:if test="@cost"><xsl:value-of select="substring-after(substring-before(@cost,']'),'[')"/></xsl:if>
		</xsl:variable>
		<xsl:variable name="transportcost">
			<xsl:if test="regiment[@stat_set = 0]/@cost"><xsl:value-of select="substring-after(substring-before(regiment[@stat_set = 0]/@cost,']'),'[')"/></xsl:if>
		</xsl:variable>
		<xsl:choose>
			<xsl:when test="((../@composition='HQ' and @depth = 1) or (@depth = 0)) and (regiment[@stat_set = 0])">
					<xsl:value-of select="concat('[',$itemcost - $transportcost,'] ')"/>
			</xsl:when>
			<xsl:when test="@stat_set = 0">
					<xsl:value-of select="concat('[',$itemcost,'] ')"/>
			</xsl:when>
			<xsl:when test="../@composition = 'HQ' and ../@stat_count > 1">
					<xsl:value-of select="concat('[',$itemcost,'] ')"/>
			</xsl:when>
		</xsl:choose>
 
		<xsl:call-template name="formatName"><xsl:with-param name="strName" select="concat(name,': ')"/></xsl:call-template>
 
		<xsl:for-each select="item[not(name = preceding-sibling::item/name)]">
			<xsl:variable name="namedItem" select="name"/>
			<xsl:choose>
				<xsl:when test="count(../item[name=$namedItem]) > 1"><xsl:value-of select="concat($namedItem,'(x',count(../item[name=$namedItem]),');')"/></xsl:when>
				<xsl:otherwise><xsl:value-of select="concat(name,';')"/></xsl:otherwise>
			</xsl:choose>
		</xsl:for-each>
		<xsl:for-each select="choice"><xsl:value-of select="concat(name,';')"/></xsl:for-each>
		<xsl:value-of select="$newlinefeed"/>
 
	</xsl:template>
	<xsl:template name="getRetinueCost">
		<xsl:param name="retinuecostSum" />
		<xsl:param name="current" />
		<xsl:param name="rest" />
		<xsl:variable name="curCost">
			<xsl:choose>
				<xsl:when test="contains($current/@cost,'[')"><xsl:value-of select="substring-after(substring-before($current/@cost,']'),'[')" /></xsl:when>
				<xsl:otherwise><xsl:value-of select="$current" /></xsl:otherwise>
			</xsl:choose>
		</xsl:variable>
		<xsl:choose>
			<xsl:when test="$current">
				<xsl:call-template name="getRetinueCost">
					<xsl:with-param name="retinuecostSum" select="$retinuecostSum + $curCost"/>
					<xsl:with-param name="current" select="$rest[position()=1]"/>
					<xsl:with-param name="rest" select="$rest[position()!=1]"/>
				</xsl:call-template>
			</xsl:when>
			<xsl:otherwise>
				<xsl:value-of select="$retinuecostSum" />
			</xsl:otherwise>
		</xsl:choose>
	</xsl:template>
	<xsl:template name="halfCost">
		<xsl:param name="itemCost"/>
		<xsl:value-of select="round($itemCost div 2)" />
	</xsl:template>
	<xsl:template name="formatName">
		<xsl:param name="strName"/>
		<xsl:choose>
			<xsl:when test="contains($strName,' (')"><xsl:value-of select="substring-before($strName,' (')"/></xsl:when>
			<xsl:otherwise><xsl:value-of select="$strName"/></xsl:otherwise>
		</xsl:choose>
	</xsl:template>
	<xsl:template name="doReplaceCar">
		<xsl:param name="text"/>
		<xsl:param name="replace"/>
		<xsl:param name="by"/>
		<xsl:choose>
			<xsl:when test="contains($text, $replace)">
				<xsl:value-of select="substring-before($text, $replace)" disable-output-escaping="yes"/>
				<xsl:value-of select="$by" disable-output-escaping="yes"/>
				<xsl:call-template name="doReplaceCar">
					<xsl:with-param name="text" select="substring-after($text, $replace)"/>
					<xsl:with-param name="replace" select="$replace"/>
					<xsl:with-param name="by" select="$by"/>
				</xsl:call-template>
			</xsl:when>
			<xsl:otherwise>
				<xsl:value-of select="$text" disable-output-escaping="yes"/>
			</xsl:otherwise>
		</xsl:choose>
	</xsl:template>
</xsl:stylesheet>

No, I am not writing a stylesheet like that again, and the one to parse the roster files would be far more complicated.

The problem with the roster file, fundamentally, is that it’s too tightly linked with ArmyBuilder. That makes sense, in a way, but is still irksome. The <link> elements don’t have any nodes under them, just assloads of attributes, and it’s not easy to figure out which ones I am interested in:

<link id="HeavyArmor" count="1" actual="1" script="0" sequence="26" pseudo="no" totalcost="0" name="Heavy Armor" category="Equip" \
 abbrev="Hv" description="5+ Armor Save" equipment="yes" footnote="yes" sourceid="dwWarrVet" sourceindex="5"></link>

Versus ones I’m not interested in:

<link id="ItemCost" count="1" actual="1" script="0" sequence="28" pseudo="no" totalcost="0" name="Item Cost Worker" category="Equip"\
 visible="no" sourcetype="3" sourceid="Globals" sourceindex="1"></link>

Without passing a long hashlist of element.attribute[$thing] values, or specifically excluding anything with “Helper” or “Worker” or whatever in the name, etc. Not to mention it’s formatted as:

<document>
  <squad>
  <!-- unit name and cost is here -->
    <entity>
    <!-- unit stats are here, along with composition and whatnot -->
      <link>
      <!-- sometimes there's nothing of note in the link tags -->
         <entity>
         <!-- this might be a magic item, warmachine crew, magic banner, champion, and probably other stuff, but is not easily  \
         identified, and there may be more than one -->
            <link>
            <!-- might be info for whatever is in entity, might be a helper which I don't want -->
            </link>
            <unitstat>
            <!-- if it's crew, champion, whatever, stats would be here, but this node may not exist -->
            </unitstat>
        </entity>
      </link>
   </entity>
 </squad>
</document>

The problem with some of these is that by the mantra of whoever wrote ArmyBuilder, champions fall into the “Equip” category. There is, in fact, a “isunit” attribute, but it isn’t set to yes anywhere. Only set to “no” for items, which I can’t figure out (unless there’s some kind of magic item which qualifies as a unit you can add? I don’t know).

I’ve got a parser that works in Ruby written, but I haven’t converted it to C# yet. Also, I’ve not tested it against anything that might have more complicated schema than dwarves: mounted units, chariots, to check if it’s undead/daemon/greenskin and see if special rules apply (since not everything in the army is guaranteed to be), embedded assassins, magic, et al. Sadly, the only roster I’ve got at work is for dwarves, so I’ll have to dump some more output from ArmyBuilder and run the parser against it to see how it handles it.

Any other niche cases either of you can think of that may have specific rules? I’m going to try to stabilize the parser and get it to properly validate every army type, then move it to .NET

Also, thinking about it, I’m utterly convinced that snapping things to some kind of a grid is the only real feasible solution. Querying the object via System.Drawing or GDI might work, but I’m not sure how accurate the pixel mapping is. At any rate, for things like the Lance Formation, line of sight on skirmishers, determining base contact for champions/characters embedded, reforming the unit, and templates, a grid seems like the only way to go without doing occlusion detection (for the templates). Convert inches to millimeters, and make it 1mm x 1mm squares or something.

New Plan

I spend way too much time doing, well, nothing of consequence. Between Slashdot, Reddit, and Fark, I probably spend 30% of my waking time checking news (to say nothing of all the feeds Google Reader). I’ve come to the conclusion that this is really a waste. Ironically, a story on Slashdot pushed me in that direction (namely, that Stanford is starting to put courses online similar to MIT’s OpenCourseWare, except that they’re trying to only post complete courses). What the hell am I doing with my time?

To be sure, keeping up on news has its high points. It’s nice to be informed. Really, it is. It occupies my free time when I’m at work, and I think that’s part of the problem. There are better things I could be doing with my time. I don’t intend to stop reading the news at any point, but Google Reader will keep me up on that. I do intend to stop using my laptop for 99% of things at home, and actually migrate down to my desk (in front of my workstation and my server). Why? Ahh… so I can stop getting distracted by the television.

As irritating as it is to me (and the power bills are irksome), the television in my bedroom is on 95% of the month. Heather watches a shitload of television, and it’s too hard to lose my concentration with $whatever playing in the background constantly (even if it is Animal Cops or something). I watch very little TV. Good Eats. Mythbusters. The Daily Show. Battlestar Galactica (which will be over soon enough, at any rate). The Wire (which is already done). That 90 minutes between Good Eats and The Daily Show gets sucked into random channel flipping too often. Even C-SPAN is distracting at times.

At this point, I have a fairly good grasp of Hebrew (grammar, if not vocabulary). The intention, I suppose, is to start doing something else with my spare time. Going to school (which I am) is one thing, but my course load is light of necessity (financial necessity, more than anything, since I don’t want an assload of student loans while I still have credit card debt, and $work only covers $5,000 a year), and it’s not challenging. At this point, even though experience, aptitude, and intelligence are helpful in getting jobs, the piece of paper still counts for a lot. In particular, not having one excludes me from getting a state/federal job until such time as I have 6 years of experience or whatever. Also, it’s surprisingly difficult to move into a development role when I have years of systems administration experience, with scripting/development as a secondary duty.

I miss learning, I suppose. I do, however, have nearly as much free time in the day as I spent in high school (at the time). This free time is usually sucked up with errands, chores, and news reading when I can fit it in. That’s got to stop. What I’m planning to do now is to divide my time into blocks. I should be able to fit at least two hours a day into two distinct subjects, and possibly more on top of that. Certainly, I didn’t spend that much time actually learning in high school (largely because high school was a waste of time where the majority of my classes had nothing to teach me).

The plan now, I think, is to stop by Half-Price Books and grab some textbooks. I don’t have to take calculus in college, since I managed to test out of it. I kinda think that was by the skin of my teeth, though, since my grasp of it is hazy at this point (other than basic differentiation and integration). I’ll probably start with Calc I, then Calc II, Multi-Variable, Diff. Eq., Linear Algebra, and a couple levels of statistics thrown in (maybe after Diff. Eq.). I figure that at two hours a day, it’ll take me a month to a month and a half to get through a textbook, including the exercises, of course.

I’ll be doing Arabic contemporaneously. I mean, I have a fair amount of Dan’s coursework from DLI that’s been sitting around for years. It’s time to get to it. I don’t honestly have any idea how long I expect that to take me. I mean, having the entire MSA Basic course plus the MSA Intermediate course, and some dialect CDs besides? It could be quite an undertaking. Also at two hours a day, I guess. Given that the course at DLI was what, 64 weeks, 8-10 hours a day, it might take me a long fucking time. Then again, I don’t know how far that course went (past Intermediate?), nor do I have any idea how closely the pacing at DLI meshed with how quickly Dan (and probably I) acquire information. The attrition rate says something, but that something says little about how challenging I’ll find it. It will, of course, be paired up with al-Jazeera once I have a decent grasp of it. Hell, I still watch Deutschewelle, plus reading some Solaris blogs in German (c0t0d0s0, for instance, is German about 50% of the time), and after nearly eight years, I still don’t have any problems with it.

I haven’t stopped studying things, but, I suppose, it loses its luster after a while. Sure, I’ve learned three programming languages in the last year. I’m finally back on learning Rails (even if the job for the Rails consulting company doesn’t mandate that I know it beforehand, it’s not all that tough to learn now that the API has stabilized, and I’d like to be prepared for the interview), and I’ll probably play with Django or Pylons after that just to see how it compares (though I think I’d probably enjoy Ruby/Rails development more than Python/whatever, and deployment is a hell of a lot easier with JRuby and Glassfish/Tomcat/JBoss, by just dumping a .war on there). Working on expanding my .NET knowledge at work, too (ASP.NET, specifically), and I’ll no doubt continue that. However, it seems like such a limited focus (programming, that is), and I feel like if I re-learned calculus on my own, I probably wouldn’t forget it.

That, then, is the plan. No idea what to do after that, which will probably be a year and a half. After mathematics, chemistry/physics? After Arabic, Russian/Chinese (maybe not another language, anyway, maybe just linguistics, but Sanskrit might fill in for that nicely).

Ruby vs. Perl vs. Python

Sigh. I didn’t check my fileserver before I left home. It booted into a non-xVM kernel, so the code in my CentOS virtual machine isn’t accessible. Not that it’s a big deal, it just means that I’ll only be posting the code I’ve got here at work for a routine to check file modification times via SFTP. The reason d’être seems to be making sure customers aren’t trying to grab files whilst they’re being updated, but I’m not sure on that. They wanted the code, I wrote the code, now it’s doing… something.

At this point, I’m attempting to write every script in Perl, Python, and Ruby simultaneously. It shouldn’t be a surprise that I’m most productive in Perl, but I’d still like to keep up skills in Python (and build them in Ruby). For one reason or another (the fact that HP-UX systems don’t come with Python, for one; Ruby performing about 10 times as slow as Perl/Python on the 1.8 interpreter, and the 1.9 interpreter not being production ready for another [and yes, JRuby is really fast {faster than Ruby 1.9}, but the JVM startup time is a killer for a script running out of cron, and not every system in my datacenter has a JVM installed]), the Perl version is almost always the one that goes onto a production box somewhere.

It’s also worth noting that writing Ruby/Python in a procedural manner, like it’s Perl with methods tacked onto the objects, doesn’t make any sense to me. Sure, it’s kinda fun to write it that way, and blocks in Ruby are really handy (even if they’re slightly more confusing to a non-Rubyist than pointers were the first time I used them), but it feels against the spirit of it somehow. I don’t know.

That being said, this code isn’t nearly what would go into production anyway. The Python really isn’t far off. Perl’s kinda slow when it comes to objects, though, and I’d never use Class::Struct in something which weren’t totally network or I/O bound (probably iterate through arrays or hashes instead). The Ruby? Well, ahh, it’d probably look a lot like this. I’m not exactly a Rubyist at this point, though, and I’m sure it could be cleaned up a lot more without playing Perl Golf with it (hopefully by integrating Rubyisms which don’t make it God-awful unreadable, like the code at the top of this post).

As a total aside, I loathe Coding Horror just a little bit more than I hate Joel on Software. We’re talking about two guys who probably spend as much time blogging as they do working, treat their readers like idiots, preach bad practices (Joel’s company wrote its own fucking programming language rather than a DSL, and he advocates against using Exceptions, among other things. Jeff Atwood [Coding Horror] has no idea what the phrase “use the right tool for the right job” means, and would rather advocate using .NET for everything).

FWIW, I realize now why I changed to this theme. WIDE textarea. Not all the Wordpress themes I’ve played with work nicely once I start fucking with the margins for it in CSS. I don’t give a damn about widget ready or what have you. All that stuff is easy enough to add by hand. I want a theme that’s not going to waste 1/3rd of the page on blankness.

WTB blogs from Dan.

So, here’s one script (the Craigslist parser will get posted tomorrow, I guess).

In Perl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174

#!/usr/bin/perl
 
###############
# SFTPCheck 
#
# 1.0
#
# Description:
# Checks update times for files on Omaha's server to troubleshoot scp problems
 
#Import what we need
use Class::Struct;
use Net::SFTP;
use Net::SFTP::Util qw{fx2txt};
use Net::SFTP::Attributes;
use strict;
use utf8;
 
#Set up  a struct we can dump data into
 
struct fileinfo => { filename => '$', 
 			  modtime => '$',
			  oldtime => '$',
			  waschanged => '$'};
 
#Set up logging
open(OUTFILE, ">> /data/testscript/ryan/sftpcheck.out");
print OUTFILE "-----------------------------------------------------------------------------------------\n";
my $now = localtime(time());
print OUTFILE "Starting at $now\n";
 
#Set up the connection parameters
my ($user, $pass, $host, $dir, $stamp, $diff) = @ARGV;
 
my @listers;
 
 
#Open it
my $sftp = undef;
$sftp = OpenSFTP($sftp, $user, $pass, $host);
 
#Set the directory path, since Net::SFTP->ls doesn't return a fully-qualified one
my $prefix = $dir . "/";
 
#Run through a loop 30 times, sleeping for one second inbetween
for(my $count = 0; $count <= 30; $count++) {
 
	#Do an ls
	@listers = &lookforfiles;
 
	foreach my $file (@listers) {
		#Set the fully qualified name so we can stat is
		my $remote = $prefix . $file;
		#Grab the modification time
		my $stat = $sftp->do_lstat($remote);
		#If it changed, set waschanged and put the new value in
		if ($file->oldtime != $stat->mtime) {
			$file->modtime($stat->mtime);
			$file->waschanged(1);
		}
		else {
			$file->modtime($stat->mtime);
		}
	}
	sleep(1);
}
 
foreach my $file (@listers) {
	if ($file->waschanged) {
		print OUTFILE $file->filename . " was changed during the check!\n";
	}
 
	#Compare it to now
	my $mtime = $file->modtime;
	my $difference = time() - $mtime;
	#If it's less than five minutes, pass it off in seconds
	if ($difference <= 300) {
		print OUTFILE $file->filename . " was updated $difference seconds ago\n";
	}
	elsif ($difference <=3600) {
		#Otherwise, minutes with two decimal places should be precise enough
		my $minutes = $difference / 60;
		printf(OUTFILE $file->filename . " was updated %.2f minutes ago\n", $minutes);
	}
	else {
		#If it's really that old, just print hours
		my $hours = $difference / 3600;
		printf(OUTFILE $file->filename . " was updated %.2f hours ago\n", $hours);
	}
}
 
sub OpenSFTP
{
	my ($sftp, $username, $pass, $host) = @_;
	my %args = ( 
		user => $username,
		password => $pass,
		debug => '1',
	);
	$username = utf8::encode($username);
	$pass = utf8::encode($pass);
	print "Trying to connect to $host as $username:$pass\n";
	$sftp = Net::SFTP->new($host, %args);
	my $status = fx2txt($sftp);
	if ($sftp) {
		return($sftp);
	}
}
 
sub lookforfiles
{
	#Figure out what the timestamp should be
	my @lookfor = &sftpstamp($stamp, $diff);
	#Check for yesterday's date too, why not.
	$lookfor[1] = $lookfor[0]--;
	my @found;
 
	#Do the LS, which doesn't support globbing, and pass it off to a regexp to find the files we need
	my @list = $sftp->ls("$dir", wanted => sub { $_[0]->{filename}});
	foreach my $day (@lookfor) {
		foreach my $file (@list) {
			my $name = $file->{filename};
			if ($name =~ /$day/) {
				my $there = 0;
				while (!$there) {
					foreach my $loop (@listers) {
						if($loop->filename($name)) {
							#If the filename is already in the array, break
							$there = 1;
						}
					}
					my $foundname = fileinfo->new();
					$foundname->filename($name);
					my $stat = $sftp->do_lstat($prefix . $name);
					$foundname->oldtime($stat->mtime);
					push(@listers, $foundname);
					$there = 1
				}
			}
		}
	}
	return(@found);
}
 
sub sftpstamp
{
    (my $stamp, my $diff) = @_;
	#Set the tzinfo so we know what the stamp should be
	my $timediff = 3600 * $diff; 
 
	#Check if we're in DST
	my $dststatus = 0;
	my $dstfile = "/data/.DST_Status";
	open(DST, $dstfile);
	$dststatus = (<DST>);
	chomp($dststatus);
	close(DST);
 
	#If so, modify the time accordingly
	if ($dststatus) {
		$timediff -= 3600;
	}
 
	#Get the time and format it
	my @StampNow = localtime(time() - $timediff);
	my $day = $StampNow[3];
	$day = "0$day" if $day < 10;
	my $month = $StampNow[4] + 1;
	$month = "0$month" if $month < 10;
 
	#Pass the appropriate value back
	$stamp = "$month$day";
	return $stamp;
}

In Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184

import base64
import getpass
import os
import platform
import socket
import sys
import time
import traceback
 
import paramiko
 
class foundfile:
    def __init__(self, filename=None, modtime=None, oldtime=None, waschanged=None):
        self.filename = filename
        self.modtime = modtime
        self.oldtime = oldtime
        self.waschanged = waschanged
 
class sftpcheck:
 
    def __init__(self, host=None, username=None, password=None, dirname=None, stamp=None, diff=None):
 
 
 
        #Set up the initial variables for login
        self.host = host
        self.username = username 
        self.password = password
        self.dirname = dirname
        self.stamp = stamp
        self.diff = diff
        self.sftp = None
        self.found = []
 
        #Set up logging
        if platform.uname()[0] == "Windows":
            self.LogPath = "\\\\filer2\\data\\testscript\\ryan\\"
            self.DSTPath = "\\\\filer2\\data\\.DST_Status"
            self.DebugPath = "\\\\filer2\\data\\testscript\\ryan\\"
        else:
            self.LogPath = "/data/testscript/ryan/"
            self.DSTPath = "/data/.DST_Status"
            self.DebugPath = "/data/testscript/ryan/"
        self.DebugFilename = "sftpdebug.log"
        self.LogFilename = "sftpcheck.log"
        self.LogFileObj = None
 
    def login(self, host, username, password):
        #Connect to the server
        t = paramiko.Transport((host, 22))
        t.connect(username=username, password=password)
        self.sftp = paramiko.SFTPClient.from_transport(t)
 
        return (self.sftp != None)
 
    def listfiles(self):
        #Pick up how it's supposed to be formatted
        lookfor = self.formattime(self.diff)
 
        #Get a list of files in the directory
        ls = self.sftp.listdir(self.dirname)
 
        #Check if any match
        for fileinfo in ls:
            if re.search(lookfor, fileinfo):
                there = False
                while there != True:
                    for record in found:
                        if record.filename == fileinfo:
                            there = True
                    newone = foundfile(filename = fileinfo)
                    mtime = self.sftp.lstat(self.dirname + fileinfo)[8]
                    newone.oldtime = mtime
                    self.found.append(newone)
                    there =