The Prodigal Boyfriend

Importing From MySpace

Ryan — Sat, 22 Nov 2008 04:13:39 +0000

Given that I’ve deleted my Facebook account, and I’ll be deleting my MySpace account shortly (they never get used, and I don’t see the point), I decided to look into the possibility of importing posts from MySpace to Wordpress. As it turns out, the Wordpress developers are apparently really ridiculously lazy, or just don’t give a shit about importing. Somebody had written a Perl script which pulled MySpace blogs into RSS, but bringing RSS into Wordpress doesn’t get comments with it.

After looking over the formats Wordpress -could- import from, I threw out everything with [!CDATA] tags in XML (almost every format). Fortunately, the Movable Type developers don’t see a need to dump binary blobs in XML, or use XML at all. Their format is refreshingly simple. Hence, a parser that runs through every blog post on somebody’s MySpace, pulls out data that matters (title, date, post, commenters and their comments), then puts those into Movable Type format. You’ll note that I now have way more posts on here than I did before, some of those being from before this website existed.

Code:

#!/usr/bin/ruby
 
require 'open-uri'
require 'time'
 
class Comment
  attr_accessor :author, :datewritten, :comment
  def initialize (author, datewritten, comment)
      @author, @datewritten, @comment = author, datewritten, comment
  end
end
 
class Post
  attr_accessor :author, :title, :datewritten, :body, :comments
  def initialize (author, title, datewritten, body)
    @author, @title, @datewritten, @body = author, title, datewritten, body
    @comments = []
  end
  def addcomment (author, datewritten, comment)
    @comments.push(Comment.new(author, datewritten, comment))
  end
end
 
class Ripper
  def initialize  
    @pages = []
    @posts = Array.new
  end
  def get (uri)
    connection = open(uri)  
    content = connection.read
    return content
  end
 
  def parse (uri)
    content = get(uri)
    #blogContentInfo points to links to posts
    links = content.scan(/class="blogContentInfo">.*?<a href=".*?">/m)
    links.each do |link|
        #Strip out the bullshit amazon links
        unless link =~ /amazon/
            #Pull the URL out of the link
            link = (/.*<a href="(.*)">/m).match(link)[1]
            @pages.push(link)
        end
    end
    #Checking if there are any older pages with a hyperlink
    if content =~ /\[.*?<a href="(.*?)">Older<\/a>/
      #If so, call itself recursively to pull out the rest
      #Myspace breaks the URI standard.  Replace the spaces with real escapes
      parse($1.gsub(/\s/, "%20"))
    else
      #Edge case to break out of the loop for when there aren't any more older
      parsepages()
    end
  end
 
  def parsepages()
    @pages.each do |uri|
      #Replace with yourself, if you want
      author = "Ryan"
      content = get(uri)
      #Pull out the fields I want
      title = (/blogSubject">(.*?)\n/m).match(content)[1]
      body = (/blogContent">(.*?)<table/m).match(content)[1]
      datewritten = (/blogTimeStamp">(.*?)<\/p>/m).match(content)[1].gsub(/(^\s+|\n+)/, "")
      time = (/blogContentInfo"><b>.*?(\d+:\d+)/m).match(content)[1]
      datewritten = datewritten + " #{time}:00"
      #Parse the time, and force it into something Wordpress can deal with
      t = Time.parse(datewritten)
      datewritten = t.strftime("%m/%d/%Y %H:%M:%S")
      puts "Title: #{title}\n"
      #Create a new Post object
      post = Post.new(author, title, datewritten, body)
      #Pull out an array of all the comment blocks
      comments = content.scan(/id="blogComments.*?commentSpacer/m)
      #Pass off the post object along with the list of comments
      parsecomments(comments, post)
    end
  end
 
  def parsecomments(comments, post)
    comments.each do |com|
      author = (/profileLinks">(.*?)m).match(com)[1]
      puts "Author: #{author}\n"
      #MySpace decided to make the CSS ids identical here, except that the
      #actual comment doesn't have "Posted" after the closing tag
      #Filter it as such
      comment = (/blogCommentsContent">(.*?)<\/p>/m).match(com)[1]
      datewritten = (/blogCommentsContent">Posted by.*?> on(.*?)<b/m).match(com)[1].gsub(/\n|\t|\r/, "")
      t = Time.parse(datewritten)
      #The same datetime munging as before
      datewritten = t.strftime("%m/%d/%Y %H:%M:%S")
      #Commit each commment to our post object
      post.addcomment(author, datewritten, comment)
    end
  #Push them all into our class array
  @posts.push(post)
  end
 
  def print(file)
    @posts.each do |post|
      #Using Movable Type's export syntax, so I don't need to mess with XML
      #It's documented here: http://www.sixapart.com/moveabletype/docs/mtimport#example
      #Basically, 5 hyphens separates the categories
      #Eight hyphens separate each post
      file.puts "TITLE: #{post.title}"
      file.puts "AUTHOR: #{post.author}"
      file.puts "DATE: #{post.datewritten}"
      #Change this, too, if you want
      file.puts "CATEGORY: MySpace"
      file.puts "-----"
      #Get rid of empty lines and fucking Windows ^M newlines, plus convert   to " "
      file.puts "BODY:\n#{post.body.gsub(/^(\s+|\t+|\n+)$/, "").gsub(/\015/, "").gsub(/ /, " ")}"
      file.puts "-----"
      post.comments.each do |com|
        #More stuff is possible here, but isn't necessary
        file.puts "COMMENT:"
        file.puts "AUTHOR: #{com.author}"
        file.puts "DATE: #{com.datewritten}"
        file.puts "#{com.comment}"
        file.puts "-----"
      end
      file.puts "--------"
    end
  end
end
 
#Instantiate it
ripper = Ripper.new
#Parse my blog (substitute whatever yours is here)
ripper.parse("http://blog.myspace.com/lykurgos")
#Output it
 
output = File.open("posts.txt", "a")
ripper.print(output)
puts "Done!\n"
#Import into Wordpress!

Maybe somebody will actually find it useful.

Real World Regexes

Ryan — Thu, 09 Oct 2008 04:18:53 +0000

Dan mentioned that he wasn’t that knowledgeable about regular expressions (a topic I am intimately familiar with), so I figured I’d put up some examples from code I’ve actually written, along with the text they’re actually supposed to match.

To begin with, here are the general rules for regexes. To begin with, “operator” refers to any of these (so \s+, [A-Z], (Word), etc). Greedy means it’ll continue matching as far as possible, and if the operator/character you want to match occurs more than once in the string, it’ll eat the first one and only stop matching at the last one.

. Match any character
\w Match “word” character (alphanumeric plus “_”)
\W Match non-word character
\s Match whitespace character
\S Match non-whitespace character
\d Match digit character
\D Match non-digit character
\t Match tab
\n Match newline
\r Match return
\f Match formfeed
\a Match alarm (bell, beep, etc)
\e Match escape
^ Beginning of the line
$ End of the line
+ matches the preceding operator one or more times (greedy)
* matches the preceding operator zero or more times (greedy)
? matches the preceding operator once if it exists, but it doesn’t have to be there. Mostly used to stop greedy operators (*? or +?, for instance) at the match you want.
() is used for grouping (either to use later as a backreference or to exclude)
(?) (or (?P) in Python and maybe others) is used for a named backreference. There’ll be some examples of that.
| is used as a logical or
{n} is used to match the preceding character n times
{n, m} matches n to m times
{n,} matches 1 or more times (may as well use +)
[A-Za-z] is used to match whatever is in the middle, but it only counts as one character (so [A-Za-z] would match any of those characters ONCE. Useful if you want [a-f] or [0-5]+ or something).
[^] is used to exclude things. [^word] excludes “w”, but the caret only matches ONCE (this can be chained as [^(word)], since groups count as a single operator.

Sound confusing? It is, which is why I’ll put up real examples. FYI, these are PCRE (Perl Compatible Regular Expressions) rather than SCRE (Sed Compatible Regular Expressions), but Dan’ll almost certainly never use sed compatible (which doesn’t have a ? operator, among other things).

Using a backreference later depends on the language. .NET uses ${n} where n is the reference number (note that they start from 1, as the entire string you matched is ${0}), Perl (and a lot of others) us $n, Ruby uses \1 (as does Python, but Python {like .NET} needs an operator in front to use a raw string {.NET is @, Python is r}, otherwise it’s \\1). Language reference is your best bet here.

First example.

(Oct6 0423z) Dec4100: C, was acknowledged by, ek

string regexPattern = @".*?\)\s
                      (?\S+?)
                      :\s
                      (?\w)
                      .*,\s
                      (?.*)";
Regex re = new Regex(regexPattern, RegexOptions.ExplicitCapture);

It eats everything up until the right parenthesis (escaped so the regex parser doesn’t try to interpret it) followed by a space, then it gets all non-whitespace characters until the colon as the system name. Ignores the colon and a space, then grabs all word characters ([A-Z0-9_]) as the tape number. Ignores zero or more matches of any character (the “.”) until it finds a comma followed by a space, then yanks the rest of the line as the initials.

C is the tape name.

ek are the initials.

This means Dec4100 is available as ${system} (if doing Regex.Replace) or m.Groups["system"] if you matched the regex with m = Regex.Match(logfilestring, re);

Another example:

	
		
		
		
		
		
		
	
		(AE) BOW SUMMIT
		ALTA

And the parser:

if ($chunk =~ /.*StationID.*?"(\d+)".*?prov.*?"(\w+).*?TableRowHeader">(.*?)<.*abbr title.*?>(\w+).*?/s) {
     my $stationid = $1;
     my $province = $2;
     my $name = $3;
     my $abbrprov = $4;
}

This is a multi-line regex (hence the //s, like //g is global, //i is case insensitive, //gi is both g and i, etc), and a good example of non-greedy matching. It snags everything up until StationId, then the next quotation mark followed by numbers, and captures those numbers. It comes out as “10700″.

Does the same thing following “prov” up until the next word characters in quotation marks, and captures those. As .* rather than .*?, it would have grabbed “data”, which precedes TableRowHeader (inside the same parenthesis). Comes out as “CA”.

Grabs everything from TableRowHeader”> until the next < Comes out as “(AE) Bow Summit”.

Drops everything up until the next < after “abbr title”, then captures all word characters. “ALBA”

These are all assigned to variables via backreferences. $1, $2, $3, $4 are the groups in order. It’s worth noting that (at least in .NET), named backreferences are assigned numbers BEFORE regular backreferences. So (?a)(b)(?c)(d) would be acbd as ${0}${1}${2}${3}.

Another example:

04:26:23 [2] Error creating WLAAAP06.FS8 = 1 : Unrecognized KGFXENG Error Code

And the parser:

re.match(line, r'^(?P.*?)\s+\[(?P\d+)\]\s+(?P.*?(KGFXENG|LeadTools).*)'

Grabs everything from the beginning of the line until the first space as “time”. Comes out as “04:26:23″.

Then skips whitespace and a bracket (escaped with \[) and grabs one or more numbers (\d+) as "engine". Comes out as "2", of course. Skips a space, then captures anything which contains "KGFXENG" or "LeadTools" as "error". Basically, the rest of the line.

This line, for instance, wouldn't match, and nothing in the regex would be captured:

00:15:18 [1] Error producing WPATAZ00.FSD = F088 : Error while saving the graphic

These are used later with this:

message = "ERROR: %s %s: %s" % (re.sub(r'.*?([A-Za-z]+Engine[A-Za-z]*?)(Errors)?.*', r'\1', 
                         logfilename), 
                         engine, 
                         match.group('error'))

“logfilename” is something like “2008_Oct_07__ProductEngineErrors.log”. This grabs everything up until A through Z (uppercase or lowercase) one or more times followed by Engine, optionally followed by something else (*, though ? would have worked if I said r’Engine([A-Za-z]+)?’). It stops on Errors, if it exists (the question mark afterwards), and replaces the entire name with the first backreference (”ProductEngine” in this case).

Last example is a nested bitch of increasingly complicated rules:

#Match plain ol' timezones
if ($brpos =~ /^\[(\w+)\](.*)/)
{
	$DateZone = $1;
	$newname = $2;
}
#Match timezones with a day modification, and grab that along with the +/-
elsif ($brpos =~ /^\[(\w+)(\S\d+)\](.*)/)
{
	$DateZone = $1;
	$TempDay2 = ONE_DAY * $2;
	$newname = $3;
}
#Check for a delete flag
elsif ($brpos =~ /^(\d)\[.*/)
{
	$DeleteFilesStatus = $1;
	#If the status is one, we want to capture everything after the timezone as the DeleteName
	if ($DeleteFilesStatus == 1)
	{
		if ($brpos =~ /^(\d)\[(\w+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$DeleteFilesNames = $3;
			$newname = $3;
		}
		elsif ($brpos =~ /^(\d)\[(\w+)(\S\d+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$TempDay2 = ONE_DAY * $3;
			$DeleteFilesNames = $4;
			$newname = $4;
		}
	}
	#Otherwise, the DeleteName is in more brackets
	elsif ($DeleteFilesStatus == 2)
	{
                #Grab it all, but without a time modification
		if ($brpos =~ /^(\d)\[(\w+)\]\[(.*\.\w+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$DeleteFilesNames = $3;
			$newname = $4;
		}
                #Grab it with a time modification
		elsif ($brpos =~ /^(\d)\[(\w+)(\S\d+)\]\[(.*\.\w+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$TempDay2 = ONE_DAY * $3;
			$DeleteFilesNames = $4;
			$newname = $5;
		}
	}
}

Examples of what I’m catching (hopefully in order). The stuff in brackets later is filled in for date/time stamps:

[EDT]DOV-F-[MM][dd][yy][hh].csv
[CST-1][MM][dd].act
1[PDT]Actual[yy][MM][dd][hh][mm].csv
1[EST-3]KLGA[yy][MM][dd].mtx
2[EDT][WBD*.txt]WBD[yy][MM][dd]05.txt
2[MST+2][WSM*.txt]WBD[yyyy][MM].txt

Sadly, I’m out of work for the night, but these matches aren’t that complicated. Lots of escaping brackets, and use of the \S character to match “-” or “+”, then grabbing the rest of them. I may write more tomorrow…

RosterParserFixed.XmlParser parser = new RosterParserFixed.XmlParser()

Ryan — Thu, 09 Oct 2008 02:44:41 +0000

Fixed the nesting problem. Fixed item parsing. Item stats for nested ones units show up now. As with the Ruby parser, throw different combinations at it and see what happens.

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Text.RegularExpressions;
using System.Xml;
using System.Xml.Serialization;
using System.Xml.Schema;
using System.Xml.XPath;
 
namespace abparser
{
    class Program
    {
        static void Main(string[] args)
        {
            RosterParserTest.XmlParser parser = new RosterParserTest.XmlParser();
            parser.ParseRoster(@"C:\Temp\de7th.rst", @"C:\Temp\output.xml");
            Console.ReadLine();
        }
    }
}
 
 
 
namespace RosterParserTest
{
    class XmlParser
    {
        static XmlDocument Roster = new XmlDocument();
 
        static XmlElement rootElement = Roster.CreateElement("", "Army", "");
 
        public static string RemoveWhitespace(string str)
        {
            try
            {
                //Ryan's Regex
                return new Regex(@"(\s+|\{.*?\}|\(.*?\)|\/+|\.+)").Replace(str, String.Empty);
            }
            catch (Exception)
            {
                return str;
            }
    }
 
        public void ParseNestedXML(XmlElement thisElement, XmlElement rosterElement)
        {
            bool linkUnitStatsDone = false; //This is a dirty hack.
 
            string replaceMe = thisElement.GetAttribute("name").ToString();
 
            replaceMe = RemoveWhitespace(replaceMe);
 
            XmlElement baseElement;
 
            XmlNodeList linkUnitStatNodeList = thisElement.SelectNodes("./link | ./unitstat");
 
            /*Grab the last of the PascalCase names.  HarGanethExecutioners becomes Executioners
             * SupremeSorceress becomes Sorceress, etc.  Replace the rest of the name with a backreference */
            string regexMatcher = Regex.Replace(replaceMe, @".*?([A-Z][a-z]+)$", "${1}"); 
 
            //This way, it'll actually parse the NodeList for stats in nested things.
            if (Regex.IsMatch(rosterElement.Name.ToString(), regexMatcher)) //So that I don't get duplicate empty nodes.
            {
                baseElement = rosterElement; //Adding to the previous node in the tree.
 
                foreach (XmlElement parseElement in thisElement)
                    {
                        if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                        {
                            ParseNestedXML(parseElement, baseElement); //Parsing out nested.
                        } //if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                        else if (!linkUnitStatsDone)
                        {
                            ParseLinkUnitStats(linkUnitStatNodeList, baseElement);
                            linkUnitStatsDone = true; //Hack implemented.
                        } //else if (!linkUnitStatsDone)
                    } //foreach (XmlElement parseElement in thisElement)
            }
            else
            {
                baseElement = Roster.CreateElement(replaceMe);
 
                foreach (XmlElement parseElement in thisElement)
                {
                     if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                    {
                        ParseNestedXML(parseElement, baseElement); //Whee recursion.
                    } //if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                    else
                    {
                        ParseLinkUnitStats(parseElement, baseElement); //This has always worked.
                    } //else
 
                    rosterElement.AppendChild(baseElement); //Add to the local node.
 
                    rootElement.AppendChild(rosterElement); //Add to the Army node.
                } //foreach (XmlElement parseElement in thisElement)
 
            } //else
 
        } //public void ParseNestedXML(XmlElement thisElement, XmlElement rosterElement)
 
        public void ParseRoster(string path, string output)
        {
            XmlDocument parsingRoster = new XmlDocument();
 
            parsingRoster.Load(path);
 
            XmlNodeList parsingElements = parsingRoster.SelectNodes("/document/squad");
 
            foreach (XmlElement thisElement in parsingElements)
            {
                XmlElement rosterElement = Roster.CreateElement("Unit");
 
                ParseNestedXML(thisElement, rosterElement);
            } //foreach (XmlElement thisElement in parsingElements)
 
            Roster.AppendChild(rootElement);
 
            Roster.Save(output);
        } //public void ParseRoster(string path, string output)
 
        public void ParseLinkUnitStats(XmlElement parseElement, XmlElement baseElement)
        {
 
            foreach (XmlElement correctElement in parseElement)
            {
                if (correctElement.HasAttribute("name"))
                {
                    string subReplaceMe = correctElement.GetAttribute("name").ToString();
 
                    subReplaceMe = RemoveWhitespace(subReplaceMe);
 
                    XmlElement addElement = Roster.CreateElement(subReplaceMe);
                    if (!Regex.Match(subReplaceMe, @"(Left|Worker|Helper|Pts|Coun|Group)").Success)
                    {
                        if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                        {
                            //Console.WriteLine("Found an item (XmlElement)");
                            ParseNestedXML(addElement, correctElement);
                        }
                        else if (correctElement.HasAttribute("description"))
                        {
                            addElement.InnerText = correctElement.GetAttribute("description").ToString();
                        } //if correctElement.HasAttribute("description"))
                        else if (correctElement.HasAttribute("value") && (Regex.IsMatch(correctElement.GetAttribute("value"), @"[^0|-]")))
                        {
                            addElement.InnerText = RemoveWhitespace(correctElement.GetAttribute("value").ToString());
                            baseElement.AppendChild(addElement);
                        } //else if (correctElement.HasAttribute("value"))
                    } //else
                    if (parseElement.HasAttribute("basename"))
                    {
                        /*It's a non-dwarf item.  Whee!  They don't show up in the XmlNodeList one.
                        Get rid of newlines and periods at the end, then set it as the InnerText
                        This doesn't catch cases where the item has other properties inside it, but
                        I haven't seen those */
                        baseElement.InnerText = Regex.Replace(parseElement.GetAttribute("itemsummary"), @"(\\n|\.)", String.Empty);
                    }
                } //if (correctElement.HasAttribute("name")
 
 
            } //foreach (XmlElement correctElement in parseElement)
 
        } //public void ParseLinkUnitStats(XmlElement parseElement, XmlElement baseElement)
 
        public void ParseLinkUnitStats(XmlNodeList parseNodeList, XmlElement baseElement)
        {
            foreach (XmlElement correctElement in parseNodeList)
            {
                if (correctElement.HasAttribute("name"))
                {
                    string subReplaceMe = correctElement.GetAttribute("name").ToString();
                     subReplaceMe = RemoveWhitespace(subReplaceMe);
 
                    if (!Regex.Match(subReplaceMe, @"(Left|Worker|Helper|Pts|Coun|Group)").Success)
                    {
 
                        XmlElement addElement = Roster.CreateElement(subReplaceMe);
                        if (correctElement.HasChildNodes && correctElement.InnerXml.Contains("entity"))
                        {
                            //Console.WriteLine("Found an item (XmlNodeList)");
                            ParseNestedXML(addElement, correctElement);
                        }
 
                        if (correctElement.HasAttribute("description"))
                        {
                            addElement.InnerText = correctElement.GetAttribute("description").ToString();
                            baseElement.AppendChild(addElement);
                        } //if (correctElement.HasAttribute("description"))
                        else if (correctElement.HasAttribute("value") && (Regex.IsMatch(correctElement.GetAttribute("value"), @"[^0|-]")))
                        {
                            addElement.InnerText = RemoveWhitespace(correctElement.GetAttribute("value").ToString());
                            baseElement.AppendChild(addElement);
                        } //else if (correctElement.HasAttribute("value"))
                    } //else
 
                } //if (correctElement.HasAttribute("name"))
 
            } //foreach (XmlElement correctElement in parseNodeList)
 
        } //public void ParseLinkUnitStats(XmlNodeList parseNodeList, XmlElement baseElement)
 
    } //class XmlParser
 
} //namespace RosterParserTest

Ugh.

Ryan — Sat, 04 Oct 2008 04:30:38 +0000

I’m already not that fond of working with XML in .NET. Here are a couple of fixes:

public static string RemoveWhitespace(string str)
{
    try
    {
        return new Regex(@"(\s+|\{.*?\}|\(.*?\)|\/+|\.+)").Replace(str, String.Empty);
    }
    catch (Exception)
    {
        return str;
    }
}

Which actually gets rid of the crap in the braces, parentheses, etc (as well as getting rid of periods).

Secondly, I loathe empty nodes (stats, etc).

replaceMe = RemoveWhitespace(replaceMe);
Console.WriteLine(replaceMe);
if (replaceMe != String.Empty) 
{
    XmlElement baseElement = Roster.CreateElement(replaceMe);
 
    foreach (XmlElement parseElement in thisElement)
    {
        if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
        {
 
            ParseNestedXML(parseElement, baseElement);
        }
        else
        {
            foreach (XmlElement correctElement in parseElement)
            {
                if (correctElement.HasAttribute("name"))
                {
                    string subReplaceMe = correctElement.GetAttribute("name").ToString();
 
                    subReplaceMe = RemoveWhitespace(subReplaceMe);
 
                    XmlElement addElement = Roster.CreateElement(subReplaceMe);
 
                    if (correctElement.HasAttribute("description"))
                    {
                        addElement.InnerText = correctElement.GetAttribute("description").ToString();
                        baseElement.AppendChild(addElement);
                    }
                    //Bye, stats with a value of zero or a hyphen!
                    else if (correctElement.HasAttribute("value") && (Regex.Match(correctElement.GetAttribute("value").ToString(), @"[^0|-]").Success))
                    {
                        addElement.InnerText = correctElement.GetAttribute("value").ToString();
                        baseElement.AppendChild(addElement);
                    }
 
                }
            }
        }
 
        rosterElement.AppendChild(baseElement);
 
        rootElement.AppendChild(rosterElement);
    }
}

I find it kind of ironic that recursion is used after bitching about recursion. I’ll probably take a look at the nesting problems, and whatnot this weekend, assuming I have any time.

I wonder if it’s possible to get a job doing nothing but writing regular expressions…

Parser, part deux

Ryan — Thu, 02 Oct 2008 21:54:57 +0000

Fixed the nested blocks. Ran into another problem where it didn’t parse nested characters and their items properly, then yet another where some Gifts of Khaine (and probably other things I haven’t seen in either list) are essentially nested worthlessness. Fixed code for it:

def parseitem(d, addto)
  added = addto.add_element(d.attributes["name"].gsub(/\s+/, '')).add_text(d.attributes["description"].gsub(/\./, ''))
  d.elements.each('link') do |ele|
    unless ele.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
      if !ele.attributes["description"].nil?
        added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text(ele.attributes["description"])
      else
        #This is necessary for some Gifts of Khaine, apparently
        if added.text == ele.attributes["name"]
          print "Found duplicated #{added.text}!\n\n"
          added.text = ''
        end
        added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text("True")
      end
    end
  end
end
 
def parsenested(process, addto)
  #Try to guess if it's a champion, character in the unit, or item
  process.elements.each('entity') do |d|
    if d.attributes["statset"] =~ /Normal/
      #It's a character, crew, or mount.  Figure out which
      if d.attributes["totalcost"] !~ /^0/
        #It's a champion or character
        adder = addto.add_element("champion")
        puts "Found champ\n"
        parse(d, adder)
      else
        #It's crew or the like
        adder = addto.add_element("crew")
        puts "Found crew\n"
        parse(d, adder)
      end
    else
      #It's an item
      puts "Found item\n"
      if addto.elements["item"].nil?
        @adder = addto.add_element("item")
      end
      parseitem(d, @adder)
    end
  end
end