0 RESPONSES comments
NOVEMBER 21ST, 2008
By RYAN

Importing From MySpace

Given that I’ve deleted my Facebook account, and I’ll be deleting my MySpace account shortly (they never get used, and I don’t see the point), I decided to look into the possibility of importing posts from MySpace to Wordpress. As it turns out, the Wordpress developers are apparently really ridiculously lazy, or just don’t give a shit about importing. Somebody had written a Perl script which pulled MySpace blogs into RSS, but bringing RSS into Wordpress doesn’t get comments with it.

After looking over the formats Wordpress -could- import from, I threw out everything with [!CDATA] tags in XML (almost every format). Fortunately, the Movable Type developers don’t see a need to dump binary blobs in XML, or use XML at all. Their format is refreshingly simple. Hence, a parser that runs through every blog post on somebody’s MySpace, pulls out data that matters (title, date, post, commenters and their comments), then puts those into Movable Type format. You’ll note that I now have way more posts on here than I did before, some of those being from before this website existed.

Code:

#!/usr/bin/ruby
 
require 'open-uri'
require 'time'
 
class Comment
  attr_accessor :author, :datewritten, :comment
  def initialize (author, datewritten, comment)
      @author, @datewritten, @comment = author, datewritten, comment
  end
end
 
class Post
  attr_accessor :author, :title, :datewritten, :body, :comments
  def initialize (author, title, datewritten, body)
    @author, @title, @datewritten, @body = author, title, datewritten, body
    @comments = []
  end
  def addcomment (author, datewritten, comment)
    @comments.push(Comment.new(author, datewritten, comment))
  end
end
 
class Ripper
  def initialize  
    @pages = []
    @posts = Array.new
  end
  def get (uri)
    connection = open(uri)  
    content = connection.read
    return content
  end
 
  def parse (uri)
    content = get(uri)
    #blogContentInfo points to links to posts
    links = content.scan(/class="blogContentInfo">.*?<a href=".*?">/m)
    links.each do |link|
        #Strip out the bullshit amazon links
        unless link =~ /amazon/
            #Pull the URL out of the link
            link = (/.*<a href="(.*)">/m).match(link)[1]
            @pages.push(link)
        end
    end
    #Checking if there are any older pages with a hyperlink
    if content =~ /\[.*?<a href="(.*?)">Older<\/a>/
      #If so, call itself recursively to pull out the rest
      #Myspace breaks the URI standard.  Replace the spaces with real escapes
      parse($1.gsub(/\s/, "%20"))
    else
      #Edge case to break out of the loop for when there aren't any more older
      parsepages()
    end
  end
 
  def parsepages()
    @pages.each do |uri|
      #Replace with yourself, if you want
      author = "Ryan"
      content = get(uri)
      #Pull out the fields I want
      title = (/blogSubject">(.*?)\n/m).match(content)[1]
      body = (/blogContent">(.*?)<table/m).match(content)[1]
      datewritten = (/blogTimeStamp">(.*?)<\/p>/m).match(content)[1].gsub(/(^\s+|\n+)/, "")
      time = (/blogContentInfo"><b>.*?(\d+:\d+)/m).match(content)[1]
      datewritten = datewritten + " #{time}:00"
      #Parse the time, and force it into something Wordpress can deal with
      t = Time.parse(datewritten)
      datewritten = t.strftime("%m/%d/%Y %H:%M:%S")
      puts "Title: #{title}\n"
      #Create a new Post object
      post = Post.new(author, title, datewritten, body)
      #Pull out an array of all the comment blocks
      comments = content.scan(/id="blogComments.*?commentSpacer/m)
      #Pass off the post object along with the list of comments
      parsecomments(comments, post)
    end
  end
 
  def parsecomments(comments, post)
    comments.each do |com|
      author = (/profileLinks">(.*?)</m).match(com)[1]
      puts "Author: #{author}\n"
      #MySpace decided to make the CSS ids identical here, except that the
      #actual comment doesn't have "Posted" after the closing tag
      #Filter it as such
      comment = (/blogCommentsContent">(.*?)<\/p>/m).match(com)[1]
      datewritten = (/blogCommentsContent">Posted by.*?> on(.*?)<b/m).match(com)[1].gsub(/\n|\t|\r/, "")
      t = Time.parse(datewritten)
      #The same datetime munging as before
      datewritten = t.strftime("%m/%d/%Y %H:%M:%S")
      #Commit each commment to our post object
      post.addcomment(author, datewritten, comment)
    end
  #Push them all into our class array
  @posts.push(post)
  end
 
  def print(file)
    @posts.each do |post|
      #Using Movable Type's export syntax, so I don't need to mess with XML
      #It's documented here: http://www.sixapart.com/moveabletype/docs/mtimport#example
      #Basically, 5 hyphens separates the categories
      #Eight hyphens separate each post
      file.puts "TITLE: #{post.title}"
      file.puts "AUTHOR: #{post.author}"
      file.puts "DATE: #{post.datewritten}"
      #Change this, too, if you want
      file.puts "CATEGORY: MySpace"
      file.puts "-----"
      #Get rid of empty lines and fucking Windows ^M newlines, plus convert &nbsp; to " "
      file.puts "BODY:\n#{post.body.gsub(/^(\s+|\t+|\n+)$/, "").gsub(/\015/, "").gsub(/&nbsp;/, " ")}"
      file.puts "-----"
      post.comments.each do |com|
        #More stuff is possible here, but isn't necessary
        file.puts "COMMENT:"
        file.puts "AUTHOR: #{com.author}"
        file.puts "DATE: #{com.datewritten}"
        file.puts "#{com.comment}"
        file.puts "-----"
      end
      file.puts "--------"
    end
  end
end
 
#Instantiate it
ripper = Ripper.new
#Parse my blog (substitute whatever yours is here)
ripper.parse("http://blog.myspace.com/lykurgos")
#Output it
 
output = File.open("posts.txt", "a")
ripper.print(output)
puts "Done!\n"
#Import into Wordpress!

Maybe somebody will actually find it useful.

Tags: , ,

categories General

1 RESPONSE comments
OCTOBER 8TH, 2008
By RYAN

Real World Regexes

Dan mentioned that he wasn’t that knowledgeable about regular expressions (a topic I am intimately familiar with), so I figured I’d put up some examples from code I’ve actually written, along with the text they’re actually supposed to match.

To begin with, here are the general rules for regexes. To begin with, “operator” refers to any of these (so \s+, [A-Z], (Word), etc). Greedy means it’ll continue matching as far as possible, and if the operator/character you want to match occurs more than once in the string, it’ll eat the first one and only stop matching at the last one.

. Match any character
\w Match “word” character (alphanumeric plus “_”)
\W Match non-word character
\s Match whitespace character
\S Match non-whitespace character
\d Match digit character
\D Match non-digit character
\t Match tab
\n Match newline
\r Match return
\f Match formfeed
\a Match alarm (bell, beep, etc)
\e Match escape
^ Beginning of the line
$ End of the line
+ matches the preceding operator one or more times (greedy)
* matches the preceding operator zero or more times (greedy)
? matches the preceding operator once if it exists, but it doesn’t have to be there. Mostly used to stop greedy operators (*? or +?, for instance) at the match you want.
() is used for grouping (either to use later as a backreference or to exclude)
(?<name>) (or (?P<name>) in Python and maybe others) is used for a named backreference. There’ll be some examples of that.
| is used as a logical or
{n} is used to match the preceding character n times
{n, m} matches n to m times
{n,} matches 1 or more times (may as well use +)
[A-Za-z] is used to match whatever is in the middle, but it only counts as one character (so [A-Za-z] would match any of those characters ONCE. Useful if you want [a-f] or [0-5]+ or something).
[^] is used to exclude things. [^word] excludes “w”, but the caret only matches ONCE (this can be chained as [^(word)], since groups count as a single operator.

Sound confusing? It is, which is why I’ll put up real examples. FYI, these are PCRE (Perl Compatible Regular Expressions) rather than SCRE (Sed Compatible Regular Expressions), but Dan’ll almost certainly never use sed compatible (which doesn’t have a ? operator, among other things).

Using a backreference later depends on the language. .NET uses ${n} where n is the reference number (note that they start from 1, as the entire string you matched is ${0}), Perl (and a lot of others) us $n, Ruby uses \1 (as does Python, but Python {like .NET} needs an operator in front to use a raw string {.NET is @, Python is r}, otherwise it’s \\1). Language reference is your best bet here.

First example.

(Oct6 0423z) Dec4100: C, was acknowledged by, ek
string regexPattern = @".*?\)\s
                      (?<system>\S+?)
                      :\s
                      (?<tape>\w)
                      .*,\s
                      (?<initials>.*)";
Regex re = new Regex(regexPattern, RegexOptions.ExplicitCapture);

It eats everything up until the right parenthesis (escaped so the regex parser doesn’t try to interpret it) followed by a space, then it gets all non-whitespace characters until the colon as the system name. Ignores the colon and a space, then grabs all word characters ([A-Z0-9_]) as the tape number. Ignores zero or more matches of any character (the “.”) until it finds a comma followed by a space, then yanks the rest of the line as the initials.

C is the tape name.

ek are the initials.

This means Dec4100 is available as ${system} (if doing Regex.Replace) or m.Groups["system"] if you matched the regex with m = Regex.Match(logfilestring, re);

Another example:

	<form action="http://www.climate.weatheroffice.ec.gc.ca/climateData/Interform.cfm" method="post" name="stnRequest1">
		<input type="Hidden" name="hlyRange" value="N/A">
		<input type="Hidden" name="dlyRange" value="1998-4-1|2007-11-30">
		<input type="Hidden" name="mlyRange" value="1998-4-1|2007-11-1">
		<input type="Hidden" name="StationID" value="10700">
		<input type="Hidden" name="prov" value="CA">
		<input type="Hidden" name="urlExtension" value="_e.html">
	<tr id="dataTableOddRow">
		<td id="dataTableRowHeader">(AE) BOW SUMMIT</td>
		<td id="dataTableRowHeader"><abbr title="ALBERTA">ALTA</abbr></td>
		<td>
			<select name="timeframe" size="1" class="formElement75w" onChange="elementChange(document.stnRequest1,1)">
	<option value="2">Daily</option><option value="3">Monthly</option><option value="4">Almanac</option>
			</select>
		</td>
	<td>
	<select name="day" size="1" class="formElement" disabled><option value="1" >1</option><option value="2" >2</option><option value="3" >3</option><option value="4" >4</option><option value="5" >5</option><option value="6" >6</option><option value="7" >7</option><option value="8" >8</option><option value="9" >9</option><option value="10" >10</option><option value="11" >11</option><option value="12" >12</option><option value="13" >13</option><option value="14" >14</option><option value="15" >15</option><option value="16" >16</option><option value="17" >17</option><option value="18" >18</option><option value="19" >19</option><option value="20" >20</option><option value="21" >21</option><option value="22" >22</option><option value="23" >23</option><option value="24" >24</option><option value="25" >25</option><option value="26" >26</option><option value="27" >27</option><option value="28" >28</option><option value="29" >29</option><option value="30" Selected>30</option><option value="31" >31</option>
		</select>
	</td>
	<td>
	<select name="month" size="1" class="formElement" onChange="elementChange(document.stnRequest1,1)" ><option value="1" >Jan</option><option value="2" >Feb</option><option value="3" >Mar</option><option value="4" >Apr</option><option value="5" >May</option><option value="6" >Jun</option><option value="7" >Jul</option><option value="8" >Aug</option><option value="9" >Sep</option><option value="10" >Oct</option><option value="11" Selected>Nov</option><option value="12" >Dec</option>
		</select>
	</td>
	<td>
	<select name="year" size="1" class="formElement" onChange="elementChange(document.stnRequest1,1)"><option value="1998" >1998</option><option value="1999" >1999</option><option value="2000" >2000</option><option value="2001" >2001</option><option value="2002" >2002</option><option value="2003" >2003</option><option value="2004" >2004</option><option value="2005" >2005</option><option value="2006" >2006</option><option value="2007" Selected>2007</option>
	</select>
	</td>
	<td>
	<input type="submit" name="stnSubmit" value="Go" class="formElement">
</td>
</form>

And the parser:

if ($chunk =~ /.*StationID.*?"(\d+)".*?prov.*?"(\w+).*?TableRowHeader">(.*?)<.*abbr title.*?>(\w+).*?/s) {
     my $stationid = $1;
     my $province = $2;
     my $name = $3;
     my $abbrprov = $4;
}

This is a multi-line regex (hence the //s, like //g is global, //i is case insensitive, //gi is both g and i, etc), and a good example of non-greedy matching. It snags everything up until StationId, then the next quotation mark followed by numbers, and captures those numbers. It comes out as “10700″.

Does the same thing following “prov” up until the next word characters in quotation marks, and captures those. As .* rather than .*?, it would have grabbed “data”, which precedes TableRowHeader (inside the same parenthesis). Comes out as “CA”.

Grabs everything from TableRowHeader”> until the next < Comes out as “(AE) Bow Summit”.

Drops everything up until the next < after “abbr title”, then captures all word characters. “ALBA”

These are all assigned to variables via backreferences. $1, $2, $3, $4 are the groups in order. It’s worth noting that (at least in .NET), named backreferences are assigned numbers BEFORE regular backreferences. So (?<a>a)(b)(?<c>c)(d) would be acbd as ${0}${1}${2}${3}.

Another example:

04:26:23 [2] Error creating WLAAAP06.FS8 = 1 : Unrecognized KGFXENG Error Code

And the parser:

re.match(line, r'^(?P<time>.*?)\s+\[(?P<engine>\d+)\]\s+(?P<error>.*?(KGFXENG|LeadTools).*)'

Grabs everything from the beginning of the line until the first space as “time”. Comes out as “04:26:23″.

Then skips whitespace and a bracket (escaped with \[) and grabs one or more numbers (\d+) as "engine". Comes out as "2", of course. Skips a space, then captures anything which contains "KGFXENG" or "LeadTools" as "error". Basically, the rest of the line.

This line, for instance, wouldn't match, and nothing in the regex would be captured:

00:15:18 [1] Error producing WPATAZ00.FSD = F088 : Error while saving the graphic

These are used later with this:

message = "ERROR: %s %s: %s" % (re.sub(r'.*?([A-Za-z]+Engine[A-Za-z]*?)(Errors)?.*', r'\1', 
                         logfilename), 
                         engine, 
                         match.group('error'))

“logfilename” is something like “2008_Oct_07__ProductEngineErrors.log”. This grabs everything up until A through Z (uppercase or lowercase) one or more times followed by Engine, optionally followed by something else (*, though ? would have worked if I said r’Engine([A-Za-z]+)?’). It stops on Errors, if it exists (the question mark afterwards), and replaces the entire name with the first backreference (”ProductEngine” in this case).

Last example is a nested bitch of increasingly complicated rules:

#Match plain ol' timezones
if ($brpos =~ /^\[(\w+)\](.*)/)
{
	$DateZone = $1;
	$newname = $2;
}
#Match timezones with a day modification, and grab that along with the +/-
elsif ($brpos =~ /^\[(\w+)(\S\d+)\](.*)/)
{
	$DateZone = $1;
	$TempDay2 = ONE_DAY * $2;
	$newname = $3;
}
#Check for a delete flag
elsif ($brpos =~ /^(\d)\[.*/)
{
	$DeleteFilesStatus = $1;
	#If the status is one, we want to capture everything after the timezone as the DeleteName
	if ($DeleteFilesStatus == 1)
	{
		if ($brpos =~ /^(\d)\[(\w+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$DeleteFilesNames = $3;
			$newname = $3;
		}
		elsif ($brpos =~ /^(\d)\[(\w+)(\S\d+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$TempDay2 = ONE_DAY * $3;
			$DeleteFilesNames = $4;
			$newname = $4;
		}
	}
	#Otherwise, the DeleteName is in more brackets
	elsif ($DeleteFilesStatus == 2)
	{
                #Grab it all, but without a time modification
		if ($brpos =~ /^(\d)\[(\w+)\]\[(.*\.\w+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$DeleteFilesNames = $3;
			$newname = $4;
		}
                #Grab it with a time modification
		elsif ($brpos =~ /^(\d)\[(\w+)(\S\d+)\]\[(.*\.\w+)\](.*)/)
		{
			$DeleteFilesStatus = $1;
			$DateZone = $2;
			$TempDay2 = ONE_DAY * $3;
			$DeleteFilesNames = $4;
			$newname = $5;
		}
	}
}

Examples of what I’m catching (hopefully in order). The stuff in brackets later is filled in for date/time stamps:

[EDT]DOV-F-[MM][dd][yy][hh].csv
[CST-1][MM][dd].act
1[PDT]Actual[yy][MM][dd][hh][mm].csv
1[EST-3]KLGA[yy][MM][dd].mtx
2[EDT][WBD*.txt]WBD[yy][MM][dd]05.txt
2[MST+2][WSM*.txt]WBD[yyyy][MM].txt

Sadly, I’m out of work for the night, but these matches aren’t that complicated. Lots of escaping brackets, and use of the \S character to match “-” or “+”, then grabbing the rest of them. I may write more tomorrow…

Tags: , , , ,

categories General

0 RESPONSES comments
OCTOBER 8TH, 2008
By RYAN

RosterParserFixed.XmlParser parser = new RosterParserFixed.XmlParser()

Fixed the nesting problem. Fixed item parsing. Item stats for nested ones units show up now. As with the Ruby parser, throw different combinations at it and see what happens.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Text.RegularExpressions;
using System.Xml;
using System.Xml.Serialization;
using System.Xml.Schema;
using System.Xml.XPath;
 
namespace abparser
{
    class Program
    {
        static void Main(string[] args)
        {
            RosterParserTest.XmlParser parser = new RosterParserTest.XmlParser();
            parser.ParseRoster(@"C:\Temp\de7th.rst", @"C:\Temp\output.xml");
            Console.ReadLine();
        }
    }
}
 
 
 
namespace RosterParserTest
{
    class XmlParser
    {
        static XmlDocument Roster = new XmlDocument();
 
        static XmlElement rootElement = Roster.CreateElement("", "Army", "");
 
        public static string RemoveWhitespace(string str)
        {
            try
            {
                //Ryan's Regex
                return new Regex(@"(\s+|\{.*?\}|\(.*?\)|\/+|\.+)").Replace(str, String.Empty);
            }
            catch (Exception)
            {
                return str;
            }
    }
 
        public void ParseNestedXML(XmlElement thisElement, XmlElement rosterElement)
        {
            bool linkUnitStatsDone = false; //This is a dirty hack.
 
            string replaceMe = thisElement.GetAttribute("name").ToString();
 
            replaceMe = RemoveWhitespace(replaceMe);
 
            XmlElement baseElement;
 
            XmlNodeList linkUnitStatNodeList = thisElement.SelectNodes("./link | ./unitstat");
 
            /*Grab the last of the PascalCase names.  HarGanethExecutioners becomes Executioners
             * SupremeSorceress becomes Sorceress, etc.  Replace the rest of the name with a backreference */
            string regexMatcher = Regex.Replace(replaceMe, @".*?([A-Z][a-z]+)$", "${1}"); 
 
            //This way, it'll actually parse the NodeList for stats in nested things.
            if (Regex.IsMatch(rosterElement.Name.ToString(), regexMatcher)) //So that I don't get duplicate empty nodes.
            {
                baseElement = rosterElement; //Adding to the previous node in the tree.
 
                foreach (XmlElement parseElement in thisElement)
                    {
                        if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                        {
                            ParseNestedXML(parseElement, baseElement); //Parsing out nested.
                        } //if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                        else if (!linkUnitStatsDone)
                        {
                            ParseLinkUnitStats(linkUnitStatNodeList, baseElement);
                            linkUnitStatsDone = true; //Hack implemented.
                        } //else if (!linkUnitStatsDone)
                    } //foreach (XmlElement parseElement in thisElement)
            }
            else
            {
                baseElement = Roster.CreateElement(replaceMe);
 
                foreach (XmlElement parseElement in thisElement)
                {
                     if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                    {
                        ParseNestedXML(parseElement, baseElement); //Whee recursion.
                    } //if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                    else
                    {
                        ParseLinkUnitStats(parseElement, baseElement); //This has always worked.
                    } //else
 
                    rosterElement.AppendChild(baseElement); //Add to the local node.
 
                    rootElement.AppendChild(rosterElement); //Add to the Army node.
                } //foreach (XmlElement parseElement in thisElement)
 
            } //else
 
        } //public void ParseNestedXML(XmlElement thisElement, XmlElement rosterElement)
 
        public void ParseRoster(string path, string output)
        {
            XmlDocument parsingRoster = new XmlDocument();
 
            parsingRoster.Load(path);
 
            XmlNodeList parsingElements = parsingRoster.SelectNodes("/document/squad");
 
            foreach (XmlElement thisElement in parsingElements)
            {
                XmlElement rosterElement = Roster.CreateElement("Unit");
 
                ParseNestedXML(thisElement, rosterElement);
            } //foreach (XmlElement thisElement in parsingElements)
 
            Roster.AppendChild(rootElement);
 
            Roster.Save(output);
        } //public void ParseRoster(string path, string output)
 
        public void ParseLinkUnitStats(XmlElement parseElement, XmlElement baseElement)
        {
 
            foreach (XmlElement correctElement in parseElement)
            {
                if (correctElement.HasAttribute("name"))
                {
                    string subReplaceMe = correctElement.GetAttribute("name").ToString();
 
                    subReplaceMe = RemoveWhitespace(subReplaceMe);
 
                    XmlElement addElement = Roster.CreateElement(subReplaceMe);
                    if (!Regex.Match(subReplaceMe, @"(Left|Worker|Helper|Pts|Coun|Group)").Success)
                    {
                        if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
                        {
                            //Console.WriteLine("Found an item (XmlElement)");
                            ParseNestedXML(addElement, correctElement);
                        }
                        else if (correctElement.HasAttribute("description"))
                        {
                            addElement.InnerText = correctElement.GetAttribute("description").ToString();
                        } //if correctElement.HasAttribute("description"))
                        else if (correctElement.HasAttribute("value") && (Regex.IsMatch(correctElement.GetAttribute("value"), @"[^0|-]")))
                        {
                            addElement.InnerText = RemoveWhitespace(correctElement.GetAttribute("value").ToString());
                            baseElement.AppendChild(addElement);
                        } //else if (correctElement.HasAttribute("value"))
                    } //else
                    if (parseElement.HasAttribute("basename"))
                    {
                        /*It's a non-dwarf item.  Whee!  They don't show up in the XmlNodeList one.
                        Get rid of newlines and periods at the end, then set it as the InnerText
                        This doesn't catch cases where the item has other properties inside it, but
                        I haven't seen those */
                        baseElement.InnerText = Regex.Replace(parseElement.GetAttribute("itemsummary"), @"(\\n|\.)", String.Empty);
                    }
                } //if (correctElement.HasAttribute("name")
 
 
            } //foreach (XmlElement correctElement in parseElement)
 
        } //public void ParseLinkUnitStats(XmlElement parseElement, XmlElement baseElement)
 
        public void ParseLinkUnitStats(XmlNodeList parseNodeList, XmlElement baseElement)
        {
            foreach (XmlElement correctElement in parseNodeList)
            {
                if (correctElement.HasAttribute("name"))
                {
                    string subReplaceMe = correctElement.GetAttribute("name").ToString();
                     subReplaceMe = RemoveWhitespace(subReplaceMe);
 
                    if (!Regex.Match(subReplaceMe, @"(Left|Worker|Helper|Pts|Coun|Group)").Success)
                    {
 
                        XmlElement addElement = Roster.CreateElement(subReplaceMe);
                        if (correctElement.HasChildNodes && correctElement.InnerXml.Contains("entity"))
                        {
                            //Console.WriteLine("Found an item (XmlNodeList)");
                            ParseNestedXML(addElement, correctElement);
                        }
 
                        if (correctElement.HasAttribute("description"))
                        {
                            addElement.InnerText = correctElement.GetAttribute("description").ToString();
                            baseElement.AppendChild(addElement);
                        } //if (correctElement.HasAttribute("description"))
                        else if (correctElement.HasAttribute("value") && (Regex.IsMatch(correctElement.GetAttribute("value"), @"[^0|-]")))
                        {
                            addElement.InnerText = RemoveWhitespace(correctElement.GetAttribute("value").ToString());
                            baseElement.AppendChild(addElement);
                        } //else if (correctElement.HasAttribute("value"))
                    } //else
 
                } //if (correctElement.HasAttribute("name"))
 
            } //foreach (XmlElement correctElement in parseNodeList)
 
        } //public void ParseLinkUnitStats(XmlNodeList parseNodeList, XmlElement baseElement)
 
    } //class XmlParser
 
} //namespace RosterParserTest

Tags: , ,

categories General

0 RESPONSES comments
OCTOBER 3RD, 2008
By RYAN

Ugh.

I’m already not that fond of working with XML in .NET. Here are a couple of fixes:

public static string RemoveWhitespace(string str)
{
    try
    {
        return new Regex(@"(\s+|\{.*?\}|\(.*?\)|\/+|\.+)").Replace(str, String.Empty);
    }
    catch (Exception)
    {
        return str;
    }
}

Which actually gets rid of the crap in the braces, parentheses, etc (as well as getting rid of periods).

Secondly, I loathe empty nodes (stats, etc).

replaceMe = RemoveWhitespace(replaceMe);
Console.WriteLine(replaceMe);
if (replaceMe != String.Empty) 
{
    XmlElement baseElement = Roster.CreateElement(replaceMe);
 
    foreach (XmlElement parseElement in thisElement)
    {
        if (parseElement.HasChildNodes && parseElement.InnerXml.Contains("entity"))
        {
 
            ParseNestedXML(parseElement, baseElement);
        }
        else
        {
            foreach (XmlElement correctElement in parseElement)
            {
                if (correctElement.HasAttribute("name"))
                {
                    string subReplaceMe = correctElement.GetAttribute("name").ToString();
 
                    subReplaceMe = RemoveWhitespace(subReplaceMe);
 
                    XmlElement addElement = Roster.CreateElement(subReplaceMe);
 
                    if (correctElement.HasAttribute("description"))
                    {
                        addElement.InnerText = correctElement.GetAttribute("description").ToString();
                        baseElement.AppendChild(addElement);
                    }
                    //Bye, stats with a value of zero or a hyphen!
                    else if (correctElement.HasAttribute("value") && (Regex.Match(correctElement.GetAttribute("value").ToString(), @"[^0|-]").Success))
                    {
                        addElement.InnerText = correctElement.GetAttribute("value").ToString();
                        baseElement.AppendChild(addElement);
                    }
 
                }
            }
        }
 
        rosterElement.AppendChild(baseElement);
 
        rootElement.AppendChild(rosterElement);
    }
}

I find it kind of ironic that recursion is used after bitching about recursion. I’ll probably take a look at the nesting problems, and whatnot this weekend, assuming I have any time.

I wonder if it’s possible to get a job doing nothing but writing regular expressions…

Tags: , ,

categories General

0 RESPONSES comments
OCTOBER 2ND, 2008
By RYAN

Parser, part deux

Fixed the nested <item> blocks. Ran into another problem where it didn’t parse nested characters and their items properly, then yet another where some Gifts of Khaine (and probably other things I haven’t seen in either list) are essentially nested worthlessness. Fixed code for it:

def parseitem(d, addto)
  added = addto.add_element(d.attributes["name"].gsub(/\s+/, '')).add_text(d.attributes["description"].gsub(/\./, ''))
  d.elements.each('link') do |ele|
    unless ele.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
      if !ele.attributes["description"].nil?
        added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text(ele.attributes["description"])
      else
        #This is necessary for some Gifts of Khaine, apparently
        if added.text == ele.attributes["name"]
          print "Found duplicated #{added.text}!\n\n"
          added.text = ''
        end
        added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text("True")
      end
    end
  end
end
 
def parsenested(process, addto)
  #Try to guess if it's a champion, character in the unit, or item
  process.elements.each('entity') do |d|
    if d.attributes["statset"] =~ /Normal/
      #It's a character, crew, or mount.  Figure out which
      if d.attributes["totalcost"] !~ /^0/
        #It's a champion or character
        adder = addto.add_element("champion")
        puts "Found champ\n"
        parse(d, adder)
      else
        #It's crew or the like
        adder = addto.add_element("crew")
        puts "Found crew\n"
        parse(d, adder)
      end
    else
      #It's an item
      puts "Found item\n"
      if addto.elements["item"].nil?
        @adder = addto.add_element("item")
      end
      parseitem(d, @adder)
    end
  end
end

Tags: , ,

categories General

2 RESPONSES comments
OCTOBER 1ST, 2008
By RYAN

Parser

Ok, boring. I didn’t spend as much time working on it tonight as I intended to, but it parses the Dwarf roster, at least, fine. It does not parse the Dark Elf roster properly (namely, it doesn’t pull the description out of items or Gifts of Khaine, and it doubles up the <item> tag for reasons I’m not sure of), but that’ll get fixed when I’m at work tomorrow.

Ruby code:

#!/usr/bin/ruby
require "rexml/document"
require "pp"
require "rexml/formatters/default"
include REXML
 
inputxml = File.read('dwarfroster.rst')
@roster = Document.new inputxml
 
@army = Document.new.add_element("army")
 
def parsenested(process, addto)
  #Try to guess if it's a champion, character in the unit, or item
  process.elements.each('entity') do |p|
    #puts p
    if p.elements["link"].has_elements?
      #Recursively run through these to figure out what the hell it is
      if p.elements["link/entity"].attributes["itemsummary"].any?
         adder = addto.add_element("item")
         puts "Found nested\n"
         parsenested(p.elements["link"], adder)
      else
        #This is really just stubbed out, since I haven't seen it
      end
    elsif p.attributes["statset"] =~ /Normal/
      #It's a character, crew, or mount.  Figure out which
      if p.attributes["totalcost"] !~ /^0/
        #It's a champion or character
        adder = addto.add_element("champion")
        puts "Found champ\n"
        parse(p, adder)
      else
        #It's crew or the like
        adder = addto.add_element("crew")
        puts "Found crew\n"
        parse(p, adder)
      end
    else
      #It's an item
      puts "Found item\n"
      if addto.elements["item"].nil?
        @adder = addto.add_element("item")
      end
      added = @adder.add_element(p.attributes["name"].gsub(/\s+/, ''))
      p.elements.each('link') do |ele|
        unless ele.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
          added.add_element(ele.attributes["name"].gsub(/\s+/, '')).add_text(ele.attributes["description"])
        end
      end
    end
  end
end
 
def parse(s, addto)
    #In some cases, the basename differs (i.e. Supreme Sorc vs. High Sorc)
    #Also, it'll pick up whether there's a champion in the unit by the diff
    #of base and count
    %w[basename count base].each do |b|
      if s.attributes[b].any?
        addto.add_element(b).add_text(s.attributes[b])
      end
    end
    stats = addto.add_element("stats")
    s.elements.each('unitstat') do |a|
      #unit.fetch(:stats) { |el| unit[el] = {}}
      #I don't want blank stats
      if a.attributes["value"].any? && (a.attributes["value"] !~ /(0|-)/)
         stats.add_element(a.attributes["name"].gsub(/\s+/, '')).add_text(a.attributes["value"])
      end
    end
    s.elements.each('link') do |link|
      if link.has_elements?
        #Figure out what the hell it is
        parsenested(link, addto)
      else
        #unitatt = unit.add_element("attributes")
        #Rip out the name if it doesn't have "Helper, Worker, Points Left, or Cost"
        unless link.attributes["name"] =~ /(Worker|Helper|Cost|Left)/
          #Get rid of the stuff in braces AB puts in
          if addto.elements["attributes"].nil?
            @unitatt = addto.add_element("attributes")
          end
          @unitatt.add_element(link.attributes["name"].gsub(/\{.*?\}/, '').gsub(/\s+/, '')).add_text('true')
        end
 
      end
    end
end
 
@roster<