CodeSushi

My business partner Josh and I have started a new venture, Busted Loop. As part of that venture, we are making stuff and blogging about it. Today was my turn and I blogged about a tool I created, The RackspaceCloud Command Line Tool.

There will be more stuff in the coming weeks. I hope some of you will follow along… It won’t all be as geeky as this post… well.. pretty geeky, but not quite this geeky.

danb Busted Loop, groovy, RackspaceCloud

There have been a few occasions where I have wanted to express historic prices in terms of their inflated value or vice-versa.. e.g. $100 dollars in 2007 was worth about $63 in 1990.



This is actually a very simple thing to calculate… we take the CPI from 2007 (207.34) and the CPI from 1990 (130.65) and do the computation thusly:



$100.0 * (1.0 + ((130.65833333333333 - 207.3424166666667) / 207.3424166666667))



I have written a grails service that grabs the latest CPI data and does this for me… it's just groovy, so it should be easy to use outside of a grails app.

class InflationService {
    boolean transactional = true

    // this is where the live CPI data lives
    String cpiURL = 'ftp://ftp.bls.gov/pub/special.requests/cpi/cpiai.txt'

    // map of year to CPI average String -> double
    def cpiMap = [:]

    // track the last time we updated from the source data,
    // for long lived apps we don't want to get stale.
    long lastLoadTime = 0;

    // always use the current year as the target year
    double calculateCurrentDollars(String fromYear, double value) {
        refreshCPIData(false)

        GregorianCalendar date = new GregorianCalendar()
        def year = date.get(Calendar.YEAR)

        // not sure what the CPI table looks like in january...
        // this should account for no record for the current year...
        // more that 1 missing year is a serious problem
        def toYear = String.valueOf((cpiMap[String.valueOf(year)] != null) ? year : year-1)

        return calculateYearDollars(fromYear, toYear, value)
    }

    double calculateYearDollars(String fromYear, String toYear, double value) {
        refreshCPIData(false)

        def toCPI = cpiMap[toYear]
        def fromCPI = cpiMap[fromYear]
        // here's the actual math...
        // pretty simple when you final get down to it          // TODO - check for nulls or zeros and throw exception.
        return value * (1.0 + ((toCPI - fromCPI) / fromCPI))
    }

    // refresh cpiMap if last update was more than 1 day ago
    // OR forceUpdateNow param is true.
    public void refreshCPIData(boolean forceUpdateNow) {
        long oneDay = (1000 * 60 * 60 * 24)
        // update once per day. (or if forced)
        if (forceUpdateNow || (lastLoadTime + oneDay) < System.currentTimeMillis()) {
            lastLoadTime = System.currentTimeMillis()

            // grab the "tabular" data right from The Man
            cpiURL.toURL().text.eachLine {yearLine ->
                // find all the lines that start with a year..
                // these are the useful bits of the doc
                if (yearLine.matches(/^\s+\d\d\d\d\s.*/)) {
                    // split the line up into individual values
                    def yearList = yearLine.split(/\s+/).findAll {it.size() > 0}
                    // the first item is the year.. hang onto that
                    def year = yearList[0]
                    // the rest of the values are CPI values for each month
                    // (some averages at the end that we ignore)
                    yearList = yearList.tail().collect {Double.valueOf(it)}
                    // average all 12 months for every year...
                    // except the current year could have less than 12 months
                    int idx = yearList.size() < 12 ? yearList.size() - 1 : 11;
                    cpiMap[year] = yearList[0..idx].sum() / (idx + 1)
                }
            }
        }
    }
}

here is a simple test.

class InflationServiceTests extends GroovyTestCase {
    InflationService inflationService;

    void test2007to1990Dollars() {
        assertEquals('got wrong answer', 63.01572800870056,inflationService.calculateYearDollars('2007', '1990', 100.0));
    }
}

It shouldn't matter which direction you want to do the conversion in.. 1990 dollars to 2007 dollars or 2007 dollars to 1990 dollars. The data starts in 1913 (the year the federal reserve was created) and runs about a month behind.. so I write this in September 2008 and the CPI is available through August 2008.

admin cpi, grails, groovy, inflation

I wrote a groovy script that grabs the Maine legislation public hearing schedule and outputs all the items. It kinda looks like this:

committee       : LVA
document number : LD-2261
date            : 2008-04-02 13:00
room            : Room 437 State House
bill title      : I.B. 3, An Act To Allow a Casino in Oxford County

committee       : ACF
document number : LD-2262
date            : 2008-03-26 14:00
room            : Room 206, Cross Building
bill title      : H.P. 1626, An Act Pertaining to the Definition of "Milk"

committee       : BEC
document number : LD-2257
date            : 2008-03-25 13:00
room            : Room 208 Cross Office Building
bill title      : H.P. 1619, An Act To Establish a Uniform Building and Energy Code

I'm going to push this stuff into a database or maybe directly into an RSS feed once I grab a few more bits of data to go with it.



the following is the entire script I used to produce the above output… it's nothing too exciting, but shows how I used groovy and tidy to grab some data from several entirely un-styled documents as a first step in providing some structure to that data for future applications.

import org.w3c.tidy.Tidy
import java.text.SimpleDateFormat

// some date formats we'll use to validate and reformat dates later
SimpleDateFormat dateInHtml = new SimpleDateFormat('EEE MMM dd, yyyy, h:mm a')
SimpleDateFormat dateWeLike = new SimpleDateFormat('yyyy-MM-dd HH:mm')

String base = 'http://www.mainelegislature.org'

// this url is for the index page for all public hearings..
// right now it's hard coded to look 180 days from today
String url = base + '/legis/lio/phSched.asp?DAYS=180'

// get the "index" file and tidy it.
download(url, 'out.xml')

// parse the index page so we can rip through it looking for links we care about.
def indexPage = new XmlParser().parse(new File('out.xml'))

// find all the links in the main hearing schedule document that
// link to specific committee schedules.
// TODO - once I find all the committee codes I can just go after them without this loop
indexPage.depthFirst().grep{ it.'@href'?.contains('phSched.asp') }.each {flag ->
    // find the committee code from the url... we will do something with this later
    String cmty = (flag.'@href' =~ /.*?CODE=(.*?)&.*?/)[0][1]

    // download and Tidy the HTML schedule for each committee
    // this url looks just like the index url with the addition of a
    // "CODE" param which contains a unique code for the committee
    download(base + flag.'@href', 'schedule.xml')
    // parse this sucker
    def scheduleNodes = new XmlParser().parse(new File('schedule.xml'))

    // there are no classes or ids to use to navigate this document..
    // so we have to go by structure...
    // we are looking for all table rows with 4 columns
    // where the first column does NOT contain the string "LD"
    scheduleNodes.breadthFirst().findAll {it.name().localPart == 'tr' &&
            it.children().size() == 4 &&
            !it.children()[0].value().contains('LD')
    }.each { row ->
        // each of the rows we found has the following 4 columns

        // the LD number (legislative document number)
        String ld = findText(row.children()[0])
        // the partial title of the bill... not much use as it is truncated.
        String title = findText(row.children()[1])
        // the date and time of the hearing
        String date = findText(row.children()[2])
        // the room and building in which the meeting will take place.
        String room = findText(row.children()[3])

        // for some reason the white space character right before the am/pm in
        // the html doesn't appear to be a space... so we'll replace it with one.
        // then parse the date into something we like more
        date = date.replaceAll('.pm$', ' PM').replaceAll('.am$', ' AM')

        // print out some details for now
        println "committee       : $cmty"
        println "document number : LD-$ld"
        println "date            : ${dateWeLike.format(dateInHtml.parse(date))}"
        println "room            : $room"
        println "bill title      : $title"
        println ""
    }
}
new File('schedule.xml').delete()
new File('out.xml').delete()

// pass this a node and it takes the first child until it finds a text node and returns that.
// also replaces line breaks with spaces... so... watch that.
private def findText(node) {
    def var = node;
    while (var.class.name != 'java.lang.String' && var.children().size() > 0) {
        var = var.children()[0]
    }
    return var.replaceAll('\n', ' ');
}

// this will grab an html page (url)
// then run it through Tidy to clean it up and save it to outFile.
def download(String url, String outFile) {
    // temporary file which will contain the html in need of a good tidy
    File tmpOutFile = File.createTempFile('out', '.html');

    // write the url to the tmp file
    def file = new FileOutputStream(tmpOutFile)
    def out = new BufferedOutputStream(file)
    out << new URL(url).openStream()
    out.close()
    file.close();

    // run tmp file through tidy
    Tidy tidy = new Tidy();
    tidy.setQuiet(true);
    tidy.setShowWarnings(false);
    tidy.setMakeClean(true);
    tidy.setXHTML(true);
    tidy.parseDOM(new FileInputStream(tmpOutFile),new FileOutputStream(outFile));

    // delete the temp file.
    tmpOutFile.delete();
}

click “expand source” to see the script.



This is one of several irons I have in the fire related to increasing transparency in my local government and providing a modicum of usability to our amazingly scattered and 1996 looking Maine state web services.

admin code, groovy, hearings, java, maine, transparency