In one of my last projects at AxisPhilly*, I was working with Julia Bergman on transparency and looking at open records requests.
The open records request website is a classic case of government website. Well intentioned, but horribly formed data. There’s no clean url to snatch things up, so you can’t directly link to any results. So when Julia asked about getting all the records pulled down into a digital format, I asked Casey if he had any ideas.
Turns out that earlier this year, Brian Abelson wrote a scraper using Selenium at BCNI, and I didn’t totally get the theory at the time. However, I’m hoping this will help someone who’s never used Selenium (or only seen it used for say, integration tests) see the benefit.
Here’s the deal. Selenium is a browser automation tool. Meaning, you can write a script and have it go do things for you. Computers, right? So instead of clicking the “next” button, you tell Selenium to do so. I also really like that it does launch a browser instance, so you can watch what it’s doing. Of course, if you prefer running it “headless” (meaning you don’t have to/get to watch it), Chris Le just published a post on doing that, using the Node.js Selenium package.
So, what does that look like?
I blatantly “forked” (ripped off of) Brian’s example. This is definitely not elegant, but it’s brute forcing its way through pagination to save all the pages to text files, which are easily manipulated. With just a little more work, you could loop through each row and directly save to CSV, if you wanted.
* On AxisPhilly
I don’t know a clear way to write these things out on le internet, but many folks on the internet know by now that I’ve left AxisPhilly. Sean and Erika have spoken openly about how some of the drama there shook out, but all I will say from my end is that I was so happy to get the time and opportunity to work solely on open source, journalistic projects. I was especially happy to work with awesome devs like Casey and Jeff and be in the same room collaborating with great journalists. I start my next gig soon, so there’ll be an announcement of sorts when that happens I suppose. I wish AxisPhilly the best of luck, and hopefully they can continue to put out some good work.