NOTE: All of this has been massively superseded (here in 2010) with WordPress and Zenfolio together with Lightroom. Again, I leave this here for “historical” purposes…
I’ve been working off and on for a week or more on a major workflow automation project for the blog. Let me start off by quickly pointing out what this means for the readers in terms of new features:
So what’s the “workflow automation” all about? Well, generating galleries and linking them to a blog article is already a multi-step, albeit relatively easy, process. The addition of the slideshows, though, added a significant extra layer of work in the form of carefully structured XML configuration files. Since I want to be consistent and I don’t want a ton of extra work to stymie the urge to post new galleries, I figured it was time to streamline the process as much as possible. Past experience has told me that once the posting process becomes too complex, spur of the moment and casual posting grinds to a halt. I want the process to be trivial enough that I won’t have any excuses not to post more frequently. 🙂
The behind-the-scenes story of how this process works is, of course, disturbingly geeky. Fortunately, there are plenty of other fellow geeks out there that might be interested. The rest of this piece (and I predict it will be rather long) will cover the gist of what went into this project thus far. If the binding together of iView MediaPro, AppleScript, linux, OS X, ssh, scp, a big chunk of python code, MySQL databases and various bits of HTML and XML technology interest you… read on.
Here’s a linked list to the various tools and technologies use:
Detailed description below the fold…
Background
In a nutshell, making a photo gallery for this site is a pretty straight forward process. I currently do all of my photo management with iView MediaPro on the Mac.
It’s a fantastic piece of software, all in all, and I manage literally tens of thousands of images with it across multiple drives and servers. I use its templatable HTML gallery feature to generate the halfpress galleries. By doing this I have a consistent means of pushing any catalog I create in iView out with just a few clicks of the mouse. I generally just write them out to a specific directory within the structure of the blog and then code a link into the article.
The random image box at the top right of the halfpress pages was one of my earlier developments. Using iView’s Text Table feature, I could write out a text file in CSV format that contains the filenames, captions, etc., of the images in the current catalog. I would then feed that CSV file through a CGI interface I wrote in python on the halfpress server. This script would parse the data into a MySQL database that had all the information needed to build HTML links to the thumbnails and images in any gallery. The database doesn’t need to contain the photos… just the paths and filenames as they appear on the server. The little CGI script that selects images for the page simply makes a random selection of a photo from the database on each page load, builds the relevant HTML to display the existing thumbnail and embeds the image-specific link into the gallery. Voila… lightweight random images that are clickable taking you right into gallery it came from, and without duplicating any of the assets.
So, in a nutshell, the pre-automated workflow process was:
Once I became attracted to using the flash-based SlideShowPro, though, the complexity went up a bit. Rather than go all flash, I want every gallery to be available in both forms so the user can choose what they like best. This means I do everything listed above plus create an XML configuration file for SlideShowPro that includes the somewhat complex, server-specific paths, captions, etc. I’d then be creating two links in the text to the two viewing options.
My first attempts at automating the XML creation became possible with the new iView MediaPro 3.0 release. It has the ability to not only write out XML dumps of catalog data, but also the option to template the XML via XSLT which is, basically, the CSS of the XML world. I wrote an XSL template that did much of the work of making a SlideShowPro configuration file, but it was still going to require per-gallery hand editing of the XML.
The Automated Workflow
Basically, what I’ve ended up with is a combination of AppleScript on the desktop and Python on the server with MySQL in the middle to hold the data.
Here’s how the process goes:
<!--#include virtual="/galleries/st-thomas/include.html" -->
Behind The Scenes: Desktop
What’s happening behind this script is fairly complex and, despite initial appearances, more stable than you might think.
The AppleScript does a series of things:
This is my first hands-on experience with AppleScript and, frankly, I’m not especially fond of it. I spent a lot of time fighting with the syntax and screaming about the lack of coherent documentation. I think this is one of those cases where the “english” syntax is frustrating since I’ve already spent my life thinking in the more abbreviated, algol-ish, logic-driven syntax of various programming languages.
The use of SSH and SCP was a matter of convenience and relative security. I didn’t want to store passwords in any of the scripts, so I used the public-key, “passwordless” login capabilities of ssh to pull this off smoothly. I have it highly restricted, so I think it’s a sane tradeoff… not to mention everything is encrypted with 2048 bit keys. This is also testimony to the power of the UNIX underpinnings of OS X that I can call these tools without a second thought and interact with the Linux server transparently.
Behind The Scenes: Server
When I first started thinking this project through, I considered doing the MySQL work on the desktop side. I quickly decided, though, that I’d separate it out so only those actions most directly related to the desktop’s real role (photo management) would reside here and the rest would be done on the server. This also makes it more likely that everything will continue to work nicely when I’m traveling and behind bizarre proxies and firewalls that might crap out the MySQL connectivity. As it stands nows, if the SSH and SCP stages were somehow blocked when traveling, I can hand-execute the server side with a single statement once I get the files transferred by other means (FTP, for instance).
The server side of this process is all encapsulated in a fairly efficient Python application I wrote that manages all the file creation, filesystem manipulation and MySQL interaction.
The system is bound together with a core MySQL database consisting of two simple tables. One table contains a basic index of the galleries including the filesystem path, the formal gallery title and the comment that should appear in the slideshow index. The other table has an entry for every image in every gallery on the system including the filenames, paths, captions, EXIF data, display order within the gallery and a flag indicating whether a given image should be included in the random pool or not. Note that there are no images in the database itself… just references to the existing thumbnails and full sized images as they appear in the gallery. This makes it far more efficient on several levels.
Basically, every incoming gallery carries with it an XML dump of the catalog that contains everything I want to share: captions, EXIF metadata, etc. My python application, when called from the AppleScript system via SSH or on the command line, does the following:
Notes and Comments
As with any system like this, the design is predicated on a consistent layout to the filesystem. Everything is stored in a standardized format within the core database and, because the filesystem is laid out in pre-defined manner, HTML links and other references can be built on the fly by the python script for use in the XML and the server-side includes.
Since all the galleries exist within a certain directory, all that is needed to be known is the name of the enclosing directory and the individual names of the images within. The thumbnails and full-sized images use the same filenames and are stored in parallel directories and scaled to their required sizes. Since iView puts out its HTML Gallery using a set layout, a single filename reference in the database can yield either the thumbnail or the full-sized image just by pre-pending the enclosing directory names.
In the St. Thomas gallery, for instance, image “_mg_1346.jpg” is stored in the database with a gallery name of “st-thomas”, its relevant caption, EXIF data, etc. A reference to this file in any of its incarnations is easily built as follows:
SlideShowPro, as stated before, is a flash application that gets it marching orders from an XML configuration file. I’ve described aboved how the XML files are generated based on the consistent design of the filesystem layout and HTML gallery assets.
I’ll mention that I have at least three copies of the SlideShowPro app “published” from Flash Studio 8.0 to fit each need: the random slideshow, the master slideshow gallery with index, and the individual gallery slideshows. The individual galleries share a single instance of the swf file and read the XML file based on a path-relative call.
Closing Thoughts
To summarize, my workflow is now:
That’s pretty much it. All of the galleries, random images, slideshows, slideshow indexes, etc., take care of themselves as a result of the script(s). I can focus on shooting, editing and far less on gluing the blog together.
Over time I’ll continue to test, tweak and likely make changes to the overall design. This is just the first functional incarnation of the concept and it will probably improve over time. I don’t claim that the code is the most efficient thing on the planet nor do I assume the overall design is nearly as ideal as it could be. That’s half the fun, though, since there are a zillion ways to approach any one problem with these kinds of tools.
If anynone has any questions or comments, don’t hesitate to email me or use the comment fields below.
Wow, really cool. I use iView media myself and love it. I do a lot of custom templates but nothing to the extent that you have done.
Congratulations. Looks great. The only thing I would change is to make the iView export look a little more slick like the rest of the site.
David – thanks! And, yes, I agree entirely concerning the gallery template being kind of bland… it’s definitely something I’m going to focus on soon! 🙂
This is great!
I was wondering what program you are using in the last screen shot?
Thanks!
The last screenshot in the post is the top part of the window for MySQL Query Browser. It can be found here:
http://dev.mysql.com/downloads/query-browser/1.1.html
A year or so later, I wonder if you’re still following this method and what you now think of it.