Sunday, November 09, 2008

to XMP or not to XMP (part II)

(Goto Part I)

Why XMP?


Ok, so I'm supposed to talk about XMP. Here's how it is. I think the best way to organize pictures is the same way that you should organize mp3s; metadata. Sure, you can create a database and store all of your info about each picture so that you can do nifty queries and stuff. Maybe even store the pictures themselves in the database. I know that certainly crossed my mind. That's one of the reasons why I went down the road with Gallery at the start. But it won't be long before you've copied a few pictures around and realized that all of your data telling people about when the picture was taken and who's in it is sitting back in your database where nobody can reach it.

Afterward I started thinking about all the reasons I liked ID3Tags for my mp3s. I won't go into all the pros and cons here, I'll just summarize that I think metadata tags is the best option. Why can't I have ID3Tags for images? It's something I never really paid much attention to before. I knew there was some sort of tagging going on cause some image viewers could show me various tidbits of information about my pictures that went beyond the normal information I thought would be stored in a .jpg. Things like ISO speed and focal length.

Some quick research on Wikipedia had the explanation (you'll find that I link to Wikipedia quite a bit). The are essentially 3 metadata tag formats for image files: EXIF, IPTC and XMP. Exif is an earlier standard specified in 1998 and is generally supported by all digital cameras. It has many shortcomings however that include the fact that it is no longer maintained by a governing standards body. Also, many camera manufactures use their own modifications of the structure and the data can easily be corrupted by other programs that don't fully parse and recode the data when modifying it. The biggest drawback though is that it is only supported for JPG and TIFF images. In any case, the odds are pretty good that all of your digital pictures have an EXIF tag.

Next is IPTC. This was actually developed as a standard for specifying metadata in news items and any media in general. What we commonly see in photos is a subset of this standard. Just like EXIF, it is really only supported in JPG and TIFF images.

This brings us to XMP. XMP was developed by Adobe in 2001. It is an XML schema that actually incorporates the IPTC definitions. The best part though is that it isn't limited to just JPG and TIFF, it is also supported for these file types too: JPEG2000, GIF, PNG, HTML, PostScript, PDF, SVG, Adobe Illustrator, and DNG.

I was a little hesitant when I found out that XMP is essentially under the control of Adobe but I felt a bit better when I discovered that they released a software toolkit for it under a BSD license. I'm still reading the documentation for the specification and the toolkit so I'll post some more info when I'm done.

Friday, November 07, 2008

to XMP or not to XMP (part I)

Ok, so I've got my mp3s organized. What's next? Well, I'm really overdue getting my pictures in order and that's what I decided to tackle last week.

Here are my main objectives:
  • Private storage. The various web services are pretty good and feature rich but I really don't want to trust my pictures to somebody else.
  • Archive originals. I want to keep my originals completely intact. Edits will always be done to copies.
  • Detect duplicates. Pictures have a habit of multiplying onto every writable device you have.
  • Tagging. I want to be able to search through my collection based on people, things, locations, events and anything else notable.
  • Editing. I don't need anything complicated, just the basics: crop, resize, color adjustment, red eye removal and rotate.
  • Sharing. Even though I don't want to trust online services for preserving my memories, I do eventually want others to see them.
If I was thinking clearly, I would have thought about each of these points and started at the beginning by answering the question "How do I get my pictures from my camera into an archive?" Instead, I jumped right in and tried installing Gallery 2.

Gallery 2 is just a single piece of a much larger "web server" puzzle. To run Gallery 2 you need a web server (Apache), a database (MySQL or PostgreSQL), PHP, and some graphic libraries (ImageMagick or NetPBM). You don't have to use these specific components but they are probably your best choices. I'm sure if I had read the Gallery documentation a little more carefully I would have realized it didn't quite cover everything I wanted but who does that, right?

I downloaded Apache and installed it on my server computer, easy. Next I downloaded PHP and installed that.. easy again. I had used MySQL before and knew that would be easy so I decided to try Postgres because I had read good things and wanted to try it.. and what do you know?.. more easy! This was going pretty good.

The next step was the big one, installing Gallery. To their credit, they are one of the best documented open source projects I've encountered in awhile. They have a very good help file that gives you detailed step-by-step instructions on how to install it. It got a little confusing in parts when they try to explain the different steps depending on which components you use but it is light years ahead of most walk-thrus you will find. There is alot to read though and you must read everything or you will have problems.

Finally, I got to the 'setup' part where you actually run a gallery .php script through your web browser served up by your apache server (or whatever you are using as a web server) and it will take you through all the steps and checks to get Gallery configured and running. This was all going good too until I got to the part that asked for the DB info. No matter what I did, it could not talk to my Postgres server. I spent many hours trying to find information on the web but in the end, I had to bite the bullet and install MySQL. Once I did that, everything went fairly smoothly.

Gallery runs very well and I recommend it to anyone that wants to serve up their own web albums. You can customize it quite a bit and set up finely controlled permissions for users. Be prepared to dedicate the better part of a day installing it and getting it running. As good as the documentation is, you will need to know what you are doing.

I immediately logged in and started uploading pictures after it was all working. It didn't take long to figure out that it wasn't quite what I was looking for. I was hoping I could get it all setup and then go to my wife and say "Look hun! It's easy. You just upload the pictures and then add some tags for each one. You can click here to fix the red-eyes and you are done." Oh well, back to Facebook for now.

I'll get to the XMP stuff in Part II, promise.

Sunday, November 02, 2008

photo stitching

One thing I like to do sometimes when taking pictures with my digital camera is panoramas. You take a bunch of overlapping pictures from the same spot and later you load them up in your favorite image editor and put them together into one wide panoramic view. That's how it's usually done anyways.

If you are lucky, you may have a camera with panorama stitching capabilities built in. I've never tried such a thing but I'm sure it gives satisfactory results for most (feel free to comment on this with your own experiences). Most of the time though, you will use an image editor and most of the time it will give you decent results. When it doesn't, you usually have the option of 'helping' out and sliding the images around to get a better fit. This is usually the point you discover that the camera wasn't quite as level as it should have been for each shot and no matter how much you play around, you can't get a perfect stitch. And that's just for plain horizontal panoramas. What if you wanted to do some arbitrary stitching of a complete mosaic? You know, take a bunch of pictures horizontally and vertically to capture some great scene in far more detail than you ever could with a single picture. Good luck trying to get them all stitched together.

At least, that's what I would be saying before I found Hugin. This program is certainly the king of all photo stitchers. In fact, it is a cleverly unified collection of a few different photo processing technologies. It is open source and is available for many platforms including windows. Finding an actual windows binary can be a bit tricky so I've provided a link to the current installer here.

This is a powerful program so if you are really interested in using its full potential, you'll have to poke around in the documentation a bit. In the mean time, it has a wizard based option that should give you pretty good results. The only part that confused me a bit was the 'align' stage. The program actually did all the aligning automatically and presented the results in a viewer. It wasn't immediately obvious what I was supposed to do next. I eventually figured out that I could click around the image to set a center point and then adjust the sliders at the edge to trim off the rough edges. There is no 'proceed' or 'next' button here, you just have to close the window and then click on the button for the 3rd and final stage that actually outputs the final image. This will come out as a .tiff file so you will need to load that into your favorite image editor to convert it to something more portable and usable. I suggest saving it as a .png and archive it away somewhere. You can then use this master to create a .jpg or resize it to fit your needs.

I've only scratched the surface on this program so please, download it and try it out for yourself. I can't wait to try some more complicated stitches.

Saturday, November 01, 2008

getting organized

Lately I've been trying to organize my various collections of media, starting with my mp3 files. It had been many years since I even thought about them so I figured I had better start from scratch looking for software to organize and manage them. Surely things had advanced greatly in that area by now.

One thing that had not changed in my mind is that the best way to manage mp3 files is to ensure that each one has a sufficiently filled out ID3Tag. If each file has a good tag, it is completely trivial to get software to read all those tags in and give you access to your music in incredibly flexible ways. If you are ripping CDs to mp3 then whatever program you are using to rip will most likely fetch track info from CDDB or FreeDB and you will probably end up with propely filled out tags. Sometimes however you may find that you have some tracks with missing or incomplete tags and these can be a pain to fix.

I had a number of tracks with incomplete or bad tags and it was going to take me a very long time to find the correct info and type it in. Fortunately, finding the correct info is alot easier with a site like AMG's allmusic.com and getting it into a tag was even easier with a program I bought many years ago. It was called Tag&Rename. I looked at alot of programs at the time and this one was clearly the best. It had all the functionality you needed for mass tagging and renaming and above all, it could scrape track info from AMG's site! I didn't have to type anything.

Alas, that functionality had to be removed as it was against AMG's TOS. When I updated the software I found that it now used Amazon for this source of data. Amazon is a good source for track info but it is not nearly as extensive as what AMG has. ID3Tag&Rename also provides access to tracktype.org which goes a long way to filling in the gaps. Where Amazon's data falls short is when you start trying to tag tracks earlier than 1990. Sure, you may find the album/track info but you will probably have to intervene to get the original release dates correct as they are more likely to list dates for re-releases or CD versions of earlier LPs.

If you want quick an dirty automatic tag filling you may want to try Winamp. I should mention however that I find much of Winamp completely confusing and should generally be considered a perfect example of how not to design a user interface. But once you figure out how to get it to fill tags though, it can do a pretty decent job. The show stopper for me unfortunately was I couldn't figure out how to tell it to save cover art to each track. I really hate the concept of putting a .jpg in a directory and hoping that it manages to stay associated with the tracks it is referenced by when I move them around and transfer them between computers and mp3 players. Some people will tell you that it's a waste of space to store the same image in 12 or 15 tags but the size is small enough to ignore and the savings in peace of mind is much greater.

Ultimately, I had to give Winamp a miss and replaced it with Media Monkey. I found this to be a more than capable music player and had a reasonably sensible interface. I especially liked the fact that I could tell it to recognize tracks by a check sum based only on the music data. This means it will not easily get confused when I need to fix a tag for example which more often than not will mean that I need to fix the file name as well. By having a check sum based on the data only, Media Monkey won't think it is a new track or think that the previous version has disappeared. My play lists will still be able to find and play the track when it gets to it.

File renaming is the second thing that ID3Tag&Rename does extremely well. It's another reason why you want to have properly filled out tags too. With a single click, you can tell this software (and many others) to rename all of your files using a specific pattern based on almost any tag attribute. It will also create any level of sub directories based on tag info too.

One final consideration for managing your collection effectively is being able to take it offline. If your collection is large, you probably want it backed up to removable media like recordable DVDs. Media Monkey makes some mention of this but I haven't played with it yet. I will mention that I was very please with a program called Music Library. It hasn't been updated since 2006 but I think this is because it's one of those few examples of software that picks one thing to do and does it right. Basically you would backup all of your music on to multiple CDs or DVDs and then feed them into this program one at a time. It will slurp up all the tag info and create a database of it. Once this is done, you can search and create play lists at will. What was really cool though is if you created a play list of songs that spanned 8 or 9 different discs and wanted to play them from your hard drive, it would prompt you for each disc in succession and copy all of the tracks effortlessly.

I used these programs to tag and organize not only all my mp3s but my wife's and kid's music too. They are now all neatly available from a network drive and are safely stored on a RAID 2 array with read-only access. I took a little time to show them how to create play lists and use them to sync to their mp3 players. It's alot easier for everyone. They don't have to worry about copying files around by hand and I don't have to worry about losing files :)

There are many other programs that are pretty good for these tasks as well, far too many to list or talk about here. If you know of any that deserve mention, let me know in a comment.