Fairfield 5: Three ZoomWalks

During my March 2012 visit to Fairfield and MUM (Maharishi University of Management), I took photographs for three new ZoomWalk videos. (Earlier ZoomWalks are shown here.) This post will present the new videos and, at the end, discuss some changes in technique from the earlier ZoomWalks.

The Unsecret Path

In Sthapatya Veda, southern entrances are strongly discouraged, so the south face of the MUM campus not only has no entrances, it's fenced off. This can be inconvenient for traveling between half the campus and downtown or Everybody's, so an unofficial but well-tended path has evolved.

The first video is another ZoomWalk where the camera faces forward as the walker goes along this "unsecret" path. Because there are many objects (leaves, etc.) close to the walker and few objects far away, it worked best to have a short distance between the still photographs from which the video is constructed. Too many steps, and the continuity between frames would be lost. For this video, I took only one step between pictures.


The Unsecret Path @ MUM from Ben Branch on Vimeo.

Rotating the Golden Dome

 

The next ZoomWalk is quite different: the camera keeps facing a particular object, in this case the Maharishi Patanjali Golden Dome, as I walk.


Rotating the Dome @ MUM from Ben Branch on Vimeo.

For this video, I took two steps between each photograph. I had originally intended to take only one, but it was a very hot and sunny day. To stay out long enough for one step per photo I would have needed to bring along a snack, some water, and a wide-brimmed hat.

Orbiting the Maharishi Vedic Observatory

 

The third video was accumulated a couple of miles from the MUM campus, in Maharishi Vedic City, where the Vedic Observatory is located. Because of the concentric nature of the observatory, I walked around the structure about halfway out, once facing outwards, to capture the ring of instruments, and once facing inwards, to photograph the structures representing various planets and other astrological constructs. I also looped around one of the instruments.


Orbiting the Maharishi Vedic Observatory from Ben Branch on Vimeo.

I needed to take more photos, less far apart, for the outward facing portion of this video. Those instruments just fly by.

Techniques and Experiences

 

Whereas the "looking forward" ZoomWalks can use automatic methods to improve the alignment of the still images, that would be counterproductive when the camera is facing inward (Rotating the Golden Dome) or outward (Orbiting the Maharishi Vedic Observatory). Objects are supposed to slide from one side to the other through the frames. However, using the raw photos without any alignment whatsoever produces very jerky, almost unwatchable, videos. Thus I had to manually align each frame using some object or segment of the image. For the Golden Dome, it was the ornament on top, the kalash, that was my registration mark. Similarly, in the second half of the Observatory video, it was the cylinder in the center of the observatory. Aligning while looking outwards was difficult because of the lack of a central object; I would use the horizon and hope for the best. Needless to say, this manual aligning took a significant amount of time, and might discourage long-duration rotational ZoomWalks unless I find an automating technique. (The Rotating Dome video is based on about 240 photographs, plus title and credits frames.)

During the generation of the videos I experimented with "doubles." This is an animation technique where two identical pictures are taken of each step of the animation; thus, an animation with 24 frames per second requires only 12 animation steps. Previous ZoomWalks had one original image, then one or two "morphed" images, then the next original image. In shorthand, for images A and B, the frames generated would be, for two morphed images:

1. 100% A
2. 67%A/33%B
3. 33%A/67%B
4. 100%B

In all three of these videos, I doubled the originals by making a copy of each under a new name. (This I was able to automate.) Then, a two-morph video (Golden Dome) would be:

1. 100% A
2. 100% A
3. 67%A/33%B
4. 33%A/67%B
5. 100%B

A one-morph video (Unsecret Path and Vedic Observatory) would be:

1. 100%A
2. 100%A
3. 50%A/50%B
4. 100%B

I believe the higher proportion of original, unblended images produced by doubling yields a better product.

The algorithm for automatic alignment (used only in Unsecret Path) was also tweaked. Before, pairs of images would be compared, and because of the way the list of images was walked through, no more than half could ever be inspected. The new procedure is to align around "anchor points." For instance, consider the sequence of photographs 1, 2, 3, 4, 5, 6. If comparing three at a time, 2 would be the anchor point, and 1 and 3 would be compared to 2. Then 5 would be an anchor point for comparison with 4 and 6. Because of the way the align_image_stack tool works, the anchor point is unmodified; nonetheless, up to two-thirds of the images are alignable. There is no attempt to compare anchors, so occasionally a badly photographed anchor will require manual intervention. It also appears that comparing 3 images at a time (one anchor and two changeable) works better than comparing 5 at a time (one anchor and four changeable). Comparing 5 at a time puts images that are too different in the same pool, so attempting to reconcile them makes no sense.

Here is a still photograph to satisfy the 'Recent Posts' and 'Popular Posts' sidebars.

67 Days of Spring

Earlier I used my Wingscapes PlantCam for several shorter time lapse projects. In this posting I present a two-month time lapse, from mid-March to mid-May, with the camera aimed into the woods in back of our house. You'll see daffodils bobbing up and down, a brief snowfall, and the greening of the woods from bottom to top (the forest floor always greens up first, and the tall trees leaf out last). When the weather switches from cloudy to sunny, there's a flash of light and moving shadows! Also, at around the 57 second mark, the invasive bush honeysuckle in the middle of the frame gets chopped down, giving you a better view into the woods beyond.

Now, some details. The camera took a picture every 30 minutes between 10AM and 3PM each day. Once I downloaded the photos and began constructing a video, I saw that the images whipped by much too fast; it would have been preferable to take a picture every 15 or even 10 minutes. Not willing to wait and try again in 2012, I used the interpolation techniques from the ZoomWalks to add intermediate frames. I also continued my research into encoding with ffmpeg, particularly with the complex (daunting) h264 encoder (the preferred format by most video hosting sites). After some Googling and experimentation, I created the video you saw with this command:

ffmpeg -f image2 -i ffr-%05d.jpg -vcodec libx264 -vpre hq -crf 24 -g 320 -threads 0 video-d.mp4

The file created this way was only 47MB, compared to 152MB using the original technique. It downloads much faster, but makes the computer CPU work a little harder to play the video. I hope I've achieved a good tradeoff. Note that you can always toggle between HD, high definition, and SD, standard definition, by clicking on the blue HD symbol on the playback bar. The default is HD playback.

Creating ZoomWalks

This post will tour the whys and hows of the ZoomWalk video clips that were published in this post. It was a learning experience for me from both a photographic and a technological (software) point of view, one which I'm certain hasn't yet reached its end.

Photography Tips

The heart of the ZoomWalk is the individual photographs that compose it. Hundreds of photographs. Photographs which should make sense when lined up one after the other in a video. Here, in no particular order, are some of the points I discovered.

• Walking into the sun is a bad idea. The photographs aren't as good, and it's extremely difficult to see the camera display, which means the shots won't line up as well as they should. Squinting can give you a headache.
• Take advantage of any 'guide lines' or autofocus marks on the camera display to line up your shots: to keep the horizon consistent (avoiding rotation), to avoid sliding off to the right or the left or top or bottom, and so on.
• It is helpful to find a distant object to use as a benchmark, and try to place it in the same spot in the series of photos. Closer objects are unsuitable because they are supposed to be moving quickly to the side of the frame anyway.
• Plan the transition from one benchmark object to the next, so your frame doesn't suddenly jump up or down or to one side or another.
• Sometimes (in the woods) there won't be a suitable distant benchmark that's visible. Do your best and keep going.
• You don't need 10 megapixel images to create a 1280x720 video. Four would be plenty, and I got by with 3.2. Smaller pictures will be easier for your computer to cope with.
• Use a variable number of steps based on what's going on. I started with 4 paces between each shot, and my paces are about 2¼ to 2½ feet. Use fewer when visually interesting things are happening (passing over bridges, taking stairs). A few more steps per photo are OK when passing through straight,  less interesting parts.
• Take more shots when going around curves and corners. The sharper the turn, the more shots are needed to maintain continuity. Here you need to keep the horizon consistent, but the view in subsequent pictures will naturally shift from frame to frame until it settles into 'straight forward' for the new direction.
• Be patient. Yes, you're taking hundreds of photos, but you'll save yourself time and grief at the computer if the photos are better aligned to start.

Other thoughts:

• When going around a curve or corner, mimic what the human eye does and look through the curve before you arrive there. We don't lock our eyes on the last section of straight ahead trail, but tend to look ahead along the upcoming curve. (Thanks Joan!)
• When going down a hill or stairs, the far horizon seems to rise to the human eye. Keeping it at a constant height in the image (as a benchmark) may leave that segment appearing unnatural ... this needs more investigation!
• When passing by or through interesting sights -- a bridge or interesting building -- stop and pan to the sight, so that the viewer appears to pause and regard the view.
• Take a spare, fully charged battery along. The camera will never go to sleep with the steady taking of pictures, so the battery may drain faster than you anticipate.
• It would be helpful if the camera had a bubble level or inclinometer, to avoid small rotations of the frame. The iPhone has at least one such app.

Initial Renaming

Now I have a series of still photographs. What do I do with them? It starts out similarly to my discussion on time-lapse videos: for purposes of further processing, rename the images in sequential order starting with '1', in a subdirectory to avoid messing with the originals. I use a simple shell script. (All these scripts are written for the Bourne shell in a Linux/Unix environment, but they should be adaptable.)

let ix=0
/bin/rm -f fr*.jpg *.tiff

# reduce input pictures to a more manageable size that is
# still sufficent to hold a 1280x720 image after trimming
# due to rotations and translations.
#
for i in ../P*.JPG
do
    let ix=$ix+1
    fname=`printf "fr%04d" $ix`
    convert -scale 1602x1066 -crop 1600x1065+1+1 $i $fname.jpg
    echo $fname.jpg
done

This script first scales to 1602x1066 and then crops to 1600x1065 because I shot the originals in 3:2 format, and scaling alone had rounding errors (only 1598 wide). We'll see later why you need extra pixels which will be discarded later to produce a final 1280x720 video. The convert command used is part of the imagemagick package.

Alignment

Unless your photography was extremely skillful, using the images directly will result in a shaky video because of misalignments between frames. It's not your fault, because you can only align as well as your camera display will allow. I prefer to tweak the alignments wherever reasonable; that is, where the benefit is greater than the work. This is where 95% of the effort lies, and so I was curious if there was any way to automate it. The answer is, sort of ... but you must always click through the video (view it frame by frame) to look for errors.

The tool I discovered (initially through this link) was align_image_stack. This command line tool is part of the larger hugin project. It is designed to align multiple images of the same subject taken at different exposures, a preprocessing step for High Dynamic Resolution (HDR) imaging. As such, it aligns images that are expected to be almost the same. My ZoomWalk images are different from each other, some very much so (corners), some not so much. Also, objects close to the edge of the frame move quickly. How well would align_image_stack work for my ZoomWalk pictures?

The first problem was to determine how good a job align_image_stack thought it did. I had to intercept the status printouts from the command and identify the final RMS (root mean square) error that align_image_stack reported. If it was too large, it was likely though not certain that align_image_stack had overcorrected.

My first script compared successive images; that is, it would:

• compare #1 with #2
• based on the rms, either accept the modified #2 or stick with the original
• compare #2 with #3
• etc

After looking at the results, I felt that this wasn't working. For one thing, because the replaced frames were changed, comparing a changed frame N with an unchanged frame N+1 allowed small corrections to accumulate from frame to frame. For example, given this method, here are frame #1 and frame #10:

To avoid this problem, I modified the script; if image N was successfully modified, the original image for N+1 was automatically chosen next -- without even running align_image_stack. This prevents accumulation of errors, but means that no more than 50% of the images can be automatically aligned!

Another problem was that, because align_image_stack assumes that the pictures are intended to be identical, it would struggle against the view turning around a curve or corner, even to the point of skewing the perspective:

Because align_image_stack works by selecting control points to compare between frames, it is possible, if the control points are poorly chosen (for whatever reason), for things to go very wrong:

To catch these mistakes, I added a test for the percentage change in the median brightness of the image. If the change was too large, because of large black background areas being added, the script would stick with the original.

Align_image_stack has an option for compensating for small variances in magnification of the images (-m option). I thought this sounded promising for the ZoomWalk photos, because the central part of the image would be undergoing, in effect, magnification. However, it didn't work out as I had hoped:

 

Sometimes an alignment would be rejected by these tests that, on visual inspection, appeared reasonable. Therefore, the script saves rejected images for later inspection, and in two groups, one for RMS rejections and one for percentage brightness rejections. As I said earlier, you must always inspect the results of the alignment process.

There are two possible enhancements to the alignment script that I still need to investigate:

• instead of comparing image N with image N+1 and accepting or rejecting the realignment of N+1, and skipping any possible alignment of N+2 if the changes for N+1 are accepted, compare image N with N+1 and accept or reject the changes for N. Then image N+1 is untouched and we can compare N+1 with N+2, etc. This way all, rather than half, of the images could be potentially realigned by align_image_stack. (Technical note: the script would have to list N+1 as the first image, and N as the second, because align_image_stack leaves untouched the first image in the set it is working on.)
• to reduce the impact of a single poorly taken photo, experiment with comparing three images at a time rather than two. Compare N, N+1, and N+2, accepting or rejecting the changes to N. Then move on to compare N+1, N+2, and N+3.

If I could request one enhancement to align_image_stack -- and I realize it wasn't intended for the uses to which I put it -- it would be to have an option to scale back its changes. For example, a way to say that small changes should be adopted in their entirety, moderate changes should be applied but only by half, and large changes shouldn't be applied at all.

Here is the current state of the alignment script:

#!/bin/bash
#

# a subroutine to perform align_image_stack and return a whole number
# approximation of the rms value in the align_image_stack output.
#
pair_rms()
{
    /bin/rm -f rms*.tif foo.jpg

    frms=`align_image_stack -s 2 $1 $2 -a rms 2>&1 | grep after | \
        tail -1 | awk ' { print $4 } ' - `
    #
    # if no return value, or no tif file, align_image_stack couldn't do
    # anything with this pair.
    #
    if [ -z "$frms" -o ! -f rms0001.tif ]
    then
        echo "99"
        return
    fi

    rmsi=`echo $frms | cut -f1 -d'.' `
    if [ $rmsi -gt 0 ]
    then
        rmsd=`echo $frms | cut -f2 -d'.' | cut -c1`
        #
        # set rounding-up threshold to suit
        #
        if [ $rmsd -ge 5 ]
        then
            let rmsi=$rmsi+1
        fi
    fi
    echo $rmsi
}

# a subroutine to calculate delta percentages using dc to return a
# floating point number.
#
pct_brt_delta()
{
pct=`dc<8k
$1
$2 -
100 *
$1 / p
!
`
echo "$pct"
}

/bin/rm -f algn*.jpg reject*.jpg

# align the frames
#
let i=1
fname=`printf "fr%04d" $i`
aname=`printf "algn%04d" $i`
cp $fname.jpg $aname.jpg

let i=$i+1
aname=`printf "algn%04d" $i`
xname=`printf "fr%04d" $i`

# create an all-black image for background, at the same size as
# what we're getting in our input frames.
#
convert -size 1600x1065 xc:black blackback.tif
repeat=false

while [ -f $xname.jpg ]
do
    #
    # do NOT even bother accepting two image_align_stacks in a row!
    #
    if [ $repeat == false ]
    then
        echo compare $fname.jpg $xname.jpg
        br=`identify -format "%[mean]" $xname.jpg`
        rms=`pair_rms $fname.jpg $xname.jpg`
        if [ $rms -gt 1 ]
        then
            echo "  rms is $rms"
            echo "  cp $xname.jpg $aname.jpg"
            cp $xname.jpg $aname.jpg
            #
            # save rejected realignment, if it exists, for manual inspection.
            #
            if [ -f rms0001.tif ]
            then
                rname=`printf "reject%04d-rms" $i`
                convert rms0001.tif $rname.jpg
            fi
            # composite -geometry +0+0 rms0001.tif blackback.tif foo.jpg
            # aft=`identify -format "%[mean]" foo.jpg`
            # pct=`pct_brt_delta $br $aft`
            # echo "  brightness delta would have been $pct%"
        else
            echo "  rms is $rms, check brightness delta"
            #
            # put image over a black background to prevent passing transparent
            # pixels to ffmpeg later on.
            #
            composite -geometry +0+0 rms0001.tif blackback.tif foo.jpg
            aft=`identify -format "%[mean]" foo.jpg`
            pct=`pct_brt_delta $br $aft | cut -f1 -d'.'`
            if [ -z "$pct" ]
            then
                pct=0
            fi
            if [ $pct -ge 7 ]
            then
                echo "  brightness delta would have been $pct%"
                echo "  cp $xname.jpg $aname.jpg"
                cp $xname.jpg $aname.jpg
                #
                # save rejected realignment for manual inspection.
                #
                rname=`printf "reject%04d-brt" $i`
                cp foo.jpg $rname.jpg
            else
                echo "  brightness delta is $pct%"
                echo "  cp foo.jpg $aname.jpg"
                cp foo.jpg $aname.jpg
                repeat=true
            fi
        fi
    else
        echo "cp $xname.jpg $aname.jpg (no repeats)"
        cp $xname.jpg $aname.jpg
        repeat=false
    fi

    let i=$i+1
    fname="$aname"
    xname=`printf "fr%04d" $i`
    aname=`printf "algn%04d" $i`
done

To manually adjust/align a frame, I use the GIMP. Open the first frame normally, with File -> Open. If the Layers window isn't open, start it with Windows -> Layers. Then open the second frame as a separate layer with File -> Open as Layers. Set the Opacity of the new layer to roughly 50%. This allows you to see through to the other layer, and you can move either layer to create the alignment you want. Before saving the modified layer, delete the other layer (you don't want it to be part of the frame!), set the Opacity back to 100%, and invoke Layer -> Layer to Image Size. This last step is necessary to fit the modified layer within the original image size, cropping as needed.

Creating a set of aligned frames that you are happy with is the most time-consuming step in creating a ZoomWalk video. It will take too much time to manually adjust every frame, so run the video several times to see where it needs help. You can also click through the aligned frames one-by-one to look for any surprises. You may end up replacing aligned frames with the original version, manually modifying an original or aligned frame, or accepting one of the rejected frame alignments.

Frame Interpolation/Morphing

Taking a photo every 10 feet, and playing back the video at 25 frames per second, would mean that the point of view would hurtle forward at 250 feet/second, or 170 miles/hour. However, to take a photo every one or two steps would double or treble the number of photos taken, the time required to take the photos, the workload for the computer, and the visual inspection of the frames. I used the morph command from the imagemagick package to ameliorate this problem.

Morph does not understand objects, or the concept of objects changing their position between frames. It just blends the color of each pixel; if you invoke "morph -1" you will get one image that is 50% of each real frame. If you specify "morph -2," which is what I settled on, you get two interpolated or in-between frames, the first of which is 67% frame #1 and 33% frame #2, and the second will be 33% frame #1 and 67% frame #2. To generate the fade in and fade out of the title sequences, I used "morph -10" to gradually transition from a black background to the title frame and back again. Morph is in essence a shorthand for this particular use of the blend operator.

This image shows the results of a "morph -2". The top left image is the first real frame, and the lower right image is the second real frame.

I decided that having three blended images was too much; the appearance became more of fading in and out rather than moving forward. The videos I produced all used two interpolated frames.

Here is the script fragment for generating the title sequence:

#!/bin/bash
#

echo "create title frames"

/bin/rm -f smalgn*.jpg

convert -size 1280x720 xc:black blackf.tif

convert -size 1280x720 xc:black -font Cooper-Blk-BT-Black -pointsize 40 \
    -gravity center -draw \
        "fill white text 0,-28 \"ZoomWalk #3\" \
        fill white text 0,28 \"Chautauqua Park to Walton Lake\" " \
        title.png

# create intermediate frames to fade from black to title and back.
# Total number of frames generated is 23.
#
convert blackf.tif title.png blackf.tif -morph 10 smalgn%05d.jpg
let seq=23

After the title sequence has been generated, it is time to enhance the frames and trim them to their final size, here 1280x720.  The trimming also removes the black edges created by realignment, and is accomplished by extracting the 1280x720 frame from the center of the larger image. The file names start with a sequence number (seq) established from the final title sequence frame.

cnt=`ls -1 algn*.jpg | wc -l`
echo "buff and resize $cnt frames"

# to compare same-size and same-color frames, we defer contrasting,
# sharpening, and trimming the edges until this point, when we are
# about to generate the intermediate frames. We are trimming from
# 1600x1065 to 1280x720.
#
for i in algn*.jpg
do
    of=`printf "smalgn%05d" $seq`
    convert -crop 1280x720+160+172 -contrast-stretch 0.30x0.35% -unsharp 4x1.5+0.36+0.5 $i $of.jpg
    let seq=$seq+1
done

Using morph on all the images at once uses a lot of the computer memory. Even on my desktop computer, juno, with 4 GB of RAM, processing 1280x720 images would fill the memory rapidly, causing the computer to use the disk (swap/pagefile) to hold information and slowing the processing down immensely. The script works around this limitation by generating the interpolated frames in batches of 100 real frames, and resequencing the numbers:

# we must generate the morphed/interpolated frames in batches, else
# most desktop computers will run out of memory!

/bin/rm -f fbatch*.jpg ffr*.jpg

seq=0
bigseq=0

smbase=`printf "smalgn0%02d" $seq`
first=`printf "smalgn0%02d00.jpg" $seq`

while [ -f "$first" ]
do
    let seq=$seq+1
    echo "create interpolation batch ${seq} ($smbase))"
    #
    # handle gap between last of this and first of next batch by
    # including "next"....
    #
    next=`printf "smalgn0%02d00.jpg" $seq`

    if [ -f "$next" ]
    then
        convert $smbase*.jpg $next -morph 2 fbatch%03d.jpg
        #
        # handle dup of last of this batch and first of next batch
        # by removing last of this batch.
        #
        ls -l fbatch300.jpg
        /bin/rm fbatch300.jpg
    else
        convert $smbase*.jpg -morph 2 fbatch%03d.jpg
    fi

    for i in fbatch*.jpg
    do
        oname=`printf "ffr-%05d" $bigseq`
        mv $i $oname.jpg
        let bigseq=$bigseq+1
    done

    smbase=`printf "smalgn0%02d" $seq`
    first=`printf "smalgn0%02d00.jpg" $seq`
done

Another area for future experimentation is with the morphing values. By using the blend operator directly, you can play with percentages other than those used by morph. For instance, you could still have two interpolated frames, but instead of 67% and 33%, they could be further apart (75% and 25%) or asymmetrical (80% and 50%). Video from stills is a very large sandbox in which to play!

Generating the Video

Now, finally, the frames can be assembled into a video. This script generates two slightly different versions, one at a high quality/less compression setting (-qmax 3) and another at a slightly less high (but still high) quality setting (-qmax 4).

/bin/rm -f video-HD-3.mov
/bin/rm -f video-HD-4.mov

# now we can finally assemble the video.
#
ffmpeg -f image2 -i ffr-%05d.jpg -qmax 3 video-HD-3.mov
ffmpeg -f image2 -i ffr-%05d.jpg -qmax 4 video-HD-4.mov

These scripts have used a consistent naming convention for each step of the process, so that one step does not interfere with prior steps. For example, you could experiment with the interpolation/morphing step many times while leaving the alignment work untouched. The convention is,

• files starting with 'fr' are the resized copies of the original photographs.
• files starting with 'algn' are the aligned versions of the frames, which could be untouched copies of the 'fr' file, automatically aligned versions, or manually aligned versions.
• files starting with 'smalgn' are the title sequence followed by the enhanced and trimmed frames.
• files starting with 'ffr' are the final frames, real plus interpolated.

Video Services

Any video service provider translates the videos that are uploaded into a particular encoding scheme or schemes, and allows a certain bandwidth for replaying them. As I documented in the ZoomWalk post, it was necessary to alter those videos (embedding a smaller video in a larger black image) to obtain decent reproduction from YouTube; the gambit caused YouTube to use the HD (High Definition) settings for a video, but the videos had a modest (for HD) bandwidth requirement because of the static black background.

The ZoomWalk videos are handicapped, in a sense, because they contain more change per frame than a typical video. If I look at some of the earlier videos shot as videos by my camera, and adjust for image size, using the same encoding scheme (mp4/mov), the regular videos require roughly 3 MB of uploaded file for each second of playback time, while the ZoomWalks consume about 6¼ MB/second. The Griggs Dam video was taken by my camera at 1280x720 in AVCHD Lite format, which is considered a lossy compressing format, but so is mp4, for which I used high-quality settings to compensate. The AVCHD Lite video used just under 2 MB/second.

Given my dissatisfaction with the embedded-in-black trick for YouTube -- it creates an odd appearance in the blog -- I decided to experiment with Vimeo, an alternative video hosting service. The experience was similar to using YouTube until I upgraded to a paid "Plus" membership, qualifying me for better playback. Here are the three ZoomWalk photos, hosted by Vimeo, in widescreen (16:9 aspect ratio) and in a size that works well embedded in a blog post, without the distracting black background trick.








The companies offering video services are in competition, and will occasionally leapfrog each other in technology or service. YouTube recently bought a video enhancement company (Green Parrot), so in six months or a year my choice might change again.

These three compositions were intriguing and very instructive for me, and I hope to create better ZoomWalks, both technically and artistically. I have enjoyed sharing them with you (although I can do without blogger trying to eat them).

Fairfield 3, Updates and ZoomWalks

I recently returned from another visit to Fairfield, Iowa. This trip was similar to the two I've described earlier, here and here, but there are updates to pass along, and a new project, which I call ZoomWalks, that I started while there.

The gap between this trip and the previous one, from November through March, covers the winter months, so the changes are far fewer than the busier and longer April to November span. Even so, there are several items to relate.

The Mayor's Report

The March 24-30 issue of the Fairfield Weekly Reader included a report from the Mayor, Ed Malloy. A brief list of the highlights:

• 2011 should bring the completion of the Loop Trail system, with work on the northwest section of the trail.
• Inspired by Karla Christensen's mural in the alleyway next to Revelations (I took a picture of it, scroll about halfway down), a new public art project entitled Maze of Murals will be introduced.
• Renovations of City Hall in 2011, including handicap access.
• Hy-Vee constructed a Gold LEED certified building in Fairfield, the second in the company's history.
• The Federal Railroad Administration is reviewing the city's application for a Quiet Zone, wherein with appoved crossing upgrades the trains will no longer blow their horns as they pass through town.
• Fairfield is working with Partners for Livable Communities to host a national conference in the fall.

Ben's Notes

I spent most of my time on or near the MUM (Maharishi University of Management) campus, so the progress on the new Sustainable Living Center immediately caught my eye. First, exterior shots.

Here is the entryway, as of late March.

You can see a few loops of the radiant floor heating in the above picture; here's a view looking back at the entrance.

The southern face of the building is a solar hall providing space for various projects and to absorb heat.

Here is one of the classrooms under construction. Note the bricks used in an interior wall; they are made locally.

The interior walls are substantial, with the bricks coated with a brown substance to smooth them out and a final whitewash layer.

The Loop Trail bridge over Highway 1, at the north edge of the campus, now has lights along its arch. This was a brilliant stroke and transforms the bridge into a welcoming gateway. Necessarily, I took these pictures under diminished lighting conditions (no tripod).

After two previous posts about Fairfield, I will abstain from further early-light pictures of the Argiro Student Center and other campus buildings. My only early morning photo of this trip is of some visitors to the campus.

This early evening cloud formation was an impressive post-meditation sight. Dinner could wait a few minutes while I studied it and its slow transformations.

 

The popular grocery Everybody's is in the process of transforming its (auspicious north) entrance.  The current entrance is only two doors wide, one in and one out, and the 'airlock' is short, only a few feet, so that much of the time it is open at both ends. What used to be an open-air patio with a few tables is being transformed into a much wider and deeper airlock. In this photo, the current entrance is under the green awning, and the new entrance stretches along the entire area under the tan roof.

The interior is being reorganized, and it appears that a previously employee-only area will become an additional shopping aisle.

ZoomWalk Project

Every so often an idea bubbles up that I feel compelled to try out, and this spring it is a mobile time-lapse project I have dubbed 'ZoomWalk.' The outline is this: to take a picturesque or otherwise interesting walk, taking a photograph every few steps, and afterwards use the computer (and open-source tools) to combine them into a video. It's a 'zoom' walk because the final video gives the impression that you're walking at great speed, even up to 70-80 mph. That other people may pass you at apparent speeds even faster than yours reveals the time-lapse nature of the effort. There are tools to work with traditional, static time-lapse images, but working on a mobile time-lapse project presents challenges. The field of view is different from frame to frame, and the camera is hand-held. I'll write another post to discuss the details, both photographic and computational.

Note: the original videos are of higher quality than YouTube will reproduce. I apologize for the blotchy sections; if and when YouTube upgrades its handling of these clips will improve!

My initial experiment was a short walk on the Jefferson County Loop Trail, passing through the bridge over Highway 1. The lighting suffers every time the path turns south and I face the sun, but it was a lesson and a beginning.

My second project was an afternoon walk to the Men's Dome, with better lighting. My final opus from the visit to Fairfield is a walk from Chautauqua Park to just beyond Walton Lake, again on the Jefferson County Loop Trail.

Hmm. Watching that clip several times makes me a little dizzy, with its high-speed swooping curves, especially in the second half.  Next time I'll slow it down a little (one less step between photos)!

My Photo Tweaks; assembling and disassembling video

The previous post discussed combining more than one image to generate a stronger single image (panoramic and High Dynamic Range photos). With the time-lapse movies and screen-capture clips of recent posts, I've been involved with a different alchemy, both assembling videos from a series of still images, and extracting still images from a video in order to tweak them.

Warning: this post lets my geek side spill out. I used free, open-source tools that allowed me to twiddle and tweak as long as I had the energy to research. I apologize to the majority of readers, for whom this has little interest. A non-technical post on an entirely different topic will be next up!

Time-Lapse Files

As I mentioned in the time-lapse post, I copy the JPG files of the photos from the PlantCam to my computer. There, I first make a quick video with the ffmpeg command-line tool (more free software) to see what tweaks are needed. The JPG files are numbered sequentially, WSPC0001.JPG, WSPC0002.JPG, and so forth. (WSPC stands for WingScapes PlantCam). To make a video from the raw images, I just type:

ffmpeg -f image2 -i WSPC%04d.JPG -qmax 3 video.mov

This tells ffmpeg to assemble a video from the files with a name of WSPC followed by four digits, at a high quality/low compression level (-qmax 3). If I'm happy with the result, I'm done, but there are often frames that I would rather not include: my face peering into the camera to see if it's working, light flares from the sun when clouds part, frames that are too dark as the sun goes down, and so forth. Small numbers of these can be dealt with by deleting the individual frames, but a complication is that ffmpeg expects consecutive frame numbers, so if any frames are deleted, the video stops there! Renumbering is a must, and that is something you do not want to attempt by hand.

To avoid accidentally deleting the original images, I create a subdirectory, and run my scripts from there. A basic renumbering script looks like this (yes, you coders out there, there are many paths climbing this mountain):

let i=1
for file in ../WSPC*.JPG
do
    nname=`printf "fr-%04d.JPG" $i`
    cp $file $nname
    let i=i+1
done

This script creates a copy of the WSPC images in the current directory, with the same four-digit naming convention, but the numbers will be sequential even if I've zapped some of the original files.

What if you wish to do various operations, such as cropping, contrast adjustment, and so forth? You aren't going to tweak each frame with the GIMP ... there will be hundreds or thousands. Another free, open-source toolkit, ImageMagick, comes to the rescue with its command-line suite. We'll jump in with a script I wrote for the "snow day" time-lapse clip. It may appear intimidating but I'll go through it step-by-step. I don't claim that it's optimal, but it worked.

/bin/rm -f video.mpg fr*.JPG

for i in ../W*.JPG
do
    nname=`printf "fr-%04d.JPG" $ix`
    let ix=ix+1
    stamp=`identify -format "%[EXIF:DateTime] %w" $i`
    fdate=`echo $stamp | cut -f1 -d' ' | sed -e "s/:/\//g"`
    ftime=`echo $stamp | cut -f2 -d' '`
    fwide=`echo $stamp | cut -f3 -d' '`
    let fontsize=$fwide/25
    echo $i $fdate $ftime $fwide
    convert -contrast-stretch 0.30x0.35% -unsharp 3x1.5+0.36+.05 $i $nname

    # create a timestamp watermark.
    #
    convert -size ${fwide}x100 xc:transparent -font FreeMono-Medium -pointsize $fontsize \
            -gravity center -draw \
                "fill black text -1,-1 \"$fdate $ftime\" \
                 fill black text 1, 1 \"$fdate $ftime\" \
                 fill white text 0,0 \"$fdate $ftime\" " \
            stamp_fgnd.png

    # shrink size of the image to the contents (-trim) and change the virtual canvas to that
    # size (+repage), then add the watermark to the image.
    #
    mogrify -trim +repage stamp_fgnd.png
    composite -watermark 65.0 -gravity southwest -geometry +10+5 stamp_fgnd.png \
        ${nname} foo.jpg
    mv foo.jpg ${nname}
done

ffmpeg -f image2 -i fr-%04d.JPG -qmax 4 video.mpg

The first line of the script just removes the files from any previous run. Then, as before, for each WSPC JPG file, we create a proper sequential name. Next we use the identify tool from ImageMagick to extract information from the EXIF data kept in every modern digital image. In this case, we get the time and date that the image was taken, because I want to create a timestamp watermark, and the width of the image, to calculate how large a font to use for that watermark.

Next, we not only copy the original file into the subdirectory, but we modify it with the convert command. The '-contrast-stretch 0.30x0.35%' argument tells convert to stretch (increase) the contrast until no more than 0.30% of the pixels are completely dark, and no more than 0.35% of the pixels are completely bright. Then the '-unsharp 3x1.5+0.36+.05' argument applies an unsharp mask (to sharpen the image) with various values (mask radius, standard deviation, amount, and threshold). Contrast enhancement and sharpening were tasks I did by hand using the GIMP for single images, but here, they are automatically (and uniformly) applied to each frame. I spent a while experimenting with these values, re-running the script each time, until I reached a satisfactory result.

Next, convert is used to create a watermark file (stamp_fgnd.png) using the date and time extracted earlier. The text is drawn twice in black, once offset by one pixel to the left and above and then once to the right and below, before being drawn in white with zero offset. This makes a nice outlined text. A simple timestamp might look like this:

Not knowing how large the final text would be, the initial watermark canvas was made large (as wide as the image and 100 pixels tall) and the text placed in the center. This leaves a lot of empty space around the text, as you see above. Because I want to place this text in the lower left-hand corner of another image, I want to discard the extra space. The script uses the mogrify spell (basically a synonym for convert that writes the result directly into the original file rather than a second file) to slice away the extra (-trim):

What the above image doesn't show you is that the text and the canvas have both been trimmed, but the offset of the text hasn't been corrected. Many viewing programs automatically correct this for you, but the GIMP notices:

The two must be realigned in order for the watermark to show up as expected, so the '+repage' is also done. (Mogrify, like all ImageMagick commands, performs its magic from left to right, so in this case the trim is done before the repage.)

Finally, composite is summoned to combine the timestamp image and the original image. The timestamp is added as a watermark, with the original image contributing 65%. The addition takes place at the southwest corner, with an offset of 10 pixels in X and 5 pixels in Y. The result is written into a temporary file and then copied into the original filename. The result is pleasing, as this frame from the "snowy day" video shows.

One side note: for minimal processing of videos from my camera, such as cropping edges or clipping leading or trailing seconds, I use an ffmpeg incantation directly, rather than the painstaking work described in this post. One example: 

ffmpeg -i 00134.mov -sameq -croptop 160  -cropright 180 -acodec copy piper.mov

Screen Capture

In preparing the earlier posts I also used screen capture to reveal how a particular program works -- using xvidcap I grabbed videos of the GIMP and fotoxx in action. Being screen captures, these videos did not need any contrast enhancement or image sharpening; however, it is difficult to perform an example task without bobbling some aspect, even with practice, and if the task requires the computer to crunch away for 30 or 60 seconds, the video becomes boring. Very boring.

I needed to break the video into separate images, allowing me to delete my mistakes and distractions, and to compress the tedious parts. Fortunately, ffmpeg breaks apart a video quite handily:

ffmpeg -i test-0000.mpeg -qmax 2 -f image2 test-%04d.jpeg

This is exactly the reverse of assembling the video: the separate frames are created as individual, sequentially numbered JPG files. Then I can manually delete a small number of frames, or write a script to skip or speed up entire sections.

I also needed to make an extra copy of each frame to slow down the playback. With xvidcap and the demonstrated program (GIMP, fotoxx) competing for the computer, it was best to capture the desktop at no more than 12 or 13 frames/second. However, ffmpeg wants to produce video that plays at 25 frames/second. This mismatch spawns a hyperactive mouse that is difficult to follow. After losing several arguments with ffmpeg, I decided the easiest course was to double frames.

Here is another example script. It extracts the four-digit number from the name of the original frame, allowing it to process only every other frame between 1290 to 2064, and creating two copies of all the other frames. Thus, the waiting-for-the-computer section zips along four times faster than the moving-the-mouse parts. Finally, ffmpeg is used to assemble a video from the new frames. (I discovered that for screen capture, versus photographs, a different encoding standard, h264, produced smaller files.)

let ix=1
let skip=0

# create two copies of each frame, to make the pace of the playback reasonable
#
/bin/rm -f video.mpg fr*.jpg

for i in ../test*.jpeg
do
    if [ $skip -eq 1 ]
    then
        skip=0
        continue
    fi

    orignum=`echo $i | sed -e "s/..\/test-//" | sed -e "s/.jpeg//"`
    nname=`printf "fr-%04d.jpg" $ix`
    cp $i ${nname}
    let ix=ix+1

    # do not double the boring stretch, in fact, halve it!
    #
    if [ $orignum -gt 1290 ] && [ $orignum -lt 2064 ]
    then
        skip=1
        continue
    fi

    nname=`printf "fr-%04d.jpg" $ix`
    cp $i ${nname}
    let ix=ix+1
done

# reassemble the video using the libx264 video codec
#
ffmpeg -f image2 -i fr-%04d.jpg \
    -vcodec libx264 -vpre lossless_ultrafast -threads 0 video.mkv

The moon is setting, and it's time to wrap up this post and close our book of spells (or web sites with manual pages for command-line tools). Thank you for visiting the lab with me.