My Photo Tweaks; assembling and disassembling video

The previous post discussed combining more than one image to generate a stronger single image (panoramic and High Dynamic Range photos). With the time-lapse movies and screen-capture clips of recent posts, I've been involved with a different alchemy, both assembling videos from a series of still images, and extracting still images from a video in order to tweak them.

Warning: this post lets my geek side spill out. I used free, open-source tools that allowed me to twiddle and tweak as long as I had the energy to research. I apologize to the majority of readers, for whom this has little interest. A non-technical post on an entirely different topic will be next up!

Time-Lapse Files

As I mentioned in the time-lapse post, I copy the JPG files of the photos from the PlantCam to my computer. There, I first make a quick video with the ffmpeg command-line tool (more free software) to see what tweaks are needed. The JPG files are numbered sequentially, WSPC0001.JPG, WSPC0002.JPG, and so forth. (WSPC stands for WingScapes PlantCam). To make a video from the raw images, I just type:

ffmpeg -f image2 -i WSPC%04d.JPG -qmax 3 video.mov

This tells ffmpeg to assemble a video from the files with a name of WSPC followed by four digits, at a high quality/low compression level (-qmax 3). If I'm happy with the result, I'm done, but there are often frames that I would rather not include: my face peering into the camera to see if it's working, light flares from the sun when clouds part, frames that are too dark as the sun goes down, and so forth. Small numbers of these can be dealt with by deleting the individual frames, but a complication is that ffmpeg expects consecutive frame numbers, so if any frames are deleted, the video stops there! Renumbering is a must, and that is something you do not want to attempt by hand.

To avoid accidentally deleting the original images, I create a subdirectory, and run my scripts from there. A basic renumbering script looks like this (yes, you coders out there, there are many paths climbing this mountain):

let i=1
for file in ../WSPC*.JPG
do
    nname=`printf "fr-%04d.JPG" $i`
    cp $file $nname
    let i=i+1
done

This script creates a copy of the WSPC images in the current directory, with the same four-digit naming convention, but the numbers will be sequential even if I've zapped some of the original files.

What if you wish to do various operations, such as cropping, contrast adjustment, and so forth? You aren't going to tweak each frame with the GIMP ... there will be hundreds or thousands. Another free, open-source toolkit, ImageMagick, comes to the rescue with its command-line suite. We'll jump in with a script I wrote for the "snow day" time-lapse clip. It may appear intimidating but I'll go through it step-by-step. I don't claim that it's optimal, but it worked.

/bin/rm -f video.mpg fr*.JPG

for i in ../W*.JPG
do
    nname=`printf "fr-%04d.JPG" $ix`
    let ix=ix+1
    stamp=`identify -format "%[EXIF:DateTime] %w" $i`
    fdate=`echo $stamp | cut -f1 -d' ' | sed -e "s/:/\//g"`
    ftime=`echo $stamp | cut -f2 -d' '`
    fwide=`echo $stamp | cut -f3 -d' '`
    let fontsize=$fwide/25
    echo $i $fdate $ftime $fwide
    convert -contrast-stretch 0.30x0.35% -unsharp 3x1.5+0.36+.05 $i $nname

    # create a timestamp watermark.
    #
    convert -size ${fwide}x100 xc:transparent -font FreeMono-Medium -pointsize $fontsize \
            -gravity center -draw \
                "fill black text -1,-1 \"$fdate $ftime\" \
                 fill black text 1, 1 \"$fdate $ftime\" \
                 fill white text 0,0 \"$fdate $ftime\" " \
            stamp_fgnd.png

    # shrink size of the image to the contents (-trim) and change the virtual canvas to that
    # size (+repage), then add the watermark to the image.
    #
    mogrify -trim +repage stamp_fgnd.png
    composite -watermark 65.0 -gravity southwest -geometry +10+5 stamp_fgnd.png \
        ${nname} foo.jpg
    mv foo.jpg ${nname}
done

ffmpeg -f image2 -i fr-%04d.JPG -qmax 4 video.mpg

The first line of the script just removes the files from any previous run. Then, as before, for each WSPC JPG file, we create a proper sequential name. Next we use the identify tool from ImageMagick to extract information from the EXIF data kept in every modern digital image. In this case, we get the time and date that the image was taken, because I want to create a timestamp watermark, and the width of the image, to calculate how large a font to use for that watermark.

Next, we not only copy the original file into the subdirectory, but we modify it with the convert command. The '-contrast-stretch 0.30x0.35%' argument tells convert to stretch (increase) the contrast until no more than 0.30% of the pixels are completely dark, and no more than 0.35% of the pixels are completely bright. Then the '-unsharp 3x1.5+0.36+.05' argument applies an unsharp mask (to sharpen the image) with various values (mask radius, standard deviation, amount, and threshold). Contrast enhancement and sharpening were tasks I did by hand using the GIMP for single images, but here, they are automatically (and uniformly) applied to each frame. I spent a while experimenting with these values, re-running the script each time, until I reached a satisfactory result.

Next, convert is used to create a watermark file (stamp_fgnd.png) using the date and time extracted earlier. The text is drawn twice in black, once offset by one pixel to the left and above and then once to the right and below, before being drawn in white with zero offset. This makes a nice outlined text. A simple timestamp might look like this:

Not knowing how large the final text would be, the initial watermark canvas was made large (as wide as the image and 100 pixels tall) and the text placed in the center. This leaves a lot of empty space around the text, as you see above. Because I want to place this text in the lower left-hand corner of another image, I want to discard the extra space. The script uses the mogrify spell (basically a synonym for convert that writes the result directly into the original file rather than a second file) to slice away the extra (-trim):

What the above image doesn't show you is that the text and the canvas have both been trimmed, but the offset of the text hasn't been corrected. Many viewing programs automatically correct this for you, but the GIMP notices:

The two must be realigned in order for the watermark to show up as expected, so the '+repage' is also done. (Mogrify, like all ImageMagick commands, performs its magic from left to right, so in this case the trim is done before the repage.)

Finally, composite is summoned to combine the timestamp image and the original image. The timestamp is added as a watermark, with the original image contributing 65%. The addition takes place at the southwest corner, with an offset of 10 pixels in X and 5 pixels in Y. The result is written into a temporary file and then copied into the original filename. The result is pleasing, as this frame from the "snowy day" video shows.

One side note: for minimal processing of videos from my camera, such as cropping edges or clipping leading or trailing seconds, I use an ffmpeg incantation directly, rather than the painstaking work described in this post. One example: 

ffmpeg -i 00134.mov -sameq -croptop 160  -cropright 180 -acodec copy piper.mov

Screen Capture

In preparing the earlier posts I also used screen capture to reveal how a particular program works -- using xvidcap I grabbed videos of the GIMP and fotoxx in action. Being screen captures, these videos did not need any contrast enhancement or image sharpening; however, it is difficult to perform an example task without bobbling some aspect, even with practice, and if the task requires the computer to crunch away for 30 or 60 seconds, the video becomes boring. Very boring.

I needed to break the video into separate images, allowing me to delete my mistakes and distractions, and to compress the tedious parts. Fortunately, ffmpeg breaks apart a video quite handily:

ffmpeg -i test-0000.mpeg -qmax 2 -f image2 test-%04d.jpeg

This is exactly the reverse of assembling the video: the separate frames are created as individual, sequentially numbered JPG files. Then I can manually delete a small number of frames, or write a script to skip or speed up entire sections.

I also needed to make an extra copy of each frame to slow down the playback. With xvidcap and the demonstrated program (GIMP, fotoxx) competing for the computer, it was best to capture the desktop at no more than 12 or 13 frames/second. However, ffmpeg wants to produce video that plays at 25 frames/second. This mismatch spawns a hyperactive mouse that is difficult to follow. After losing several arguments with ffmpeg, I decided the easiest course was to double frames.

Here is another example script. It extracts the four-digit number from the name of the original frame, allowing it to process only every other frame between 1290 to 2064, and creating two copies of all the other frames. Thus, the waiting-for-the-computer section zips along four times faster than the moving-the-mouse parts. Finally, ffmpeg is used to assemble a video from the new frames. (I discovered that for screen capture, versus photographs, a different encoding standard, h264, produced smaller files.)

let ix=1
let skip=0

# create two copies of each frame, to make the pace of the playback reasonable
#
/bin/rm -f video.mpg fr*.jpg

for i in ../test*.jpeg
do
    if [ $skip -eq 1 ]
    then
        skip=0
        continue
    fi

    orignum=`echo $i | sed -e "s/..\/test-//" | sed -e "s/.jpeg//"`
    nname=`printf "fr-%04d.jpg" $ix`
    cp $i ${nname}
    let ix=ix+1

    # do not double the boring stretch, in fact, halve it!
    #
    if [ $orignum -gt 1290 ] && [ $orignum -lt 2064 ]
    then
        skip=1
        continue
    fi

    nname=`printf "fr-%04d.jpg" $ix`
    cp $i ${nname}
    let ix=ix+1
done

# reassemble the video using the libx264 video codec
#
ffmpeg -f image2 -i fr-%04d.jpg \
    -vcodec libx264 -vpre lossless_ultrafast -threads 0 video.mkv

The moon is setting, and it's time to wrap up this post and close our book of spells (or web sites with manual pages for command-line tools). Thank you for visiting the lab with me.