May 24, 2020

Organizing Astrophotography Files

Like many people, I've been struggling with how to organize all the data I capture with my telescope. In a given session, it's not uncommon to come away with 10GB of data and hundreds of files. Proper calibration requires that the lights, darks, flats, and dark flats match perfectly. With some projects taking taking multiple sessions over multiple weeks (or longer), it's easy to lose track of which files go together.

I want to share my system for organizing my data and why I think it is better than some of the other approaches you might find on the internet. I will also discuss some of the earlier approaches I took and why they didn't end up working out well.

Directory Structure

Here is the directory structure that I use:

  • Capture\
    • {date}\ — session start
      • Calibration\
        • Darks\
        • {filter}\ — flats & flat darks
      • {target}\
        • {filer}\ — lights
  • Calibration\ — master darks, bias, etc
  • Processed\ — final processed images
    • {target}-{date}\
  • Work\ — scratch area for processing
    • {target}\

Why this approach?

I had several goals in mind when designing this approach:

  1. Make it easy to keep track of the matching calibration & light frames
  2. Make it easy to transfer the data from the latest session to another device
  3. Make it easy to perform backups of the important files and ignore the unimportant ones (more on this later)
  4. Make it automated; it should not require any manual book keeping (spreadsheets)

When thinking about how to organize the data coming off the telescope, I realized that matching the directory structure to how I use the telescope is really important. It makes the book keeping happen automatically when the frames are captured.

Here's what my typical imaging session looks like:

  1. I set up my telescope in my backyard
  2. I use it for one or more nights
  3. I shoot multiple targets during those nights
  4. At one or more points during the process I shoot calibration frames (flats, darks, etc)
  5. I put it away

In this scheme, each imaging session is given its own directory, named by the date on which it started. This structure ensures that I can always find the calibration frames which match a set of light frames. I don't need to take duplicate calibration frames per day/target or maintain a separate spreadsheet mapping.

If I make changes to the telescope which don't require new calibration frames, I just continue the current imaging session. If something happens that causes new calibration frames to be needed, I start a new imaging session. For example, changes to the imaging train, rotating the camera, major new dust mote, etc.

By separating all the original/raw frames into their own Capture\ directory, it is easy to ensure they are backed up. The data for a single session can also be easily and quickly transferred from one device to another.

The Processed\ directory is where I store my master files. This includes the final images and the raw integrated masters for each filter (in case I want to reprocess).

The Work\ directory is where I store all the intermediary files generated during processing. e.g., calibrated flats, calibrated lights, registered lights, etc. I create one sub directory for each project I am working on. If I start running low on disk space, I delete these directories. I do not back them up as they can be extremely large and can always be reproduced from the originals.

Sequence Generator Pro (SGP)

I use Sequence Generator Pro to automate image captures. Here's how I configure it to work with this directory structure:

  1. At the beginning of the session, set the capture directory to D:\Astro\Capture\{date}\
  2. Set the file path pattern to %tn\%fe\%dt_%tm_%ft_%fe_%ct_%el_%su_%04
    • Example: D:\Astro\Capture\2020-05-22\M 51 - Whirlpool Galaxy\Ha\2020-05-24_105724_Light_Ha_NA_300sec_gain139_offset50_0031.<ext>

If I need to start a new session for some reason, I just change the capture directory.

To capture calibration frames, I create a new target in SGP and name it Calibration. I then fill it with the flats, darks, etc events as needed.

Alternatives

One capture directory per day

I tried using the same directory structure as above but with one key difference. I created one sub-directory per imaging date, not per session. In the beginning, this was fine since I never had an imaging session which spanned more than one night.

Things became difficult to manage as I started to image more. With multi-night sessions, the calibration frames were frequently taken on different dates from the lights. Having to constantly remember which dates match with which was too error prone for me.

This is a fine place to start, just be sure to make the switch once you start doing multi-night imaging sessions.

Organize by project

I briefly tried organizing by “project”.

  • Projects\
    • {target}-{date}\
      • {filter}\
        • {lights}\
        • {flats}\
        • {flat_darks}\
      • Processed\
      • Work\

There were a few problems with this approach:

  1. Sharing calibration frames across multiple projects captured during the same imaging session didn't work cleanly.
    • It requires duplicating the frames or manual bookkeeping, both I want to avoid.
  2. Sometimes I had multiple imaging sessions per project. This made keeping track of the calibration frames more difficult.
  3. File transfers were slow because the capture directories became bloated after several sessions.
  4. Backups were annoying because I had to exclude each of the Work\ sub-directories