Posts Tagged ‘Media management’

Data management and the CIA

Since 2003, I’ve worked for a number of media companies, ranging from a company with 16 networked HD edit suites, two online suites, audio suites and an archive going back decades, to individuals working on a single computer. One thing that most of them understood was the need for robust backups: The equity is in the intellectual property. It must be protected. There must be a process to protect it.

Legal issues aside, the CIA procedurally screwed up, and mistakes are things beast learned from.

The CIA:

The CIA has tapes of 9/11 plotter Ramzi Binalshibh being interrogated in a secret overseas prison. Discovered under a desk, the recordings could provide an unparalleled look at how foreign governments aided the U.S. in holding and questioning suspected terrorists.

Well then. I guess they missed some of the tapes that they had intended to destroy in 2005. Oops.

How is it that virtually every media company I’ve ever seen can do a better job of data management than the CIA? Even the sole proprietorships. There is a lesson to be learned from this: The brief version is don’t be like the CIA. The long version is procedural:

Multiple Backups: Many companies use a 3 tier concept at a minimum: Aside from the working dataset, On-line backups and offline backups create a decent system. In the case of something like an Avid Unity, the working dataset and online backup are the same thing, and contained in a networked RAID (The ISIS uses RAID 6). Individual hard-drives can fail, RAID cards can fail, but there are multiple copies of the data.

Backup offsite: Facilities get damaged. The offline backup isn’t enough. The offline backups could theoretically be moved offsite, but if not, a separate facility with it’s own extant dataset might be prudent. The Universal Studios fire in 2008 comes to mind as an example of why this is important.

Love the WORM: Write Once, Read Many. A DVD-ROM can only be burned once…or for the people who can afford it: AIT cartridges can set up for write-once operation. This removes the temptation to recycle, and forces the creation of physical media with a single purpose: backups. I’ve seen people lay off media to DVD-ROMs, D5s, Any number of other HD tape formats…They all work, and one will fit your needs well.

Access/Metadata: Who gets to access the backups? How is that logged? Are there check-out procedures? Is the location of all media known? This is where the CIA messed up. They didn’t have decent records of the location, access, or use of their backups.

A little bit of thought can go a long way towards preventing major problems…The CIA gives us a teachable moment. Again, legal issues aren’t my point here. My point is data management. Had they practiced good backup procedures, they would have known where all the copies of the tapes were, and would have succeeded in their end goal:

The destruction of the tapes and suppression evidence potentially relevant to the Binalshibh case.

Backups. With Procedures. It’s good for you. It’s even Lean (for those into that sort of thing…)

Filemaker: Containers galore

I’ve used Filemaker for a while, mostly for various organization tasks. I had an interesting experience with it recently.

A part of a project for someone I’m working for called for a robust file management methodology. In addition to tracking the songs used by a music publisher, there needed to be a way to store the associated files, such as sheet music, mastered tracks, etc. Basically, I needed Filemaker to handle the storage and naming of files. One cannot rely on the end-user to maintain a static file system and naming convention with 100% accuracy.

As many Filemaker users know, the two ways for Filemaker to interact with files: One can store them in the database, the second can reference them from the database, recording only a file name and path to a field.

The problems with each method quickly become apparent in a media environment:

Storing very large files in (what is usually) a local database causes it to drag. Badly. Stored files are also just that: files. There is no way to take advantage of Filemaker’s ability to handle media…No embedded audio, no pictures that display…Just <<random.file>>. This, quite frankly, sucks. Filemaker: You need to do something about it.

Storing references is also problematic, because when files move and folders are renamed, the reference is then useless. However, at least with a referenced file, it can be played or displayed within Filemaker.

I guess I should mention that the client works on a Mac, which is my preferred platform, and typical of many media companies. However, what follows would work just fine on a PC.


My kludgy (Because of Filemaker’s limitations) solution: Note that the table we are using is called ‘Songs’, and that we are storing the full track in a folder called ‘Container Master Full Songs.’ The rest of the fields should be self-explanatory.

FullTrack Load
Clear [ Songs::TempContainer ]
[ Select ]
Insert File [ Songs::TempContainer ]
Set Variable [ $file; Value:(Songs::Song SKU & (Right ( Songs::TempContainer ; 4 ))) ]
Go to Layout [ “Songs” (Songs) ]
Export Field Contents [ Songs::TempContainer; “file:Container Master Full Songs/$file” ]
Clear [ Songs::Full Track ]
[ Select ]
Insert QuickTime [ “movie:Container Master Full Songs/$file” ]
Clear [ Songs::TempContainer ]
[ Select ]

In English:

Copy the file to a ‘swap space’ in the database, generate a file name, export it with that specific file name to a specific (non-user accessible place), and use that specific name to reference the file. The one interesting tidbit is:

(Right ( Songs::TempContainer ; 4 ))

This little string grabs a 3 character file extension (plus the period), since Filemaker relies on the extension, rather than the creator code to decide how to handle a file. This is a bad way to implement anything on OS X, since the rest of the OS, and many programs rely on creator codes, rather than relying on extensions. The problem with this method is that it only works with 3 character file extensions. Not a big deal, but still annoying.

Exports happen similarly: Copy the file to the database temp field, export and prompt user to name the file and location.

Is this ideal? No. Does it work? Yes.