The bad news: I don’t have unlimited hard drive space.
The good news: I did just get 4 TB.
I have kept my files disorganized across three different computers and this time but I am vowing to do it differently.
I need your help (yes, like anything, the first step to recovery is the admission of a problem).
I am not sure what the best way is to maximize my new hard drive space or the best way to organize it.
This is my current setup:
- (WORK) G5 w/ 300 GB HD at 95% full
- (WORK) iMac w/ 200 GB HD at 30% full (this is in our main auditorium and is primarily used for running ProPresenter)
- (PERSONAL) Mac Book Pro w/ 200 GB HD at 80
- (WORK) 1 TB External USB HD at 90% full
NOTE: We like to leave the iMac empty because we have caused issues when we have used it to archive and store stuff. I am not sure if that is paranoia or a good call, but I am open to suggestions. Besides that, everything is pretty close to being clogged.
Now, I did just get an additional 4 TB (2 external and 2 internal) and here are some of my non-negotiables.
- I need to move all of the video files off of my personal computer for central access and for my sanity.
- I export all videos at full resolution Standard Definition and zero compression (yes, it takes up a lot of space but it allows for high quality playback and re-editing later).
- I keep everything we make including stuff that we purchase and tweak (this ranges from series trailers and bumpers to testimony videos to message videos).
- I desperately want to start keeping more than one copy of everything (yeah, that means we don’t have any video backups right now).
How do you store and archive videos? If you were me, how would you do it?
Jeremy Denlinger says
Here at our church we use a drobo to hold all of our archived video. We have one of the first generation Drobo’s which is networked via a USB share on our G5 desktop which makes it accessible via our internal network. The newer versions of the Drobo are much easier to network out of the box and provides a reliable backup & storage solution.
Blane Young says
Cool, I will check it out! How large is it?
Brad Parler says
That’s the cool part of Drobo products is that they are basicly a next gen JBOD (Just a Bunch or Drives) but they operate in more of a RAID 5 fashion. The down side is no hot spare, and it take way too long to recover with them if you do have a failed drive.
With the older products the networking really makes the data transfer super slow (passing thru via the USB). If you need fast access (for multiple editors) you really need to get a SAN, and I don’t really recommend Gigabit if you’re doing HD production (it’s fine if it’s for web deliverable production).
Blane Young says
Gotcha, thanks for explaining Drobo.
That makes sense, I HEART RAID!
BrianNotess says
So are you looking for a strategy or hardware?
To some degree it also depends on what software you use for editing. We recently started shooting HD footage and editing in ProRes 422 which quickly wiped out our 4 TB of storage space. We had to come up with a strategy and although it hasn’t been completely implemented, here it is.
We have one 8TB RAID striped drive for archive (and backup). http://eshop.macsales.com/shop/hard-drives/RAID/Desktop/
Raw footage is dumped onto the system or media HD of the machine used for capture/recompression until the project is finished (when it is moved to Archive)
Active projects are stored on externals (remember the footage is backed up on the system it was captured on) and shuffled around from person to person as need be. All additional project assets are added to the external (and ideally duplicated on the system on which they were created).
Once a project is finished. And really finished, all of the media and project files on the external are moved to the 8TB archive drive (sometimes we use Media Manager to make the storage smaller and get rid of footage we never used and sometimes not).
Once the project is archived, the media can be deleted from the system drives of each station as needed.
For me the key is having the archive backed up (which the RAID drive takes care of). We don’t shoot tape anymore and wiping memory cards makes me nervous. I’d be lying if I said we had implemented this perfectly, but we all WANT to.
If we have files that need to be used across multiple projects, we try to keep them in a shared network drive (lower thirds, series graphics etc…).
Hope that helps. 😉
Blane Young says
Hey! This definetely helps.
I am looking for a strategy as I am not maximizing my current hardware (because I m disorganized and don’t know how to best use my new 2 TB external). Plus, I probably won’t be able to purchase anything else for at least 5 months.
I use Final Cut Pro and export 720 x 480 (16×9) SD with zero compression.
I don’t even think I will keep project files but I want to keep the master files and the web version we make.
My main question is how to do backup with what I have…
And also how y’all organize in terms of file structure.
BrianNotess says
I might definitely use one external for “active” projects and one external (attached to a work computer) as archive.
Keep re-usable stuff somewhere where it is most accessible.
To optimize space, I would definitely suggest using the FCP “Media Manager” tool for projects you are finished with. You can move your projects AND recompress them for a smaller file size, once you’re done.
I’d look at what the data-rate difference is between SD “Uncompressed” and ProRes. ProRes is a great editing codec and retains 95% of your important pixel information. It has also a roughly 50% smaller data rate which means less storage space.
You could also do your archive on a system or drive of an iMac and use the external to run time-machine and back it up.
Hope that helps.
Blane Young says
What do you think about not keeping the project files and just keeping the videos themselves?
How often do you use or re-use the project files and elements personally?
Thanks for all your help…
I am definitely a Time Machine fan!
Brad Parler says
I always keep the project files for that exact reason! I also have a huge swipe file that I work from called Creative – it’s anything that I find that sparks a creative process in my mind.
Blane Young says
I gotcha. I have never really gone back and used but one or two things so I would rather have to find again…of course, I save all of our purchased elements.
BrianNotess says
I like to keep project files for a couple reasons.
1. They typically aren’t too big (for FCP).
2. I’ve found myself more than a few times tweaking old projects for re-use when I thought I was totally done with them.
Blane Young says
I gotcha. I think for my needs at our church right now, I am going to just save videos that people see (full res, web, etc.)
I don’t think I am going to save any elements or project files…
In our context, the only thing we have reused is videos that we made but we have made zero to minor tweaks.
Eric says
great video: http://blog.chasejarvis.com/blog/2010/06/workflow-and-backup-for-photo-video/
What i do is that i have one harddrive internal (2 tb) that i work off of and two external 2tb hard drives that i backup to (two copies) when my internal 2tb gets full i delete the files since i have 2 copies already backed up.
I also follow this system for copying data:
Data Wrangling – Protecting Your Newly Recorded Media
This was copied & pasted from the Cinematographers Mailing List (CML) as an ideal methodology for handling newly recorded media and ensuring its safe travel to its intended storage destination.
I’m of the somewhat rigid mindset that media from any camera (wether film, tape, or data) needs to be handled with a consistent protocol EVERY time; no matter what the circumstances. By going through steps that might seem like ‘overkill,’ one creates the need to THINK about each action, and follows a system of self-policing safety checks.
Personally, my own system for data handling is as follows:
– The moment the media is removed from camera, it is “locked” (if possible), and I place a strip of Red 1/4″ paper tape with the camera ID and roll number (i.e. “A001”).
– When the media is placed into the card reader, I remove the tape and place it on the card reader itself, so I always know what is inside, and that it has not yet been offloaded.
– When the media is done offloading (and verified!), I first remove the tape from the card reader, then remove the media itself.
– I then “unlock” the media (again, if possible) and place a piece of green 1/4″ paper tape with the word “WAS” and the camera ID and roll number (i.e. “WAS A001”).
– I then place the ‘green-tape’d’ media in a card wallet on my cart until such time that it is needed back at camera.
In this manner, I NEVER hand the camera assistant a piece of media that isn’t unlocked AND also have a piece of green tape on it. If any questions should ever arise as to wether or not a certain card was downloaded, there is an immediate visual indicator (“Okay, so this card was A001… Look, here in the computer we see that A001 was already downloaded… Okay, this card is good to go back to camera).
I always re-format in the camera and I never re-format until absolutely necessary. In other words, I allow even the downloaded media to live on the cards for as long as possible until the card absolutely must be recycled.
I know it may seem silly or O.C.D. but these extra steps and rituals assure that I always know the state of every piece of media at every stage of the shoot.
Best,
– Jordan
Jordan Livingston
Digital Imaging Technician
http://www.DIGILOID.com
Blane Young says
Wow, this was a huge help!
Do you save project files and elements?
I am really thinking against it because we hardly reuse, will have the files in full res no compression (so we can edit if we need to), and it takes up space…
Thoughts?
Brad Parler says
I try to save the source files / the work product / and all exported files that are derived from the project.
Brad Parler says
DVDs, BDs, and External Drives are not made for long term data storage. If you are using a RAID – make sure that it’s at least a RAID 5 + Hot Spare or a RAID 6. Depending on the budget for the hardware I’d suggest a LTO-4 solution. LTO has a 30 year life and only costs $0.05 – $0.07 per GB!! (Way less than your average external drive). If the LTO solution is too much – I’d suggest a Storage Server that will do the appropriate RAID level.
If you’re completely foreign to LTO and not sure where to start I’d suggest that you look at the Cache-A Prime it’s a linux based Archive Appliance that sets on your network and just archives your files to LTO based on the schedule that you set. Think of it as a Toaster (as it requires very little IT skill to opperate) that just works! http://bit.ly/8YmyYj
(Full Disclosure – I sell the options that I’m recommending)
As to Work Flow and File Structure – I have a very mature and meticulous structure on how I function with my post-production, and Website projects. I’ll see if I can outline what I do per-project and share with the group.
Blane Young says
Thanks for the tip, I haven’t heard of that stuff but I will be checking it out!
Brad Parler says
Drobo is a great option – LaCie also have some great storage servers that will provide a RAID 5+S or RAID 6. Also let me know if you have any questions about the Cache-A. (or would like to get a quote on a solution etc).
Simon Yost says
A few thoughts:
– First, I think that storage space will always be finite. First it was shelf space for tapes and the life span of physical media. Now it is the amount of hard drive space.
– My team shoots primarily on Sony EX1s. Therefore, we are using XD CAM 720/60 as our format across the board. This isn’t as pretty as uncompressed, but it is a control point across the board.
– We are building different RAID arrays for production/working verses archive. The archive array is slower and more redundant.
– We tie stakholder expectations on the capability of our systems to resource investments. That way, internal stakeholders and clients can understand what they are paying for and how that equates to workflow and long term expectations.
– All said and done, our workflow is looking something like this:
1. Capture in XD CAM 720/60 on Sony EX1s
2. Copy entire SxS cards to Drobo
3. Import SxS footage on to media or removable volumes for editing
4. Copy final package in XD CAM 720/60 to encoding server with RAID array to archive copy of final
Hope this helps.
Simon
Blane Young says
Thanks for sharing, this is really helpful!
I am not sure if we are will get a Drobo anytime in the near future but it is great info…
BrianNotess says
XD Cam codecs are great quality for the “small” file size. It’s definitely an option if you want to “Media Manage” and recompress your footage when you are archiving.
Blane Young says
Cool, I will keep that in mind when we review our compression standards!