Sandline Discovery - Insight, Ideas and eDiscovery
We process a lot of data for our clients and, while we've seen (and sometimes helped) them become increasingly more sophisticated in their data handling, we still work with plenty of folks who do not have official data handling procedures or corporate security policies. We offer a number of fast and secure options for clients to get data to us for review but we still receive a fair amount of questions on how to safely prepare data prior to shipping or transfer.
Last week we shared a Quick Tip called Robocopy Cheat Sheet that addresses the question, "How do I safely copy data from A to B?" This week we're answering the question, "How should I archive or compress that data before transferring it to you?"
Here's an actual email we received and an easy approach you might not yet be familiar with:
I need to send some data for processing and we care about its metadata. It came from a computer running Windows and we need to send it over the wire. What archive format should I put it in, zip, rar, 7z, anything else?
I'm glad you identified the source because it does depend a bit on where the data originated. We'll go ahead and assume the original drive was NTFS-formatted and you mainly care about the timestamps.
Consider that, if the archive format supports something, such as a filename containing the postal mark face, do the tools packing and unpacking it support that as well?
How well your archive preserves the original metadata is a mind-bending combination of factors: your original data format, the tool used to create it, the archive format, the tool to unpack it, the options you give the tool and, if that weren't enough, the operating system shell gets in the mix as well. If you choose one of these archivers, you could well be fine most of the time, but I can't recommend any for all circumstances.
Consider this, what if you could copy it from your NTFS drive to another NTFS drive that you can treat like a zip?
Fortunately, there's an option that does just that: virtual disks. These are files that, like zips and other archives, store whatever other files you please. To the software using them, however, their content looks just like it's on a regular disk. Traditionally, virtual disks use the extension .vhd for "virtual hard disk".
It's easy to create virtual disks! Encryption is optional but you probably don't need it since you'll send the archive over an encrypted connection or on a hardware-encrypted drive.
Choose "dynamically expanding," when you have the choice between dynamically expanding and "fixed size". Though the dialog recommends "fixed size", it makes no sense just for packing up some data, especially when you plan to send that data over the network. The fixed size option is like a hard suitcase, while the expanding disk is like a squishy carryon. Tossing your stuff in the suitcase might be a little quicker, but you'll lose that time checking your bag because it doesn't fit in the overhead bin.
Make the disk big. You have to pick a size, but don't waste time thinking about it. So long as it's big enough to hold the data, it will work. Being miserly won't help. If in doubt, just pick 2,000,000 MB, almost the maximum size. Don't worry: just because you selected "dynamically expanding", it won't take two terabytes.
Never choose FAT when you format the volume, if you care about timestamps.
Not all file systems can record creation and last access times, and not all file systems record them in the same manner. For example, the resolution of create time on FAT is 10 milliseconds, while write time has a resolution of 2 seconds and access time has a resolution of 1 day, so it is really the access date.
Even if that original media is FAT-formatted, do not use FAT for shipping data for processing. Getting accurate timestamps out of FAT is tricky because they don't reflect a universal time, but that's another article.
Also in the screen where you choose how to format the drive, stick with quick format, because it's, well, quicker. Volume label can be whatever you like – meaningful volume labels are considerate – and don't enable compression. Compression may or may not reduce your data size significantly, but in my tests, it has only ever made the .vhd file slightly smaller and sometimes significantly bigger. Also, in my tests, NTFS compression on a disks set up like this reduces copy speed by about one third.
Copy and Send
All you have to do now is point Robocopy at your source data and have it copy everything into your new virtual drive. Use your cheat sheet from our last blog post to get this part of it done.
When you're done, right-click the volume in disk management again, where you went to format it. Choose "detach". Now you can send the .vhd just like you would a zip.
Unlike most file formats in Windows, double-clicking won't open the .vhd. You open virtual disks from the disk management screen, the same place you create them. When you open a virtual disk, you'll notice another handy option: it lets you mount the disk read-only, which will minimize risk of damaging the data before processing.
And there you have it. Don't wonder what archive type works best – just use a virtual disk.
Originally written by Joe Ulfers. Updated by the Sandline Editorial Team.
We just sent you an email. Please click the link in the email to confirm your subscription!