Working with large files

I’ve been working on a 30gb Xml file today, importing it into one of our systems. There are always problems working with files this big – how do you get just a small segment to test on? how can you view small regions of it, or search it?

Most tools available for Windows aren’t up to the job, but i’ve finally found 3 which will help you should you have similar requirements:

1) Baretail. This will show you any part of a text file regardless of file length. It only loads what you’re looking at into memory and thus has a constant memory footprint. It also provides multiple highlights and will follow (tail) the file for updates. You can copy sections of the file from here into a new file (copy/paste).

2) Baregrep. This will search any size file for a given pattern.

3) frhed. By using the ‘partial open’ feature, you can get it to open just a fragment of any file in hex mode thus letting you see what’s actually there.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s