Archive and Delete files

Closed - This job posting has been filled and work has been completed.
Web, Mobile & Software Dev Scripts & Utilities Posted 1 year ago

Hourly Job

Less than 30 hrs/week
Less than 1 week
$$$

Expert Level

I am willing to pay higher rates for the most experienced freelancers

Details

We have a problem with our QA boxes where:

- Some components fill up the drives with log files quickly.
- We shouldn't delete all the files. We need to save the log files that have our  messages in them so that we can go back and see what is going on. ... but the Market Data logs become overwhelmingly big pretty quickly.


1. We need some overnight automated process (below).   This would need to be run using the Windows 2008 Task Scheduler on some box.

2. In addition to the automated system, we would like to find some tools that simplify management of the disk usage. We would like to have some utility that lets us quickly zero in on the subdirectory that is using the most disk space.... Kind of like a pie chart of where your disk is being used.






Step 1: Copy QA Log Files from QA Boxes to Y: (\\1.1.1.1\share)

Copy all log files from the previous day or older to Y.  A log file is  any *.txt file under any subdirectory of c:\Raptor.
Once you know they have been copied to Y drive, then delete them on the CA Box.


Step 2: Efficient Retention

Build a Utility:  QAArchiver.exe that is run from the command line.

Usage is:   QAArchiver [TopLevel Directory of Archive]

Example:

QAArchiver.exe \\1.1.1.1\share\QALogs


The objective of this utility is to store the last 30 days of "meaningful" log files for each adaptor and/or RemoteMarketSub Status Log.

This utility is only applicable to QA log files. Production log files must be retained forever.

The utility recursively searches directories and purges old files based on the following rules:

A meaningful log file is a *.TXT file that has a size over 20 kilobytes AND:

Category A: contains at least one of these strings:   35=D, 35=8, 35=R    

Category B: contains 35=X, 35=S


A FIX Filename Suffix is the name of the adaptor. For example the file:

FIX__20130702070102CONNECTOR_-Mkt_Data_Synth_TOB_from_Queen

the part of the file name in Yellow is the Suffix.

Note that files of this type are also important:

StatusLog_20130701060717


For any single sub-directory AND FileName Suffix (treat StatusLogs as their own group as well), we will only keep on the Y drive:

Up to 1 GB of the most recent files in Category A
Up to 1 GB of the most recent files in Category B (or the entire last file if it is over 1 GB)

For any file that we are not going to keep, it should be deleted from the archive drive.


About the Client

(4.73) 3 reviews

United States
Burbank 12:30 PM

6 Jobs Posted
100% Hire Rate, 1 Open Job

$377 Total Spent
6 Hires, 1 Active

$9.89/hr Avg Hourly Rate Paid
39 Hours

Member Since May 30, 2012