-- There already exists a manually invoked command-line script, called songify, for converting files from a source format to destination format.
-- The task is to create a new always-running Python script "watcher" that will run on a server that watches an AWS S3 bucket and automatically converts all new or changed files, logging actions appropriately.
-- songify.py -- Python script that accepts a source folder and song list filename -- It "processes" every song in the source folder this means to convert each file from a source format (mscz etc.) to a destination format (kxml) in a multi-stage process involving invoking external programs and doing various processing in Python
-- It produces an XML song list containing info about each song processed
-- songfiy.sh -- bash script that invokes songify.py and pipes stdout/stderr output to a log file
1. As a user, I want to upload a source file to any subfolder of the "watched" folder in a bucket on S3, and have it automatically be detected as a new file and converted to kxml (the destination format), with the resulting kxml file uploaded into the same folder, e.g.
-- The "watched" folder is songs
-- User creates a subfolder songs/batch1
-- User uploads source file songs/batch1/foo.mscz
-- songs/batch1/foo.kxml will automatically be created
2. As a user, I want to be able to see log the output from the conversion (to review errors) in a log file created in the source folder, e.g. if the source file is called foo.mcsz, the log file should be called foo.log, e.g.
-- songs/batch1/foo.log will contain log messages for the conversion
3. As a user, I want the system to automatically detect and re-convert updates to the source file, for instance after conversion it could set the datestamp of the source file and log file to be the same; later if it sees the datestamp is no longer the same (meaning it's new version of the source file), it will re-convert, e.g.
-- User updates songs/batch1/foo.mscz
-- songs/batch1/foo.kxml will be automatically recreated
4. As a user, I want each subfolder of the "watched" folder to contain an always-up-to-date song_list.xml file that contains song information about each kxml file in the subfolder (based on the existing songify.py logic that creates song lists), e.g.
-- songs contains subfolders batch1, batch2, etc.
-- songs/batch1/song_list.xml is the song list for all kxml files in songs/batch1
-- songs/batch2/song_list.xml is the song list for all kxml files in songs/batch2
5. As a user, I want to be able to see overall progress of the conversion system in a single central log file that I can watch with "tail -f" that indicates things like:
[Timestamp] Foo.mcsz -- new version detected -- starting conversion
[Timestamp] Foo.mscz -- conversion finished -- successful, song_list.xml updated
[Timestamp] Foo.mscz -- conversion finished -- encountered errors
6. As a sysadmin, I want to be able to follow clear README instructions to set up this system on a clean Ubuntu server (i.e. a Digital Ocean droplet).
7. As a sysadmin, I want to be able to set up the process as a Linux service or daemon in some way, so that I can start and stop it e.g. with the Linux supervisorctl or service commands
8. As a sysadmin, I want to have a log file for the process that I can check for errors etc. (not errors in conversions, which should go in the individual user-facing foo.log files noted above, but errors and exceptions in the conversion process, e.g. S3 login errors)
I'm a former Google product manager and am the programmer who created the existing code. I'm looking for someone to help me finish the above task quickly. I will review code and give feedback as needed. We will work together and communicate as two programmers.