Assuming a long-running file extract process, we want to prevent further processing the file (by a consumer process) until the extract is fully successfully completed.

We want to eliminate any dependencies on capabilities of the consumer process. We also want an approach that does not depend on detecting that file extract is completed based upon its content i.e. detection does not depend on existence of footer record.

Risky or complex approaches

Last Modified timestamp lock

The consumer process ignores the extract file until it is at least n minutes old.

This is unreliable: if extract process crashes or an unexpected file system unmount occurs, the extracted file will appear completed and processing will continue.

Sentinel file

The extract process will, upon successful completion, write a sentinel file. The consumer process will look for existence of sentinel file before processing main extract file.

The core problem with this approach is that both extract and consumer processes need to implement the logic and also, it requires maintaining two files.

System file level locking

The file extract process places exclusive lock on the file. Once finished, it releases the lock. The consumer process picks up the file and processes.

The core problem with this approach is that once again, unexpected system failure will unlock the file before it is ready.

Better approach

File extract process will generate, in the target folder, a file with temporary name that consumer process will not recognise and thus ignore. As the last step of file extract process, this file will be renamed to expected file name.

This protects in all eventualities:

• in a case of extract process unexpected failure, even if the extract process does not clean up the file, the file has a file name that is ignored
• in a case of unexpected file system unmount, the file remaining on the file system will again have unrecognised name.

In Java, this is easily achieved as AtomicRenamer demonstrates:

    public static void main(String[] args) throws Exception  {
        File f = new File("/home/dalen/Downloads/muppet.zip");


        try (AtomicRenamer abc = new AtomicRenamer(f);
            FileOutputStream fos = new FileOutputStream(abc.getFile()) ) {
            Random rnd = new Random();

            for (int i=0; i<50*1024; i++) {
                fos.write('a' + rnd.nextInt(10));
            }

            abc.successful();
            logger.info ("Completed");
        }
    }

Using try-with-resources and encapsulating both AtomicRenamer and actual file output stream will ensure that:

• file gets a temporary name while written
• Java will take care of flushing and closing file output stream
• Java will take care of calling close on AtomicRenamer
  • if successful method is called on AtomicRenamer, the file will be renamed
  • if successful method is not called on AtomicRenamer, the file is deleted as it is assumed that writing has failed