FileAgent: Keeping a watchful eye

Day 8

The Connect:Direct FileAgent is a Java application that runs on Windows, Linux & UNIX.  It comes with Connect:Direct and is free.

The FileAgent is useful as you can give it many directories to watch and define some rules to decide whether or not it should perform some action on any new files that appear in the directories you gave it.

The action to be performed is whatever can be put in a Connect:Direct process. So that process could forward a file onward to another C:D node, or run a local program to process the file.

This is useful as it reduces the amount of scripting you might otherwise have to do to provide the same effect.  For instance you may have an application that produces a file in a certain directory which is to be sent to another C:D node.  This might be some sort of database extract that has to be sent to an ETL (Extract Transform & Load) platform before being loaded into a data warehouse.

This could be done in a few different ways: 

You could get the application people to add a call to their extract script to the Connect:Direct command line to initiate a transfer immediately after the extract is complete.  If the application team are fine doing that then this would be the most efficient way to do this.  The application team might to be uncomfortable doing this as they might see this as the C:D team's responsibility.  They might prefer just for C:D to pick up the file when it appears in the said directory.

You could write a script that watches for new files in a directory and transfers them onward to the ETL C:D node.  This would make things easy for the application team as they only have to tell you the directory the file will appear in and the file name to look for and where to forward the file to.  It can be tricky to get the script to cover all the cases that the script may encounter.  If you have a mixed environment including Windows and UNIX you would have to solve the problem twice as the platforms are scripted differently.

You could just add the directory and a simple rule to FileAgent using a template C:D process.  As the FileAgent is a Java application it looks and behaves very similarly on Windows & UNIX with only a couple of exceptions.

In the next post I will look at how FileAgent is configured and used in practice.