The new FileAgent

Day 13

The latest version of Sterling Commerce’s FileAgent is v1.3, and is a welcome release.  It has several features that were lacking in the past.  Several of these features will come as welcome relief to support staff of large deployments of C:D & FileAgent.

Text configuration file

The first feature of note is that the configuration of FileAgent which is normally stored in a binary .ser file can now also be stored as a text file.  This is important especially for support staff as there maybe many many rules and finding the rule that applies a certain pattern match may be difficult if the name of the rule is not a good clue.  Searching the text configuration can be a lot quicker that hunting the various levels of the configuration using the GUI.

However, please note that this textual representation of the binary .ser configuration file is not something that can then be used to generate a binary .ser file.  It is more akin to documentation.

If you have chosen to save the configuration as text, then FileAgent will keep the textual configuration and the binary .ser in sync.

Sterling Control Center

FileAgent can now be monitored by Sterling Control Center (SCC).  This is done by configuring both the FileAgent to send SNMP traps when there is a problem and SCC to receive them, and to be notified of the problem.  This is another feature which will please the support teams.

Dynamic Configuration

In previous versions of FileAgent any change to the configuration meant having to stop/re-start the FileAgent which was usually an inconvenience.  in this version of FileAgent you can choose whether the saved configuration file will be noticed by FileAgent when it has been modified and the new configuration activated immediately.  This feature will be valuable where FileAgent is kept very busy due to the volume of new files to process.

Sub-directory watching

In previous versions of FileAgent all sub-directories of the watch directories would be scanned for candidate files.  If you did not want this to happen you had to include the path of the watch directory in your rules with a pattern to match only those files found in that directory only. Now in the new version you can choose whether you want that functionality or if you prefer you can disable it.

This is a useful feature as too many directories may be scanned constantly unnecessarily.  This can happen if staff unaware of how FileAgent works start creating archive directories under the watch directories. In this case FileAgent would see the archive files as candidates that match some rule or other and process them accordingly, which is probably not what you intended.

Conclusion

I have yet to try out this latest version of FileAgent, but I look forward to seeing these new features put to good use.

I’ll keep you posted as I learn more from using this latest version.

Driving the Connect:Direct command line

Day 12

So last time I showed you how to use the Connect:Direct SDK OLE/COM interface to Connect:Direct in a VBScript example.

The following example is very similar to the previous one with a few exceptions.

Firstly instead of using the OLE/COM interface we are driving the Connect:Direct command line using the "Direct.exe" program.  We are also running the "Direct.exe" program hidden in the background.

You also won't see any signing on to Connect:Direct as this is taken care of using the "CD Client Connection Utility" to set the default node and user credentials which are picked up when we run the "Direct.exe".

The commands to be sent to the Connect:Direct command line are put in a temporary file and that file is supplied as a parameter when we run the "Direct.exe".

The text of the command that contains the Connect:Direct process is also a little different. In this script you will see it prefixed with the "submit" command, while in the previous script we called the "Submit" function of the "Node" object.

It is important to note that the Connect:Direct process contains the "MAXDELAY" statement if you want the script to wait for the Connect:Direct process to be submitted then run and get the return code.

If you do not have this statement in the process then the process will be submitted and that is all, before control is returned to the VBScript.  As submitting a process hardly ever fails the script could see a zero return code i.e. the submitting the process had no error, but when the Connect:Direct process runs and possibly fails the script will have finished and been unaware of the failure.

So usually for batch scripts you will want to specify the "MAXDELAY" statement so your script gets the result of running the Connect:Direct process and not just submitting it.

You may not want to specify "UNLIMITED", but some reasonable time-out figure such as "01:00:00" for an hour.

For the script below to work it must be placed in the same directory as the "Direct.exe" program.  

This is usually "C:\Program Files\Sterling Commerce\Connect Direct v4.x.00\Common Utilities" .  Where "x" may be different depending on the version you are running.

Both this and the previous script are examples of simple dynamic Connect:Direct process creation.  Later we will look at some more elaborate generation of Connect:Direct processes that apply to a wider range of applications.

---

' Windows Script Host Sample Connect:Direct Script using the CLI
'
' Includes ability to push and pull files to/from the other Node.

' Declare variables to represent objects

Dim fs, WshShell, tempDir, handle

' Connect:Direct condition codes values 

warningConditionCode = 4
errorConditionCode   = 8

'
' Function to either pull/push a file to/from another node
'                       
Function CDTransfer (pname, fromFile, toFile, otherNode, direc)

        If direct = "PUSH" then
                localOrRemote = "LOCAL"
                remoteOrLocal = "REMOTE"
        else
                localOrRemote = "REMOTE"
                remoteOrLocal = "LOCAL"
        End If

        ' Build the text of the submit command and the process

        txtProcess = "submit"                          + vbCrLf _
                + pname & " PROCESS"                  + vbCrLf _
                + "     MAXDELAY=UNLIMITED"             + vbCrLf _
                + "     REMOTE=" & otherNode            + vbCrLf _
                + "     HOLD = NO"                      + vbCrLf _
                + "STEP01 COPY  FROM ("                 + vbCrLf _
                + "     FILE=" & fromFile               + vbCrLf _
                + "     " & localOrRemote               + vbCrLf _
                + "     TO ("                           + vbCrLf _
                + "     " & remoteOrLocal               + vbCrLf _
                + "     FILE=" & toFile                 + vbCrLf _
                + "     DISP=(RPL))"                    + vbCrLf _
                + "PEND;"                               + vbCrLf _
                + "quit;"                             + vbCrLf

        ' Get temporary file names for input/output with the CLI
        inp=tempDir.path & "\" & fs.GetTempName
        outp=tempDir.path & "\" & fs.GetTempName

        ' Write the input to the CLI in a file
        set handle = fs.OpenTextFile(inp,2,True)
        handle.write txtProcess
        handle.close

        ' Run the CLI hidden, redirecting input and recording the output to a file
        cmd="%comspec% /c Direct.exe -z" & Chr(34) & outp & Chr(34) & " < " & inp
        CDTransfer=WshShell.run(cmd,0,True)

        ' Remove temporary files
        Set handle = fs.GetFile(inp)
        handle.Delete
        Set handle = fs.GetFile(outp)
        handle.Delete
End Function

Function CDSend ( pname, fromfile, toFile, otherNode )
        CDSend = CDTransfer ( pname, fromfile, toFile, otherNode, "PUSH" )
End Function

Function CDRecv ( pname, fromfile, toFile, otherNode )
        CDRecv = CDTransfer ( pname, fromfile, toFile, otherNode, "PULL" )
End Function

' Processing starts here

set fs = CreateObject("Scripting.FileSystemObject")
set tempDir = fs.GetSpecialFolder(2)
Set WshShell = CreateObject("WScript.Shell")

rtc = CDSend ( "SEND","c:\temp\input.txt", "c:\temp\output.txt", "CD.OTHER")

if rtc > warningConditionCode then
        wscript.echo "Send Failed"
        'Return the completon code to the command prompt
        wscript.quit(rtc)
End If

rtc = CDRecv ( "RECEIVE","c:\temp\remote.txt", "c:\temp\recv.txt", "CD.OTHER")

if rtc > warningConditionCode then
        wscript.echo "Recv Failed"
        'Return the completon code to the command prompt
        wscript.quit(rtc)
End If

---

OLE/COM & Connect:Direct

Day 11

Sometimes we want to be able to automate tasks with regard to Connect:Direct.  On Windows we could do this using batch files (.bat or .cmd), or with an application using the Connect:Direct Windows SDK.  My preferred way is to use some form of VBScript and the OLE/COM interface that the Connect:Direct Windows SDK provides.

That sounds more complicated that it really is.  I'll show an example Windows Script Host (WSH) script that uses the OLE/COM interface to connect to the local node and send/receive files  with a remote node.

Start by looking at the example below from the comment "Processing starts here" where you will see the creation of an object of type "CD.Node".  If my memory serves me well this is not well documented in the Connect:Direct Windows SDK manual.

Once we have an object that represents our local node in my case called "NICKE" we can make a connection with our credentials.  In this example the user name and password are empty strings and it makes the connection based on how the "CD Client Connection Utility" is setup.  This utility allows you to define several user credentials and which one is to be used as the default when none are given as in my example below.

The rest of the script then goes on to perform a transfer using the "CDSend" function to a remote node called "CD.OTHER", check for an error and then perform a receive of a file and then another check for an error.  Both "CDSend" & "CDRecv" make use of a generic function "CDTransfer" to perform a file transfer with the direction of the transfer passed as a parameter which is then used to generate a dynamic Connect:Direct process which is the submitted using the "Node.Submit" function and then the statistics for the process just submitted and executed are analysed and the highest condition code returned to the caller of the "CDTransfer" function.

At the end of the script the connection with the local node is disconnected and the "Node" object de-allocated.

The script below can either be run by double-clicking directly on the ".vbs" script in the file explorer or by giving the name of the script as a parameter to the "CSCRIPT.EXE" program as follows:

C:\> cscript cdvbsdk.vbs

In the next post I will an alternative method of driving Connect:Direct from VBScript.

---

' Windows Script Host Sample Connect:Direct Script. 
' 
' Includes ability to push and pull files to/from the other Node. 
 
Dim Node        ' Represents the Connect Direct Node we are using. 
Dim procs       ' A collection of process records. 
Dim proc        ' An individual process record. 
Dim stats       ' A collection of statistic records. 
Dim stat        ' An individual statistic record. 
 
' Connect:Direct condition codes values 
 
warningConditionCode = 4 
errorConditionCode   = 8 
 
 
 
 
Function CDTransfer (fromFile, toFile, otherNode, direc) 
 
        If direct = "PUSH" then 
                localOrRemote = "LOCAL" 
                remoteOrLocal = "REMOTE" 
        else 
                localOrRemote = "REMOTE" 
                remoteOrLocal = "LOCAL" 
        End If 
 
        ' The process text to be submitted. 
 
        txtProcess = "TRANSFER PROCESS"                 + vbCrLf _ 
                + "     MAXDELAY=UNLIMITED"             + vbCrLf _ 
                + "     REMOTE=" & otherNode            + vbCrLf _ 
                + "     HOLD = NO"                      + vbCrLf _ 
                + "STEP01 COPY  FROM ("                 + vbCrLf _ 
                + "     FILE=" & fromFile               + vbCrLf _ 
                + "     " & localOrRemote               + vbCrLf _ 
                + "     TO ("                           + vbCrLf _ 
                + "     " & remoteOrLocal               + vbCrLf _ 
                + "     FILE=" & toFile                 + vbCrLf _ 
                + "     DISP=(RPL))"                    + vbCrLf _ 
                + "PEND"                                + vbCrLf
 
 
        set proc = Node.Submit(txtProcess)             ' Submit the process 
 
        ' Get the statistics regarding the process that has now finished. 
 
        set stats = Node.SelectStats("SELECT STATISTICS PNUMBER=" & proc.ProcessNumber) 
 
        ' If any step in the process has a condition code greater than "warning" we have a problem. 
 
        highestConditionCode =0 
 
        For Each stat in stats
 
                if stat.ConditionCode > highestConditionCode then 
                        highestConditionCode = stat.ConditionCode
                End If 
 
        Next 
 
        If highestConditionCode > warningConditionCode then 
 
                CDTRansfer = highestConditionCode
 
        End If 
 
End Function 
 
 
 
 
Function CDSend ( fromfile, toFile, otherNode ) 
        CDSend = CDTransfer ( fromfile, toFile, otherNode, "PUSH" ) 
End Function 
 
Function CDRecv ( fromfile, toFile, otherNode ) 
        CDRecv = CDTransfer ( fromfile, toFile, otherNode, "PULL" ) 
End Function 
 
 
' ------------------------------------------------------------------------------------------ 
' 
' Processing starts here 
' 
Set Node = CreateObject("CD.Node")              ' Make the OLE/COM connection to Connect:Direct 
 
Node.Connect "NICKE","",""      ' Sign on 
 
exitcode = 0 
 
rtc = CDSend ( "c:\temp\input.txt", "c:\temp\output.txt", "CD.OTHER") 
 
if rtc > warningConditionCode then 
        wscript.echo "Send Failed" 
        exitcode = exitcode + rtc
End If 
 
rtc = CDRecv ( "c:\temp\input.txt", "c:\temp\recv.txt", "CD.OTHER") 
 
if rtc > warningConditionCode then 
        wscript.echo "Recv Failed" 
        exitcode = exitcode + rtc
End If 
 
' You need to disconnect from the Node or you will leave cscript or wscript processes hanging around 
Node.Disconnect 
Set Node=Nothing 
Wscript.Quit(exitcode) 

---

FileAgent: Wood from the trees

Day 10

So last post we were about to start the FileAgent and test our FileAgent rule and associated C:D process.  First check that your Connect:Direct server is up and running.   You can do this on Windows by using the "CD Admin Tool" that you will find in the start menu for Connect:Direct.  It will show you whether or not the Connect:Direct server is started by showing a green traffic light for up and running and a red traffic light for stopped.  You can start the Connect:Direct server by right clicking the traffic light for your node and selecting "Start".  You can alternatively use the "services" control panel to check/start/stop the Connect:Direct service.

 Start the C:D Requester as we will be using that to monitor the activity of Connect:Direct, and also looking at the statistics files for any possible errors.  Double click the "Process Monitor" in the left hand pane of the requester and fill in the form as follows.

Here you can see that I have set it to update the display of what's happening with Connect:Direct every second.  Also double click the "Select Statistics" in the left hand pane of the Requester and fill in the following way:

Here I'm saying that I want the last 5 minutes worth of statistics and I want that list of statistics updated every minute.  So if anything fails while I'm trying to get the FileAgent rule or process working I can find out what happened quickly.  I find it also convenient to tile the "Process Monitor" & "Select Statistics" windows which you can do by using one of the toolbar buttons on the right in the Requester window.

I also open a file explorer on the directory to be watched by the FileAgent.  In our FileAgent rule the source file should be deleted from the watch directory only after successful transmission of the source file to its destination.  I usually create a test file in the directory above the watch directory and update the contents and save the file before dropping it in the watch directory for FileAgent to see.

Above you can see I'm in the directory above with the candidate test file before dropping it into the watch directory "DataABC".

FileAgent looks at the watch directory every minute depending on the configuration and if it finds a candidate that matches one of its rules it records the fact and decides to come back in another minute to see if the file is still growing in size.  If so it waits again until it believes the file has been finished writing to by either a local application or if it was a file being received by Connect:Direct from a remote node.

So you should start FileAgent and then leave it a minute or so to settle before dropping the updated file in the watch directory.  Then you have to be patient and wait a couple of minutes more before you might see some activity in Connect:Direct.

Open a command prompt and navigate to the FileAgent install directory. In my case that is "C:\Program Files\FileAgent".  Although on Windows the FileAgent is installed as a Windows service with manual startup.  However for testing it is often easier to run the FileAgent from the command line in verbose mode and re-directing the output to a log file.

I waited for a minute or so and then updated the data file using my favourite editor and then copied the file into the watch directory. I then watched the "Process Monitor" window and saw the following indicating that the C:D process was being run:

When the "Select Statisics" window refreshed I found the following:

Here I have highlighted the statistic row of interest showing a record type of "CTRC" which stands for Copy Termination ReCord and a condition code "CC" of 0 indicating that the transfer was a success.  Double clicking on that row will give a more detailed view of the statistic record:

So there you can see a successful transfer of the file to the remote node.

You can stop the FileAgent that you started from the command line either by pressing the CTRL+C keys if you are in a hurry and not worried about properly shutting down the FileAgent.  If you were more concerned about that you would create a file in the installation directory of FileAgent called "shut" which signals to FileAgent it should shutdown.  I consider myself working in a test environment so I use CTRL+C as I am the only one using this particular environment.

If you were to look in the FileAgent.log file that the FileAgent created you will see why the option that created the information was called "verbose".  It is sometimes very difficult to see the wood for the trees when looking at this kind of output. I myself use an editor with syntax highlighting to help me more easily see the things I'm looking for.

 Here I have highlighted in green the fact that the C:D process was submited, and in the next few lines you see that submit command was accepted.  You can also see in the lines above that one that the FileAgent variables get substituted before the C:D process is submitted to Connect:Direct.

So that's FileAgent in a flash.  It can be a bit tricky to set-up initially, but it can save you from having to write batch/wsh/shell scripts and makes it easy for applications to send/recieve data without being too tightly coupled with Connect:Direct.  What could be easier than just moving a file into a directory for it to be sent on its way?  

In many cases you can just use the single Connect:Direct process template you saw earlier for any additional FileAgent rules you might add for Windows/UNIX destinations.  At a later date I will show you how you can use FileAgent together with a script to generate dynamically specialised C:D processes for a variety of platforms while keeping the FileAgent configuration simple.

FileAgent: Rules of the game

Day 9

On Windows you can configure FileAgent by selecting the shortcut on the desktop or using the Start menu "Configure Connect Direct FileAgent".  You should see something like the following:

The FileAgent comes with a default configuration file which is what it is showing above.  You select the configuration you want to work with in the left pane of the application.  In the right pane you have two tabs.  The tab that is showing above is the "FileAgent" tab where we configure it to connect via the API with the Connect:Direct.

We also list the directories we want to FileAgent to watch.  There should be no reason to change the default values that you see above that were already in the configuration, unless they clash with some other software already using those values.  

You can have multiple instances of FileAgent running which would need you to have separate "Gate Keeper" ports for each.  In most cases that would not be necessary. 

By default when FileAgent is running it will use this "Default_Config.ser" configuration file unless overridden by options on the command line if running FileAgent from the command prompt, or by editing the appropriate .lax file to override the arguments that get passed to FileAgent.  If running as a Windows service then that would be the "cdfa$.lax", or the "cdfa.lax" if runing from the command prompt or shortcut on Windows or on UNIX.

If you just use the "Default_Config.ser" then you will not need to override anything.

On the "Rules" tab you can see a "Submit Process Rules" tab within it.  This is where we define our rules to automatically submit C:D processes for the new file detected in the directory that was specified to watch.  If there are any problems with submitting a process from FileAgent the first things to check will be the "Enabled" check-box on this tab and also within the definition of the matching file pattern as can be seen on the next screenshot.

In this rule I have defined a match for all the .dat files within the watch directory.  On detecting a new file that matches this pattern FileAgent will submit the C:D process specified in the "Process Name" field with the arguments to be passed to the C:D process on in the next field.

In my C:D process I have some variables that take their values from the "Process arguments" field within this FileAgent rule.  The C:D process variables are &NODE which is assigned to CD.REMOTE, &FILESPEC and &NAME .  They take their values from some built in FileAgent variables "%FA_FILE_FOUND." and "%FA_NOT_PATH." respectively.  It is important to note that these special FileAgent variable names start with a "%" and end with a "." . Miss off the period at the end of the name and they will not work.

Above you can see the C:D process that is used by the FileAgent rule.  Previously when we sent a test file using the C:D Requester we used the "Send/Receive File" function in the left hand pane of the Requester which presented you with a form which we filled out.  Behind the scenes that form constructed a textual C:D process like the one you see above.

I will go into C:D processes in more detail at a later date.  For now hopefully you see some sort of script with a COPY command where the source of the copy is specified by &FILESPEC and the destination by &NAME .  Also note that the destination node or secondary node (SNODE) is specified by the C:D process variable &NODE .

This means that potentially we could use this single C:D process template for all our FileAgent rules.

If the transfer is successful then the source file is removed as it has been safely received at the destination.

Next post I will explain how to run the FileAgent and test that this rule is working.

FileAgent: Keeping a watchful eye

Day 8

The Connect:Direct FileAgent is a Java application that runs on Windows, Linux & UNIX.  It comes with Connect:Direct and is free.

The FileAgent is useful as you can give it many directories to watch and define some rules to decide whether or not it should perform some action on any new files that appear in the directories you gave it.

The action to be performed is whatever can be put in a Connect:Direct process. So that process could forward a file onward to another C:D node, or run a local program to process the file.

This is useful as it reduces the amount of scripting you might otherwise have to do to provide the same effect.  For instance you may have an application that produces a file in a certain directory which is to be sent to another C:D node.  This might be some sort of database extract that has to be sent to an ETL (Extract Transform & Load) platform before being loaded into a data warehouse.

This could be done in a few different ways: 

You could get the application people to add a call to their extract script to the Connect:Direct command line to initiate a transfer immediately after the extract is complete.  If the application team are fine doing that then this would be the most efficient way to do this.  The application team might to be uncomfortable doing this as they might see this as the C:D team's responsibility.  They might prefer just for C:D to pick up the file when it appears in the said directory.

You could write a script that watches for new files in a directory and transfers them onward to the ETL C:D node.  This would make things easy for the application team as they only have to tell you the directory the file will appear in and the file name to look for and where to forward the file to.  It can be tricky to get the script to cover all the cases that the script may encounter.  If you have a mixed environment including Windows and UNIX you would have to solve the problem twice as the platforms are scripted differently.

You could just add the directory and a simple rule to FileAgent using a template C:D process.  As the FileAgent is a Java application it looks and behaves very similarly on Windows & UNIX with only a couple of exceptions.

In the next post I will look at how FileAgent is configured and used in practice.

The other way around

Day 7

For my brother Lee he only needed to use his connection in one direction, from his Windows node to the remote UNIX node.

If he had needed to do it the other way around this is what I would have told him.

The concepts are the same, just presented differently as I said previously. We already have the netmap configuration, but we will need to make a configuration to allow the remote UNIX machine to send files to our local node.

The files that are received by our local node from the CD.REMOTE will have to be owned by a user account on our local node.  We could just use any local user account but it would be better to create a user specifically for this purpose and lock down the directories as mentioned previously when we set up the UNIX side of the configuration.

First let's set up Connect:Direct to allow a local user appusr1 to work with Connect:Direct.  Double-click the "User Authorities" in the left pane of the Connect:Direct Requester.  As you can see in the image below there are some entries that start with an asterisk which indicates they are templates.  In our case we are defining appusr1 as a non-administrative user of Connect:Direct.

Click the "New Genuser" and fill in as below. The controls will inherit from the templates you saw earlier and of course you can override those values here if you desire.

So how do we associate a incoming remote transfer with the local user we want to own the files?  This is done using the proxy definition.  Double-click "Proxies" in the left hand pane of the Requester and enter the fields as below.

So we are saying here that the remote user appusr1 from the remote node CD.REMOTE will be mapped to the local user appusr1.  It doesn't have to be the same name as I have done here.  Normally I create user accounts that reflect the name of the application that will consume the file if a suitable user account doesn't already exist.

If we didn't define a proxy for the remote Connect:Direct user, the remote Connect:Direct process would have to supply a SNODEID containing a local user account name on our local node together with a password.

This is not good for several reasons.  It is not a good idea to advertise local account information to anyone outside of your organisation.  Also hard coding passwords in scripts of any kind is also not a good idea because they might be seen, and if you have an information security department they will want you to change your passwords frequently which would mean amending all your scripts that contain passwords frequently.

The nice thing about Connect:Direct proxies is that the remote node doesn't care about what local user account you might be using, and they do not have to supply a password.  If their user is authorised to use Connect:Direct on the remote node they do not need to do anything regarding the user being used as you have covered that in the proxy definition.

For my brother's connection they were not using Secure+ to encrypt the data in transit or use it to authenticate using digital certificates.  If you read my post "Secure IT" then you will know that I always use Secure+.  In a later post I will give an example of doing just that.