No Chit Chat, just Smalltalk

Day 20

This is something that you do not see every day.  Below is an extract from an old Smalltalk workspace listing I found where I was experimenting with Cincom's ObjectStudio Smalltalk programming environment and the OLE/COM interface to Connect:Direct.

It was just as easy if not easier than VBScript to control Connect:Direct using Smalltalk.

D := OLEDispatcher new: 'CD.NODE'.
N :=  D call: 'Connect' params: (Array with: 'MY.NODE' with: '' with: '').


TXT := 'TEST001 PROCESS'                           + CrLf
           + '     MAXDELAY=UNLIMITED'             + CrLf
           + '     REMOTE=CD.REMOTE'               + CrLf
           + '     HOLD = NO'                      + CrLf
           + 'STEP01 COPY  FROM ('                 + CrLf
           + '     FILE=C:\TEMP\INPUT.TXT '        + CrLf
           + '     LOCAL /*$Windows NT$*/)'        + CrLf
           + '     TO ('                           + CRLF
           + '     REMOTE /*$Windows NT$*/'        + CrLf
           + '     FILE=C:\TEMP\OUTPUT.TXT'        + CrLf
           + '     DISP=(RPL))'                    + CrLf
           + 'PEND'.

P := D call: 'Submit' params: (Array with: TXT).

P at: 'ProcessNumber'


S := D call: 'SelectStats' params: (Array with: 'select statistics pnumber=62').
I := S call: 'HasMore'.
I := S call: 'GetNext'.
I at: 'MsgId'

Smalltalk is one of my favourite programming languages, due to its' simple syntax and object orientation.

It might be interesting to see how this compares to using OLE/COM from TCL.  I'll save that for another post.

Least busiest time on the Connect:Direct node?

Day 15

Often when you need to make a change to a production Connect:Direct server,  you want to know when would be the best time to do it.

Sometimes you want to add additional transfers but want to schedule them at a time that does not put all the load at one time of the day.

Either way you can use the following short script to give you a histogram showing the activity on the local node to identify the best times for scheduling transfers or making configuration changes with as little impact to existing transfers.

I first did this for UNIX Connect:Direct nodes and was then asked if something similar could be done for Windows Connect:Direct nodes.  The Windows script for this used VBScript and an HTML application showing the histogram in HTML and the command line technique you have seen previously.  I’ll save that description for a later post.

This post will just address the UNIX solution to this problem.

You need to feed the “cdhours” shell function with the Connect:Direct UNIX statistics files for the period you are interested in.  To do this you probably want to be in the “work” directory for the local node for convenience where the statistics files are generated.

An example of how to use the script is given in the comment at the top of the script.

This script was the beginning of a collection of scripts to handle querying the Connect:Direct UNIX statistics/configuration files and the Secure+ configuration.

Next we will look at showing a histogram of the volume of data going through a Connect:Direct node ordered by remote node.  Useful for capacity planning and also billing.

---

# Connect:Direct Activity by hour Histogram
# =========================================
#
# Usage: cdhours
#
# $ cat S20100903.??? | cdhours
#
# To get a better picture for the month you could say
#
# $ cat S201009??.??? | cdhours
#
# Hours   Transfers
# =====   =========
#
# 00      54 #############
# 01      48 ############
# 02      54 #############
# 03     20 ##############################
# 04    244 ###############################################################
# 05      86 ######################
# 06      66 #################
# 07      76 ###################
# 08      36 #########
# 09       0
# 10      44 ###########
# 11    190 #################################################
# 12      48 ############
# 13      62 ################
# 14      30 #######
# 15      28 #######
# 16      85 #####################
# 17    221 #########################################################
# 18       0
# 19       0
# 20       0
# 21       0
# 22       0
# 23       0
#

function cdhours
{
       _FILES=$*
       # How many columns does the terminal have
       _COLS=`tput cols`
       # We are interested in the Copy Termination ReCords (CTRC)
       cat $_FILES | grep RECI=CTRC | nawk "{print $2}" | \
       nawk -v cols=$_COLS '
BEGIN{
       # Initialise the array that represents the histogram
       hours_per_day=24
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               tally[hour]=0
       }
}
       # This section gets executed for all records passed to nawk
       {
               split($2,fields,":")
               # Update the tally of these records that falls within this particular hour
               tally[fields[1]]++
       }
END{
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               # Keep track of the maximum number of tally marks so we can scale the histogram
               if(tally[hour] > max)
               {
                       max=tally[hour]
               }
       }
       node_name_length=16
       # Scale factor to make the histogram fit in the terminal window
       scale=(cols-(node_name_length+1))/max;

       printf("Hours\tTransfers\n")
       printf("=====\t=========\n\n")
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               # Scale the histogram bar to fit in terminal window
               bar=tally[hour]*scale
               printf "%3s\t%5d ",hour,tally[hour]
               # Generate the histogram bar
               for(b=1;b <= bar;b++)printf "#";
               printf "\n"
       }
}
'}

---

Difficult C:D questions?

Day 14


Connect:Direct records almost everything in the Connect:Direct statistics.  The statistics can be queried by using the Connect:Direct command line program “direct”, or more friendly programs such as the Connect:Direct Requester, or the browser interface, and even Sterling Control Center (SCC).

The statistics can be queried for information regarding a particular process or in general for errors and even for evidence of compliance to standards etc.

Some queries however are not possible to express in the Connect:Direct command line and difficult in the other tools mentioned earlier.

Some of these more difficult queries are listed below:

• What is the maximum number of concurrent sessions being used, at what time and which nodes had the lion share of the sessions?
• Which transfers are not secured using Secure+?
• Which nodes are using self signed digital certificates?
• What was the total volume of data transferred ordered by remote node?
• When is the least busiest time on this Connect:Direct node?
• Which remote nodes are triggering local scripts/processes?
• What are the transfers that have had a failure, but have not been successfully transmitted later?

The reason why these and other queries are difficult to express in the Connect:Direct command line is that the Connect:Direct statistics contain information that you can not get at with the Connect:Direct command line.

If you have ever taken a look at the Connect:Direct statistics files on UNIX you might not have liked what you saw:



STAR=20100902 17:00:03|PNAM=PULL|PNUM=98765|SSTA=20100902 17:00:03|STRT=20100902 17:00:03|STOP=20100902 17:00:03|STPT=20100902 17:00:03|SELA=00:00:00|SUBM=aaaacd@unx.aaaa|SBID=aaaacd|SBND=unx.aaaa|SNOD=CD.OTHER|CCOD=0|RECI=CTRC|RECC=CAPR|TZDI=3600|MSGI=SCPA000I|MSST=Copy step successful.
:
etc.



The above Connect:Direct statistic record is for a COPY statement within a Connect:Direct process.  It is just one long line with all the fields separated by the ‘|’ character.  Each field contains the 4 character name for the field name, an equals sign followed by the value of that field.

The 4 letter field names are documented in the “Connect:Direct for UNIX User Guide”.  You do not need to know them all.  Just use the ones you need when you need them.



PNAM is Process Name
PNUM is Process Number
PNOD is Primary Node Name
SNOD is Secondary Node Name
CCOD is Condition Code
SFIL is Source File Name
DFIL is Destination File Name
DBYW is Destination Bytes Written



A simple UNIX command can help make these statistics files easier to read

$ cat S20100902.047 | grep RECI=CTRC | tr '|' '\n'

The grep for records that contain the string “RECI=CTRC” filters just those records that are Copy Termination ReCords i.e. produced by a COPY statement.

Which produces something like the following:



STAR=20100902 17:00:03
PNAM=PULL
PNUM=98765
:
SUBM=aaaacd@unx.aaaa
:
SNOD=CD.OTHER
CCOD=0
RECI=CTRC
:
MSGI=SCPA000I
MSST=Copy step successful.
:
PNOD=unx.aaaa
SNOD=CD.OTHER
LNOD=P
:
CSPE=Y
CSPP=TLSv1
CSPS=TLS_RSA_WITH_AES_256_CBC_SHA
CERT=(/C=GB/ST=Cheshire/L=Congleton/O=A Global Financial Institution Plc/OU=Middleware/CN=CD.OTHER/SN=69009876789765456787654567667729)
CERI=(/O=Trusted Network/OU=Trusted, Inc./OU=Trusted CA/OU=www.trusted.org/SN=78ee48de185b2071c9c9c3b51d7bddc1)
SFIL=\share123\outgoing\AAAAAA.123456.DAT
:
DFIL=/data/projectx/from_agfi/AAAAAA.123456.DAT
:
DBYW=1161
:
etc.



Now we know the format of the stats records and we have an easier way to view them we can write some shell functions to help us with some of our tasks with Connect:Direct on UNIX and even with these more difficult questions.

Next we will look at one of those questions and how it can be answered in more detail.

The new FileAgent

Day 13

The latest version of Sterling Commerce’s FileAgent is v1.3, and is a welcome release.  It has several features that were lacking in the past.  Several of these features will come as welcome relief to support staff of large deployments of C:D & FileAgent.

Text configuration file

The first feature of note is that the configuration of FileAgent which is normally stored in a binary .ser file can now also be stored as a text file.  This is important especially for support staff as there maybe many many rules and finding the rule that applies a certain pattern match may be difficult if the name of the rule is not a good clue.  Searching the text configuration can be a lot quicker that hunting the various levels of the configuration using the GUI.

However, please note that this textual representation of the binary .ser configuration file is not something that can then be used to generate a binary .ser file.  It is more akin to documentation.

If you have chosen to save the configuration as text, then FileAgent will keep the textual configuration and the binary .ser in sync.

Sterling Control Center

FileAgent can now be monitored by Sterling Control Center (SCC).  This is done by configuring both the FileAgent to send SNMP traps when there is a problem and SCC to receive them, and to be notified of the problem.  This is another feature which will please the support teams.

Dynamic Configuration

In previous versions of FileAgent any change to the configuration meant having to stop/re-start the FileAgent which was usually an inconvenience.  in this version of FileAgent you can choose whether the saved configuration file will be noticed by FileAgent when it has been modified and the new configuration activated immediately.  This feature will be valuable where FileAgent is kept very busy due to the volume of new files to process.

Sub-directory watching

In previous versions of FileAgent all sub-directories of the watch directories would be scanned for candidate files.  If you did not want this to happen you had to include the path of the watch directory in your rules with a pattern to match only those files found in that directory only. Now in the new version you can choose whether you want that functionality or if you prefer you can disable it.

This is a useful feature as too many directories may be scanned constantly unnecessarily.  This can happen if staff unaware of how FileAgent works start creating archive directories under the watch directories. In this case FileAgent would see the archive files as candidates that match some rule or other and process them accordingly, which is probably not what you intended.

Conclusion

I have yet to try out this latest version of FileAgent, but I look forward to seeing these new features put to good use.

I’ll keep you posted as I learn more from using this latest version.