No Chit Chat, just Smalltalk

Day 20

This is something that you do not see every day.  Below is an extract from an old Smalltalk workspace listing I found where I was experimenting with Cincom's ObjectStudio Smalltalk programming environment and the OLE/COM interface to Connect:Direct.

It was just as easy if not easier than VBScript to control Connect:Direct using Smalltalk.

D := OLEDispatcher new: 'CD.NODE'.
N :=  D call: 'Connect' params: (Array with: 'MY.NODE' with: '' with: '').


TXT := 'TEST001 PROCESS'                           + CrLf
           + '     MAXDELAY=UNLIMITED'             + CrLf
           + '     REMOTE=CD.REMOTE'               + CrLf
           + '     HOLD = NO'                      + CrLf
           + 'STEP01 COPY  FROM ('                 + CrLf
           + '     FILE=C:\TEMP\INPUT.TXT '        + CrLf
           + '     LOCAL /*$Windows NT$*/)'        + CrLf
           + '     TO ('                           + CRLF
           + '     REMOTE /*$Windows NT$*/'        + CrLf
           + '     FILE=C:\TEMP\OUTPUT.TXT'        + CrLf
           + '     DISP=(RPL))'                    + CrLf
           + 'PEND'.

P := D call: 'Submit' params: (Array with: TXT).

P at: 'ProcessNumber'


S := D call: 'SelectStats' params: (Array with: 'select statistics pnumber=62').
I := S call: 'HasMore'.
I := S call: 'GetNext'.
I at: 'MsgId'

Smalltalk is one of my favourite programming languages, due to its' simple syntax and object orientation.

It might be interesting to see how this compares to using OLE/COM from TCL.  I'll save that for another post.

Self Sign

Day 19

If you're using SSL/TLS digital certificates with Connect:Direct Secure+ when connecting to your 3rd parties, you might well have a security policy that states that you will not connect with machines from 3rd parties that use self signed certificates.

You probably would prefer to connect with a 3rd party node that presents a digital certificate that is signed by someone you trust, such as VeriSign.

It might be that the history of the configuration of the Connect:Direct connection and the security policy did not coincide.

You may be in need of finding out which of your existing connections use self signed certificates.

As usual the information is within the Connect:Direct statistics, but not viewable from the "select statistics" command from  within the Connect:Direct command line.

The following function will check the statistics records piped through it to check which Secure+ connections use certificates that were signed by themselves.  In other words it checks statistics records where Secure+ was used and where the certificate issuer is the same as the subject of the server certificate used by the connection.

function selfsigncerts
{
grep RECI=CTRC | awk -F\| '
{
record["CSPE"]=""
record["CERI"]=""
record["CERT"]=""

for(i=1;i<=NF;i++)
{
key=substr($i,0,index($i,"=")-1)
value=substr($i,index($i,"=")+1)
record[key]=value
}

name = record["PNOD"] ":" record["SNOD"]

if((record["CSPE"] == "Y") && (record["CERI"] == record["CERT"]))
{
connections[name]=record["CERI"]
}
}
END{
for (name in connections)
{
print name ":" connections[name]
}
}'
}

It is used like this, assuming you are in the work directory where the statistics files are:

$ cat S20121224.001 | selfsigncerts
unx.node:OTHER.NODE:(/C=GB/L=Lincoln/O=Bank/OU=IT/CN=OTHER.NODE/emailAddress=joe.blogs@bank.co.uk/SN=12345678)

Now you know which connections use self signed certificates you can go about getting them replaced with certificates you can trust via your trusted 3rd party such as VeriSign.

Other things you could check for are the encryption algorithms used by a connection. Over time encryption algorithms lose favour as they are considered weaker than others.

Certificate signing algorithms also need checking for compliance with security policies.  For example the MD5 checksum algorithm in the past was used for signing certificates, but is considered weak, and has been shown that it can be exploited.

You may have a security policy that states you don't use certain algorithms, and you may have to demonstrate that you don't use them, and if you do, identify them for remediation.

The next few blog entries will cover these issues.

Mistaken Identity

Day 18 

Most problems with translation tables amount to a case of mistaken identity. Sometimes it is the source of the file that is assumed to be something, but turns out to be something else.

For example someone says they are having problems transferring a file from a VMS system to UNIX and the Excel spreadsheet is not arriving in the correct format.

Well in this case you can not just look at this problem from a VMS/UNIX perspective. The Excel spreadsheet probably originated from a Windows machine. So how was it transferred to the VMS machine? Was it transferred in binary mode? Is the spreadsheet file really an Excel file, or just a .csv file?

The answers to those questions have an impact on the problem and its' solution. If the file was truely an Excel spreadsheet then you would want to transfer it in binary mode so the file ends up at its' destination literally the same as at the source of the transfer, no matter how many hops there are in the transfer.

It all depends on what is being used to produce the file to be transferred and what will end up consuming/processing the file at the destination. In the case of an Excel spreadsheet it will be a piece of software expecting a file the same as would be on a Windows machine, hence the binary mode.

If the source of the transfer was . csv file (comma separated values), i.e. a text file and was to be consumed/processed on a UNIX machine by an application, then we would want the file to arrive on the UNIX machine as a UNIX text file with each line terminated with a newline character as opposed to a carriage-return and newline characters on a Windows platform.

For this to happen we want Connect:Direct to treat the file as a text file and not binary as in the previous example. So we would not specify DATATYPE binary as before but use the default DATATYPE which is TEXT.

Some times you are told which codepages are being used on both ends of the proposed Connect:Direct transfer with absolute certainty.

For example you be told that a text file transferred from a mainframe was produced using codepage IBM-1140 and that the application on a Windows machine receiving the file is using UTF-8, an encoding for Unicode.

It really does depend on the application that will consume/process the file. It might be assumed that the application can handle UTF-8, or that as ASCII is a subset of UTF-8 there should be no problem.

In this case an international character might be used within the file on the mainframe that is available to it within the IBM-1140 codepage and this will be translated to the corresponding UTF-8 encoding of Unicode.

For characters that map directly to a single ASCII/UTF-8 there will not be a problem, but international characters can be encoded as 1,2,3 or even 4 byte UTF-8 encoded Unicode characters.

This is because UTF-8 is a variable byte character encoding. If the application is written to use Windows codepage CP-1252, then it will only be expecting single byte characters and not multi-byte Unicode characters that UTF-8 can encode. It will then probably choke on the multi-byte encoding or just not recognise what it is supposed to represent, and not process the file properly.

Imagine that data entered into an application on the mainframe is using one codepage, but the application was programmed to use a field delimiter character from another codepage. The file the application produces on the mainframe will contain data from one codepage and delimiters from another, and then transferred to another machine with codepage translation specified for the destination.

You may not be surprised to find that the field delimiter characters were not translated correctly for the destination.

In this particular case I suggested that the application programmer on the mainframe use a particular hex value character for the field delimiter that was available within the IBM-1140 codepage, and it turned out that the application on the Windows machine was using CP-1252 and not UTF-8.

It turned out for this particular file and applications that there were no special codepage requirements, as the default translation was sufficient.

So next time some one is enfatically, absolutely certain about codepage requirements, it might just be a good idea to check the facts for yourself, as it is easy for people to get this wrong.

Transformers

Day 17

Connect:Direct is available on both ASCII and EBCDIC character set machines. There are also many different code pages available to cater to different regions and languages. So it naturally comes about that translations of one character set to another will be needed from time to time.

In Connect:Direct this is achieved by default for translations between ASCII and EBCDIC in either direction for DATATYPE=TEXT files. For custom requirements, tradtionally this was achieved using translation tables and referring to them within the SYSOPTS clause of a Connect:Direct process.

Translation tables come in two flavours Single Byte Character Set (SBCS) and Double Byte Character Set (DBCS).

For SBCS translation tables, it has in the past been necessary for me to decode a "custom_translation.xlt" as the source to build it was not available. To do this on UNIX I wrote a short shell function to take a binary .xlt file and produce the source to build the SBCS translation table.

function dxlt
{
 if [[ $# -ne 1 ]]
 then
  echo
  echo "Usage: dxlt file.xlt"
  echo
  echo "Dumps the C:D transaltion table file.xlt."
  echo
  return
 fi
 echo
 echo " 0 1 2 3 4 5 6 7 8 9 a b c d e f\n"
 od -A x -t x1 $1 | cut -c6- | sed 's/^00$//'
}

Below is an example of using the above shell function:

$ dxlt custom_translation.xlt

 0 1 2 3 4 5 6 7 8 9 a b c d e f

00 00 01 02 03 37 2d 2e 2f 16 05 25 0b 0c 0d 0e 0f
10 10 11 12 13 3c 3d 32 26 18 19 3f 27 1c 1d 1e 1f
20 40 5a 7f 7b 5b 6c 50 7d 4d 5d 5c 4e 6b 60 4b 61
30 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 7a 5e 4c 7e 6e 6f
40 7c c1 c2 c3 c4 c5 c6 c7 c8 c9 d1 d2 d3 d4 d5 d6
50 d7 d8 d9 e2 e3 e4 e5 e6 e7 e8 e9 ad e0 bd 5f 6d
60 79 81 82 83 84 85 86 87 88 89 91 92 93 94 95 96
70 97 98 99 a2 a3 a4 a5 a6 a7 a8 a9 c0 4f d0 bc 07
80 20 21 22 23 24 15 06 17 28 29 2a 2b 2c 09 0a 1b
90 30 31 1a 33 34 35 36 08 38 39 3a 3b 04 14 3e e1
a0 41 aa 43 44 45 46 47 48 49 51 52 53 54 55 56 57
b0 58 59 62 63 64 65 66 67 68 69 70 71 72 73 74 ab
c0 76 77 78 80 8a 8b 8c 8d 8e 8f 90 9a 9b 9c 9d 9e
d0 9f a0 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7
e0 b8 b9 ba bb bc bd be bf ca cb cc cd ce cf da db
f0 dc dd de df ea eb ec ed ee ef fa fb fc fd fe ff

Another useful function is called "chars" which just shows you the printable characters within the current locale.

function chars
{
	echo
        echo "     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f"
	nawk 'BEGIN{for(i=0;i<=255;i++){printf "%c",i}}' | od -A x -t c $1 | cut -c6- | \
	sed 's/^00$//;s/[0-9][0-9][0-9]/   /g;s/   /  /g'
}

An example of using the above function is:

$ chars

   0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

00 \0                   \a \b \t \n \v \f \r
10
20    !  "  #  $  %  &  '  (  )  *  +  ,  -  .  /
30 0  1  2  3  4  5  6  7  8  9  :  ;  <  =  >  ?
40 @  A  B  C  D  E  F  G  H  I  J  K  L  M  N  O
50 P  Q  R  S  T  U  V  W  X  Y  Z  [  \  ]  ^  _
60 `  a  b  c  d  e  f  g  h  i  j  k  l  m  n  o
70 p  q  r  s  t  u  v  w  x  y  z  {  |  }  ~
80
90
a0    ¡  ¢  £  ¤  ¥  ¦  §  ¨  ©  ª  «  ¬  ®  ¯
b0 °  ±  ²  ³  ´  µ  ¶  ·  ¸  ¹  º  »  ¼  ½  ¾  ¿
c0 À  Á  Â  Ã  Ä  Å  Æ  Ç  È  É  Ê  Ë  Ì  Í  Î  Ï
d0 Ð  Ñ  Ò  Ó  Ô  Õ  Ö  ×  Ø  Ù  Ú  Û  Ü  Ý  Þ  ß
e0 à  á  â  ã  ä  å  æ  ç  è  é  ê  ë  ì  í  î  ï
f0 ð  ñ  ò  ó  ô  õ  ö  ÷  ø  ù  ú  û  ü  ý  þ  ÿ

Together these two functions are very useful for sorting out translation table problems where a UNIX machine is involved.
As with many problems it is important to understand the context surrounding the issue at hand.
In terms of codepage translation tables this means looking at what type of file is being translated, which codepage was used to produce the file in question, which translation table was used to transform it, and what codepage is being used to view/process it at the destination. If these are not taken into account it can make solving translation tables issues very difficult to solve.
In the next post I will walk through a particular codepage translation problem using the above functions.

Nostalgia … NOT

Day 16



Today I was asked if I would dig out an example of automating a Connect:Direct transfer using a Windows batch script.  



I actually prefer to use WSH (Windows Script Host/VBScript) scripts for doing anything more complicated than a one line command when scripting with Connect:Direct on Windows.



I did find one however and have added a few comments to explain what it is doing.



send.bat

Least busiest time on the Connect:Direct node?

Day 15

Often when you need to make a change to a production Connect:Direct server,  you want to know when would be the best time to do it.

Sometimes you want to add additional transfers but want to schedule them at a time that does not put all the load at one time of the day.

Either way you can use the following short script to give you a histogram showing the activity on the local node to identify the best times for scheduling transfers or making configuration changes with as little impact to existing transfers.

I first did this for UNIX Connect:Direct nodes and was then asked if something similar could be done for Windows Connect:Direct nodes.  The Windows script for this used VBScript and an HTML application showing the histogram in HTML and the command line technique you have seen previously.  I’ll save that description for a later post.

This post will just address the UNIX solution to this problem.

You need to feed the “cdhours” shell function with the Connect:Direct UNIX statistics files for the period you are interested in.  To do this you probably want to be in the “work” directory for the local node for convenience where the statistics files are generated.

An example of how to use the script is given in the comment at the top of the script.

This script was the beginning of a collection of scripts to handle querying the Connect:Direct UNIX statistics/configuration files and the Secure+ configuration.

Next we will look at showing a histogram of the volume of data going through a Connect:Direct node ordered by remote node.  Useful for capacity planning and also billing.

---

# Connect:Direct Activity by hour Histogram
# =========================================
#
# Usage: cdhours
#
# $ cat S20100903.??? | cdhours
#
# To get a better picture for the month you could say
#
# $ cat S201009??.??? | cdhours
#
# Hours   Transfers
# =====   =========
#
# 00      54 #############
# 01      48 ############
# 02      54 #############
# 03     20 ##############################
# 04    244 ###############################################################
# 05      86 ######################
# 06      66 #################
# 07      76 ###################
# 08      36 #########
# 09       0
# 10      44 ###########
# 11    190 #################################################
# 12      48 ############
# 13      62 ################
# 14      30 #######
# 15      28 #######
# 16      85 #####################
# 17    221 #########################################################
# 18       0
# 19       0
# 20       0
# 21       0
# 22       0
# 23       0
#

function cdhours
{
       _FILES=$*
       # How many columns does the terminal have
       _COLS=`tput cols`
       # We are interested in the Copy Termination ReCords (CTRC)
       cat $_FILES | grep RECI=CTRC | nawk "{print $2}" | \
       nawk -v cols=$_COLS '
BEGIN{
       # Initialise the array that represents the histogram
       hours_per_day=24
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               tally[hour]=0
       }
}
       # This section gets executed for all records passed to nawk
       {
               split($2,fields,":")
               # Update the tally of these records that falls within this particular hour
               tally[fields[1]]++
       }
END{
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               # Keep track of the maximum number of tally marks so we can scale the histogram
               if(tally[hour] > max)
               {
                       max=tally[hour]
               }
       }
       node_name_length=16
       # Scale factor to make the histogram fit in the terminal window
       scale=(cols-(node_name_length+1))/max;

       printf("Hours\tTransfers\n")
       printf("=====\t=========\n\n")
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               # Scale the histogram bar to fit in terminal window
               bar=tally[hour]*scale
               printf "%3s\t%5d ",hour,tally[hour]
               # Generate the histogram bar
               for(b=1;b <= bar;b++)printf "#";
               printf "\n"
       }
}
'}

---

Difficult C:D questions?

Day 14


Connect:Direct records almost everything in the Connect:Direct statistics.  The statistics can be queried by using the Connect:Direct command line program “direct”, or more friendly programs such as the Connect:Direct Requester, or the browser interface, and even Sterling Control Center (SCC).

The statistics can be queried for information regarding a particular process or in general for errors and even for evidence of compliance to standards etc.

Some queries however are not possible to express in the Connect:Direct command line and difficult in the other tools mentioned earlier.

Some of these more difficult queries are listed below:

• What is the maximum number of concurrent sessions being used, at what time and which nodes had the lion share of the sessions?
• Which transfers are not secured using Secure+?
• Which nodes are using self signed digital certificates?
• What was the total volume of data transferred ordered by remote node?
• When is the least busiest time on this Connect:Direct node?
• Which remote nodes are triggering local scripts/processes?
• What are the transfers that have had a failure, but have not been successfully transmitted later?

The reason why these and other queries are difficult to express in the Connect:Direct command line is that the Connect:Direct statistics contain information that you can not get at with the Connect:Direct command line.

If you have ever taken a look at the Connect:Direct statistics files on UNIX you might not have liked what you saw:



STAR=20100902 17:00:03|PNAM=PULL|PNUM=98765|SSTA=20100902 17:00:03|STRT=20100902 17:00:03|STOP=20100902 17:00:03|STPT=20100902 17:00:03|SELA=00:00:00|SUBM=aaaacd@unx.aaaa|SBID=aaaacd|SBND=unx.aaaa|SNOD=CD.OTHER|CCOD=0|RECI=CTRC|RECC=CAPR|TZDI=3600|MSGI=SCPA000I|MSST=Copy step successful.
:
etc.



The above Connect:Direct statistic record is for a COPY statement within a Connect:Direct process.  It is just one long line with all the fields separated by the ‘|’ character.  Each field contains the 4 character name for the field name, an equals sign followed by the value of that field.

The 4 letter field names are documented in the “Connect:Direct for UNIX User Guide”.  You do not need to know them all.  Just use the ones you need when you need them.



PNAM is Process Name
PNUM is Process Number
PNOD is Primary Node Name
SNOD is Secondary Node Name
CCOD is Condition Code
SFIL is Source File Name
DFIL is Destination File Name
DBYW is Destination Bytes Written



A simple UNIX command can help make these statistics files easier to read

$ cat S20100902.047 | grep RECI=CTRC | tr '|' '\n'

The grep for records that contain the string “RECI=CTRC” filters just those records that are Copy Termination ReCords i.e. produced by a COPY statement.

Which produces something like the following:



STAR=20100902 17:00:03
PNAM=PULL
PNUM=98765
:
SUBM=aaaacd@unx.aaaa
:
SNOD=CD.OTHER
CCOD=0
RECI=CTRC
:
MSGI=SCPA000I
MSST=Copy step successful.
:
PNOD=unx.aaaa
SNOD=CD.OTHER
LNOD=P
:
CSPE=Y
CSPP=TLSv1
CSPS=TLS_RSA_WITH_AES_256_CBC_SHA
CERT=(/C=GB/ST=Cheshire/L=Congleton/O=A Global Financial Institution Plc/OU=Middleware/CN=CD.OTHER/SN=69009876789765456787654567667729)
CERI=(/O=Trusted Network/OU=Trusted, Inc./OU=Trusted CA/OU=www.trusted.org/SN=78ee48de185b2071c9c9c3b51d7bddc1)
SFIL=\share123\outgoing\AAAAAA.123456.DAT
:
DFIL=/data/projectx/from_agfi/AAAAAA.123456.DAT
:
DBYW=1161
:
etc.



Now we know the format of the stats records and we have an easier way to view them we can write some shell functions to help us with some of our tasks with Connect:Direct on UNIX and even with these more difficult questions.

Next we will look at one of those questions and how it can be answered in more detail.