Day 23 Translation what's the difference

Day 23

Translation what’s the difference

First let’s create some Connect:Direct translation tables using PowerShell:

PS C:\Users\nicke> conv ibm285 iso-8859-15 @(0..255) | Set-Content -Encoding byte ibm285-iso-8859-15.cdx

PS C:\Users\nicke> conv ibm01146 iso-8859-15 @(0..255) | Set-Content -Encoding byte ibm01146-iso-8859-15.cdx

Here we are converting from codepage ibm285 IBM EBCDIC (UK) to iso-8859-15 which has the Euro currency symbol, and converting all the byte values from 0 through to 255 (that is what the @(0..255) means), saving the result with the Connect:Direct Windows translation table file extension .cdx.

We then do the same thing for producing a translation table that will convert from IBM EBCDIC (UK-Euro) to iso-8859-15 .

Using the Powershell function below we can see the difference between these two translation tables.

# List difference between translation tables
function xlt_diff ([byte[]]$tbla,[byte[]]$tblb) {
    0..255 | %{
        if($tbla[$_] -ne $tblb[$_]) {
            "{0:x} : {1:x} | {2:x}" -f $_,$tbla[$_],$tblb[$_]
        }
    }
}

The above functions can be used as follows:

PS C:\Users\nicke> xlt_diff (cat -Encoding byte .\ibm285-iso-8859-15.cdx) (cat -Encoding byte .\ibm01146-iso-8859-15.cdx)
9f : 3f | a4

The output above shows that two translation tables differ when they map hex byte value 0x9f. In the first table it maps to hex value 0x3f, and in the other to 0xa4.
Now if we create the translation tables for translating back to either ibm285/ibm01146 from iso-8859-15, and then compare like so:

PS C:\Users\nicke> conv iso-8859-15 ibm01146 @(0..255) | Set-Content -Encoding byte iso-8859-15-ibm01146.cdx

PS C:\Users\nicke> conv iso-8859-15 ibm285 @(0..255) | Set-Content -Encoding byte iso-8859-15-ibm285.cdx

PS C:\Users\nicke> xlt_diff (cat -Encoding byte .\iso-8859-15-ibm01146.cdx) (cat -Encoding byte .\iso-8859-15-ibm285.cdx)
a4 : 9f | 6f

Here the translation tables differ in how they convert the Euro (€) symbol in iso-8859-15 (0xa4) to the two mainframe codepages.

This is not that surprising as ibm01146 has the Euro (€ 0x9f) and codepage ibm285 does not. In fact if you look up codepage 1146 on wikipedia you will see that ibm01146 was created to be ibm285 with the addition of the Euro (€) symbol.

I chose these two codepages as a simple example to showcase the finding the difference between translation tables.

These last two posts were about creating custom codepage translation tables for Connect:Direct, and spotting the differences between tables.

Next time we will look at displaying what maps to what more easily with these translation tables, and show a generally better way of translating from one codepage to another.

No Chit Chat, just Smalltalk

Day 20

This is something that you do not see every day.  Below is an extract from an old Smalltalk workspace listing I found where I was experimenting with Cincom's ObjectStudio Smalltalk programming environment and the OLE/COM interface to Connect:Direct.

It was just as easy if not easier than VBScript to control Connect:Direct using Smalltalk.

D := OLEDispatcher new: 'CD.NODE'.
N :=  D call: 'Connect' params: (Array with: 'MY.NODE' with: '' with: '').


TXT := 'TEST001 PROCESS'                           + CrLf
           + '     MAXDELAY=UNLIMITED'             + CrLf
           + '     REMOTE=CD.REMOTE'               + CrLf
           + '     HOLD = NO'                      + CrLf
           + 'STEP01 COPY  FROM ('                 + CrLf
           + '     FILE=C:\TEMP\INPUT.TXT '        + CrLf
           + '     LOCAL /*$Windows NT$*/)'        + CrLf
           + '     TO ('                           + CRLF
           + '     REMOTE /*$Windows NT$*/'        + CrLf
           + '     FILE=C:\TEMP\OUTPUT.TXT'        + CrLf
           + '     DISP=(RPL))'                    + CrLf
           + 'PEND'.

P := D call: 'Submit' params: (Array with: TXT).

P at: 'ProcessNumber'


S := D call: 'SelectStats' params: (Array with: 'select statistics pnumber=62').
I := S call: 'HasMore'.
I := S call: 'GetNext'.
I at: 'MsgId'

Smalltalk is one of my favourite programming languages, due to its' simple syntax and object orientation.

It might be interesting to see how this compares to using OLE/COM from TCL.  I'll save that for another post.

Self Sign

Day 19

If you're using SSL/TLS digital certificates with Connect:Direct Secure+ when connecting to your 3rd parties, you might well have a security policy that states that you will not connect with machines from 3rd parties that use self signed certificates.

You probably would prefer to connect with a 3rd party node that presents a digital certificate that is signed by someone you trust, such as VeriSign.

It might be that the history of the configuration of the Connect:Direct connection and the security policy did not coincide.

You may be in need of finding out which of your existing connections use self signed certificates.

As usual the information is within the Connect:Direct statistics, but not viewable from the "select statistics" command from  within the Connect:Direct command line.

The following function will check the statistics records piped through it to check which Secure+ connections use certificates that were signed by themselves.  In other words it checks statistics records where Secure+ was used and where the certificate issuer is the same as the subject of the server certificate used by the connection.

function selfsigncerts
{
grep RECI=CTRC | awk -F\| '
{
record["CSPE"]=""
record["CERI"]=""
record["CERT"]=""

for(i=1;i<=NF;i++)
{
key=substr($i,0,index($i,"=")-1)
value=substr($i,index($i,"=")+1)
record[key]=value
}

name = record["PNOD"] ":" record["SNOD"]

if((record["CSPE"] == "Y") && (record["CERI"] == record["CERT"]))
{
connections[name]=record["CERI"]
}
}
END{
for (name in connections)
{
print name ":" connections[name]
}
}'
}

It is used like this, assuming you are in the work directory where the statistics files are:

$ cat S20121224.001 | selfsigncerts
unx.node:OTHER.NODE:(/C=GB/L=Lincoln/O=Bank/OU=IT/CN=OTHER.NODE/emailAddress=joe.blogs@bank.co.uk/SN=12345678)

Now you know which connections use self signed certificates you can go about getting them replaced with certificates you can trust via your trusted 3rd party such as VeriSign.

Other things you could check for are the encryption algorithms used by a connection. Over time encryption algorithms lose favour as they are considered weaker than others.

Certificate signing algorithms also need checking for compliance with security policies.  For example the MD5 checksum algorithm in the past was used for signing certificates, but is considered weak, and has been shown that it can be exploited.

You may have a security policy that states you don't use certain algorithms, and you may have to demonstrate that you don't use them, and if you do, identify them for remediation.

The next few blog entries will cover these issues.

Nostalgia … NOT

Day 16



Today I was asked if I would dig out an example of automating a Connect:Direct transfer using a Windows batch script.  



I actually prefer to use WSH (Windows Script Host/VBScript) scripts for doing anything more complicated than a one line command when scripting with Connect:Direct on Windows.



I did find one however and have added a few comments to explain what it is doing.



send.bat

Least busiest time on the Connect:Direct node?

Day 15

Often when you need to make a change to a production Connect:Direct server,  you want to know when would be the best time to do it.

Sometimes you want to add additional transfers but want to schedule them at a time that does not put all the load at one time of the day.

Either way you can use the following short script to give you a histogram showing the activity on the local node to identify the best times for scheduling transfers or making configuration changes with as little impact to existing transfers.

I first did this for UNIX Connect:Direct nodes and was then asked if something similar could be done for Windows Connect:Direct nodes.  The Windows script for this used VBScript and an HTML application showing the histogram in HTML and the command line technique you have seen previously.  I’ll save that description for a later post.

This post will just address the UNIX solution to this problem.

You need to feed the “cdhours” shell function with the Connect:Direct UNIX statistics files for the period you are interested in.  To do this you probably want to be in the “work” directory for the local node for convenience where the statistics files are generated.

An example of how to use the script is given in the comment at the top of the script.

This script was the beginning of a collection of scripts to handle querying the Connect:Direct UNIX statistics/configuration files and the Secure+ configuration.

Next we will look at showing a histogram of the volume of data going through a Connect:Direct node ordered by remote node.  Useful for capacity planning and also billing.

---

# Connect:Direct Activity by hour Histogram
# =========================================
#
# Usage: cdhours
#
# $ cat S20100903.??? | cdhours
#
# To get a better picture for the month you could say
#
# $ cat S201009??.??? | cdhours
#
# Hours   Transfers
# =====   =========
#
# 00      54 #############
# 01      48 ############
# 02      54 #############
# 03     20 ##############################
# 04    244 ###############################################################
# 05      86 ######################
# 06      66 #################
# 07      76 ###################
# 08      36 #########
# 09       0
# 10      44 ###########
# 11    190 #################################################
# 12      48 ############
# 13      62 ################
# 14      30 #######
# 15      28 #######
# 16      85 #####################
# 17    221 #########################################################
# 18       0
# 19       0
# 20       0
# 21       0
# 22       0
# 23       0
#

function cdhours
{
       _FILES=$*
       # How many columns does the terminal have
       _COLS=`tput cols`
       # We are interested in the Copy Termination ReCords (CTRC)
       cat $_FILES | grep RECI=CTRC | nawk "{print $2}" | \
       nawk -v cols=$_COLS '
BEGIN{
       # Initialise the array that represents the histogram
       hours_per_day=24
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               tally[hour]=0
       }
}
       # This section gets executed for all records passed to nawk
       {
               split($2,fields,":")
               # Update the tally of these records that falls within this particular hour
               tally[fields[1]]++
       }
END{
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               # Keep track of the maximum number of tally marks so we can scale the histogram
               if(tally[hour] > max)
               {
                       max=tally[hour]
               }
       }
       node_name_length=16
       # Scale factor to make the histogram fit in the terminal window
       scale=(cols-(node_name_length+1))/max;

       printf("Hours\tTransfers\n")
       printf("=====\t=========\n\n")
       for(h=0;h < hours_per_day;h++)
       {
               # The keys of the array are packed with a leading zero
               hour=sprintf("%02d",h)
               # Scale the histogram bar to fit in terminal window
               bar=tally[hour]*scale
               printf "%3s\t%5d ",hour,tally[hour]
               # Generate the histogram bar
               for(b=1;b <= bar;b++)printf "#";
               printf "\n"
       }
}
'}

---

Difficult C:D questions?

Day 14


Connect:Direct records almost everything in the Connect:Direct statistics.  The statistics can be queried by using the Connect:Direct command line program “direct”, or more friendly programs such as the Connect:Direct Requester, or the browser interface, and even Sterling Control Center (SCC).

The statistics can be queried for information regarding a particular process or in general for errors and even for evidence of compliance to standards etc.

Some queries however are not possible to express in the Connect:Direct command line and difficult in the other tools mentioned earlier.

Some of these more difficult queries are listed below:

• What is the maximum number of concurrent sessions being used, at what time and which nodes had the lion share of the sessions?
• Which transfers are not secured using Secure+?
• Which nodes are using self signed digital certificates?
• What was the total volume of data transferred ordered by remote node?
• When is the least busiest time on this Connect:Direct node?
• Which remote nodes are triggering local scripts/processes?
• What are the transfers that have had a failure, but have not been successfully transmitted later?

The reason why these and other queries are difficult to express in the Connect:Direct command line is that the Connect:Direct statistics contain information that you can not get at with the Connect:Direct command line.

If you have ever taken a look at the Connect:Direct statistics files on UNIX you might not have liked what you saw:



STAR=20100902 17:00:03|PNAM=PULL|PNUM=98765|SSTA=20100902 17:00:03|STRT=20100902 17:00:03|STOP=20100902 17:00:03|STPT=20100902 17:00:03|SELA=00:00:00|SUBM=aaaacd@unx.aaaa|SBID=aaaacd|SBND=unx.aaaa|SNOD=CD.OTHER|CCOD=0|RECI=CTRC|RECC=CAPR|TZDI=3600|MSGI=SCPA000I|MSST=Copy step successful.
:
etc.



The above Connect:Direct statistic record is for a COPY statement within a Connect:Direct process.  It is just one long line with all the fields separated by the ‘|’ character.  Each field contains the 4 character name for the field name, an equals sign followed by the value of that field.

The 4 letter field names are documented in the “Connect:Direct for UNIX User Guide”.  You do not need to know them all.  Just use the ones you need when you need them.



PNAM is Process Name
PNUM is Process Number
PNOD is Primary Node Name
SNOD is Secondary Node Name
CCOD is Condition Code
SFIL is Source File Name
DFIL is Destination File Name
DBYW is Destination Bytes Written



A simple UNIX command can help make these statistics files easier to read

$ cat S20100902.047 | grep RECI=CTRC | tr '|' '\n'

The grep for records that contain the string “RECI=CTRC” filters just those records that are Copy Termination ReCords i.e. produced by a COPY statement.

Which produces something like the following:



STAR=20100902 17:00:03
PNAM=PULL
PNUM=98765
:
SUBM=aaaacd@unx.aaaa
:
SNOD=CD.OTHER
CCOD=0
RECI=CTRC
:
MSGI=SCPA000I
MSST=Copy step successful.
:
PNOD=unx.aaaa
SNOD=CD.OTHER
LNOD=P
:
CSPE=Y
CSPP=TLSv1
CSPS=TLS_RSA_WITH_AES_256_CBC_SHA
CERT=(/C=GB/ST=Cheshire/L=Congleton/O=A Global Financial Institution Plc/OU=Middleware/CN=CD.OTHER/SN=69009876789765456787654567667729)
CERI=(/O=Trusted Network/OU=Trusted, Inc./OU=Trusted CA/OU=www.trusted.org/SN=78ee48de185b2071c9c9c3b51d7bddc1)
SFIL=\share123\outgoing\AAAAAA.123456.DAT
:
DFIL=/data/projectx/from_agfi/AAAAAA.123456.DAT
:
DBYW=1161
:
etc.



Now we know the format of the stats records and we have an easier way to view them we can write some shell functions to help us with some of our tasks with Connect:Direct on UNIX and even with these more difficult questions.

Next we will look at one of those questions and how it can be answered in more detail.