Day 24 The tide is high

Day 24

The tide is high

Let us say that you do not have Sterling Control Center to run the High Water Mark report for maximum concurrent sessions.

You might want this for determining if you are making optimum use of the number of concurrent sessions that you are licensed for.

You may not have enough Connect:Direct nodes that you feel justifies the need for Sterling Control Center.

For Windows

The following script water-mark.vbs is an example of how to do this using VBScript and the Connect:Direct Windows SDK.

In fact this is a fairly simple example of how to use the Connect:Direct Windows SDK.

'
' Script to report on the high water mark for the number of concurrent Connect:Direct sessions 
'

Dim node        ' Represents the Connect Direct Node we are using. 
Dim stats       ' A collection of statistic records. 
Dim stat        ' An individual statistic record. 

' Make the OLE/COM connection to Connect:Direct 
Set node = CreateObject("CD.Node")

' Sign on using defaults from the Client Connection Utility
node.Connect "CD.NICKE","",""

' Get start and end sessions statistics records since yesterday
set stats = node.SelectStats("select statistics startt=(today) recids=(SSTR,SEND) ccode=(eq,0)")

currentNumberOfSessions = 0
highWaterMark  = 0

For Each stat in stats
       Select Case stat.RecId 
          Case "SSTR" currentNumberOfSessions=currentNumberOfSessions+1          Case "SEND" currentNumberOfSessions=currentNumberOfSessions-1       End Select
       If currentNumberOfSessions > highWaterMark Then
          highWaterMark = currentNumberOfSessions
       End If
Next

Wscript.echo "Concurrent sessions high water mark = " & highWaterMark

Set node  = nothing
Set stats = nothing
Set stat  = nothing

If you have the Connect:Direct Windows SDK installed and default sign on credentials registered with the Client Connection Utility you can run the script as follows:

C:\Users\nicke> cscript /nologo water-mark.vbs

Concurrent sessions high water mark = 5

The above script is minimal in that it is only meant to be run interactively as it does not itself check for every error.

You run the script on a Windows machine with the SDK installed, but the node in question can be a remote Windows or UNIX (I have not tested other platforms) Connect:Direct node as long as the node is registered with the Client Connection Utility.

For UNIX

If all you have are UNIX machines then you can use the following shell function for convenience:

function watermark
{
        # Usage: cd work/UNIX.NODE ; cat S201712* | watermark
        egrep "RECI=(SSTR|SEND)" | grep CCOD=0 | awk '
BEGIN{
        currentNumberOfSessions=0
        highWaterMark=0
}
/RECI=SSTR/ { currentNumberOfSessions++ }
/RECI=SEND/ { currentNumberOfSessions-- }
{
        if(currentNumberOfSessions > highWaterMark){
                highWaterMark=currentNumberOfSessions
        }
}
END{
        print "Concurrent sessions high water mark = " highWaterMark
}'
}

Put the above shell function definition either in your .profile or just paste it into a terminal session. You will need to be in the work directory for your UNIX node where the statistics files are. Then you can run it as follows by piping whatever period of statistics files you want through the watermark shell function:

[work/CD.UNIX] $ cat S201712* | watermark
Concurrent sessions high water mark = 8

Now you will know whether the number of sessions you use is appropriate for your licensing of these nodes.

generated by haroopad

Day 23 Translation what's the difference

Day 23

Translation what’s the difference

First let’s create some Connect:Direct translation tables using PowerShell:

PS C:\Users\nicke> conv ibm285 iso-8859-15 @(0..255) | Set-Content -Encoding byte ibm285-iso-8859-15.cdx

PS C:\Users\nicke> conv ibm01146 iso-8859-15 @(0..255) | Set-Content -Encoding byte ibm01146-iso-8859-15.cdx

Here we are converting from codepage ibm285 IBM EBCDIC (UK) to iso-8859-15 which has the Euro currency symbol, and converting all the byte values from 0 through to 255 (that is what the @(0..255) means), saving the result with the Connect:Direct Windows translation table file extension .cdx.

We then do the same thing for producing a translation table that will convert from IBM EBCDIC (UK-Euro) to iso-8859-15 .

Using the Powershell function below we can see the difference between these two translation tables.

# List difference between translation tables
function xlt_diff ([byte[]]$tbla,[byte[]]$tblb) {
    0..255 | %{
        if($tbla[$_] -ne $tblb[$_]) {
            "{0:x} : {1:x} | {2:x}" -f $_,$tbla[$_],$tblb[$_]
        }
    }
}

The above functions can be used as follows:

PS C:\Users\nicke> xlt_diff (cat -Encoding byte .\ibm285-iso-8859-15.cdx) (cat -Encoding byte .\ibm01146-iso-8859-15.cdx)
9f : 3f | a4

The output above shows that two translation tables differ when they map hex byte value 0x9f. In the first table it maps to hex value 0x3f, and in the other to 0xa4.
Now if we create the translation tables for translating back to either ibm285/ibm01146 from iso-8859-15, and then compare like so:

PS C:\Users\nicke> conv iso-8859-15 ibm01146 @(0..255) | Set-Content -Encoding byte iso-8859-15-ibm01146.cdx

PS C:\Users\nicke> conv iso-8859-15 ibm285 @(0..255) | Set-Content -Encoding byte iso-8859-15-ibm285.cdx

PS C:\Users\nicke> xlt_diff (cat -Encoding byte .\iso-8859-15-ibm01146.cdx) (cat -Encoding byte .\iso-8859-15-ibm285.cdx)
a4 : 9f | 6f

Here the translation tables differ in how they convert the Euro (€) symbol in iso-8859-15 (0xa4) to the two mainframe codepages.

This is not that surprising as ibm01146 has the Euro (€ 0x9f) and codepage ibm285 does not. In fact if you look up codepage 1146 on wikipedia you will see that ibm01146 was created to be ibm285 with the addition of the Euro (€) symbol.

I chose these two codepages as a simple example to showcase the finding the difference between translation tables.

These last two posts were about creating custom codepage translation tables for Connect:Direct, and spotting the differences between tables.

Next time we will look at displaying what maps to what more easily with these translation tables, and show a generally better way of translating from one codepage to another.

Power Translation

Day 22

Power Translation

While on assignment some time ago the only scripting language I had available to me was PowerShell. The PowerShell turned out to be a very useful tool. Today I will share some of its features that helped me with Connect:Direct custom translation tables.

I needed help in understanding the translation of certain characters between the Mainframe and Windows platforms, and understanding quickly what was different between a customised translation table, and the default table on a Windows machine.

I also wanted to experiment with the translation tables without actually always being on the machine that had Connect:Direct on it.

My solution was a set of PowerShell helper functions that allowed me to examine, compare, generate and test Connect:Direct translation tables without Connect:Direct necessarily.

Here are some simple PowerShell functions to illustrate. Please be aware that I have deliberately kept these minimal for brevity.

# List all codepages
function lsenc () {
    [System.Text.Encoding]::GetEncodings()
}

# Get an object representing the codepage
function getenc ($str) {
    [System.Text.Encoding]::GetEncoding($str)
}

# Simple filter that displays bytes as hex values
function hex {
    $input | %{ write-host -NoNewline ("{0:x2} " -f $_)}
    ""
}

# Converts a byte buffer to different codepage
function conv ($from,$to,$buf,$str=$false) {
    $from_enc=getenc $from
    $to_enc=getenc $to
    if($buf.gettype().BaseType.Name -ne "Array") {
        [System.Text.Encoding]::Convert($from_enc,$to_enc,$from_enc.getbytes($buf))
    } else {
        [System.Text.Encoding]::Convert($from_enc,$to_enc,$buf)
    }
}

The above functions can be used as follows:

# Test for well known EBCDIC value
PS C:\Users\nicke> conv 1252 37 " " | hex
40 
# Test for well known ASCII value
PS C:\Users\nicke> conv 37 1252 @(0x40) | hex
20 
# hex value for £ in Windows
PS C:\Users\nicke> conv 1252 1252 "£" | hex
a3 
# hex value for £ UTF-8
PS C:\Users\nicke> conv 1252 utf-8 "£" | hex
c2 a3 
# hex value for £ in UTF-16
PS C:\Users\nicke> conv 1252 utf-16 "£" | hex
a3 00 
# hex value of £ in IBM EBCDIC (UK-Euro)
PS C:\Users\nicke> conv 1252 1146 "£" | hex
5b

You can also do the same thing with longer strings and even contents of files.

So if I have a file that is encoded using the mainframe codepage IBM-1146 like so:

PS C:\Users\nicke> (cat -Encoding Byte ibm1146.1146) | hex
c9 c2 d4 60 f1 f1 f4 f6 

PS C:\Users\nicke>

I can translate it to Windows 1252 like so:

PS C:\Users\nicke> conv 1146 1252 (cat -Encoding Byte ibm1146.1146) | Set-Content -Encoding byte ibm1146.1252

PS C:\Users\nicke> type ibm1146.1252

IBM-1146

PS C:\Users\nicke>

So as you can see the 3rd paremeter to the conv function can be a string, a byte array, or the contents of a file which is then converted to a byte array.

I wouldn’t use this for large files, but with small files just to help understand the codepages and their translation between them.

Next time we will look at actual Connect:Direct translation tables, and how to create custom translation tables easily with some Powershell functions.

Limitations of templating FileAgent configs

Limitations of templating FileAgent configs

Day 21

I received the following question regarding building FileAgent .ser configuration files from templates:

" I know that I can write certain parameters into a .ser file by writing a .cfg file and running cdfa -g . What I don't know is what the limitations and possibilities of the .cfg file are. Ideally I'd like to be able to write a rule into a .ser file through the .cfg file, but I can't find any documentation as to whether or not that's possible. Do you have any insight?"

As far as I know you can only pass in substitution variables such as "&var." .

When you want the substitution variable expanded to its value you need the trailing period, but when you are editing the build template config file you just use "&var=value".

Also you can only use these substitution variables with text/string fields and not numeric fields such as the port numbers used in the FileAgent configuration.

You can not use the build configuration templates to add rules to the binary .ser configuration file.

Those would be nice features to have added to FileAgent.

No Chit Chat, just Smalltalk

Day 20

This is something that you do not see every day.  Below is an extract from an old Smalltalk workspace listing I found where I was experimenting with Cincom's ObjectStudio Smalltalk programming environment and the OLE/COM interface to Connect:Direct.

It was just as easy if not easier than VBScript to control Connect:Direct using Smalltalk.

D := OLEDispatcher new: 'CD.NODE'.
N :=  D call: 'Connect' params: (Array with: 'MY.NODE' with: '' with: '').


TXT := 'TEST001 PROCESS'                           + CrLf
           + '     MAXDELAY=UNLIMITED'             + CrLf
           + '     REMOTE=CD.REMOTE'               + CrLf
           + '     HOLD = NO'                      + CrLf
           + 'STEP01 COPY  FROM ('                 + CrLf
           + '     FILE=C:\TEMP\INPUT.TXT '        + CrLf
           + '     LOCAL /*$Windows NT$*/)'        + CrLf
           + '     TO ('                           + CRLF
           + '     REMOTE /*$Windows NT$*/'        + CrLf
           + '     FILE=C:\TEMP\OUTPUT.TXT'        + CrLf
           + '     DISP=(RPL))'                    + CrLf
           + 'PEND'.

P := D call: 'Submit' params: (Array with: TXT).

P at: 'ProcessNumber'


S := D call: 'SelectStats' params: (Array with: 'select statistics pnumber=62').
I := S call: 'HasMore'.
I := S call: 'GetNext'.
I at: 'MsgId'

Smalltalk is one of my favourite programming languages, due to its' simple syntax and object orientation.

It might be interesting to see how this compares to using OLE/COM from TCL.  I'll save that for another post.

Self Sign

Day 19

If you're using SSL/TLS digital certificates with Connect:Direct Secure+ when connecting to your 3rd parties, you might well have a security policy that states that you will not connect with machines from 3rd parties that use self signed certificates.

You probably would prefer to connect with a 3rd party node that presents a digital certificate that is signed by someone you trust, such as VeriSign.

It might be that the history of the configuration of the Connect:Direct connection and the security policy did not coincide.

You may be in need of finding out which of your existing connections use self signed certificates.

As usual the information is within the Connect:Direct statistics, but not viewable from the "select statistics" command from  within the Connect:Direct command line.

The following function will check the statistics records piped through it to check which Secure+ connections use certificates that were signed by themselves.  In other words it checks statistics records where Secure+ was used and where the certificate issuer is the same as the subject of the server certificate used by the connection.

function selfsigncerts
{
grep RECI=CTRC | awk -F\| '
{
record["CSPE"]=""
record["CERI"]=""
record["CERT"]=""

for(i=1;i<=NF;i++)
{
key=substr($i,0,index($i,"=")-1)
value=substr($i,index($i,"=")+1)
record[key]=value
}

name = record["PNOD"] ":" record["SNOD"]

if((record["CSPE"] == "Y") && (record["CERI"] == record["CERT"]))
{
connections[name]=record["CERI"]
}
}
END{
for (name in connections)
{
print name ":" connections[name]
}
}'
}

It is used like this, assuming you are in the work directory where the statistics files are:

$ cat S20121224.001 | selfsigncerts
unx.node:OTHER.NODE:(/C=GB/L=Lincoln/O=Bank/OU=IT/CN=OTHER.NODE/emailAddress=joe.blogs@bank.co.uk/SN=12345678)

Now you know which connections use self signed certificates you can go about getting them replaced with certificates you can trust via your trusted 3rd party such as VeriSign.

Other things you could check for are the encryption algorithms used by a connection. Over time encryption algorithms lose favour as they are considered weaker than others.

Certificate signing algorithms also need checking for compliance with security policies.  For example the MD5 checksum algorithm in the past was used for signing certificates, but is considered weak, and has been shown that it can be exploited.

You may have a security policy that states you don't use certain algorithms, and you may have to demonstrate that you don't use them, and if you do, identify them for remediation.

The next few blog entries will cover these issues.

Mistaken Identity

Day 18 

Most problems with translation tables amount to a case of mistaken identity. Sometimes it is the source of the file that is assumed to be something, but turns out to be something else.

For example someone says they are having problems transferring a file from a VMS system to UNIX and the Excel spreadsheet is not arriving in the correct format.

Well in this case you can not just look at this problem from a VMS/UNIX perspective. The Excel spreadsheet probably originated from a Windows machine. So how was it transferred to the VMS machine? Was it transferred in binary mode? Is the spreadsheet file really an Excel file, or just a .csv file?

The answers to those questions have an impact on the problem and its' solution. If the file was truely an Excel spreadsheet then you would want to transfer it in binary mode so the file ends up at its' destination literally the same as at the source of the transfer, no matter how many hops there are in the transfer.

It all depends on what is being used to produce the file to be transferred and what will end up consuming/processing the file at the destination. In the case of an Excel spreadsheet it will be a piece of software expecting a file the same as would be on a Windows machine, hence the binary mode.

If the source of the transfer was . csv file (comma separated values), i.e. a text file and was to be consumed/processed on a UNIX machine by an application, then we would want the file to arrive on the UNIX machine as a UNIX text file with each line terminated with a newline character as opposed to a carriage-return and newline characters on a Windows platform.

For this to happen we want Connect:Direct to treat the file as a text file and not binary as in the previous example. So we would not specify DATATYPE binary as before but use the default DATATYPE which is TEXT.

Some times you are told which codepages are being used on both ends of the proposed Connect:Direct transfer with absolute certainty.

For example you be told that a text file transferred from a mainframe was produced using codepage IBM-1140 and that the application on a Windows machine receiving the file is using UTF-8, an encoding for Unicode.

It really does depend on the application that will consume/process the file. It might be assumed that the application can handle UTF-8, or that as ASCII is a subset of UTF-8 there should be no problem.

In this case an international character might be used within the file on the mainframe that is available to it within the IBM-1140 codepage and this will be translated to the corresponding UTF-8 encoding of Unicode.

For characters that map directly to a single ASCII/UTF-8 there will not be a problem, but international characters can be encoded as 1,2,3 or even 4 byte UTF-8 encoded Unicode characters.

This is because UTF-8 is a variable byte character encoding. If the application is written to use Windows codepage CP-1252, then it will only be expecting single byte characters and not multi-byte Unicode characters that UTF-8 can encode. It will then probably choke on the multi-byte encoding or just not recognise what it is supposed to represent, and not process the file properly.

Imagine that data entered into an application on the mainframe is using one codepage, but the application was programmed to use a field delimiter character from another codepage. The file the application produces on the mainframe will contain data from one codepage and delimiters from another, and then transferred to another machine with codepage translation specified for the destination.

You may not be surprised to find that the field delimiter characters were not translated correctly for the destination.

In this particular case I suggested that the application programmer on the mainframe use a particular hex value character for the field delimiter that was available within the IBM-1140 codepage, and it turned out that the application on the Windows machine was using CP-1252 and not UTF-8.

It turned out for this particular file and applications that there were no special codepage requirements, as the default translation was sufficient.

So next time some one is enfatically, absolutely certain about codepage requirements, it might just be a good idea to check the facts for yourself, as it is easy for people to get this wrong.