Day 24 The tide is high

Day 24

The tide is high

Let us say that you do not have Sterling Control Center to run the High Water Mark report for maximum concurrent sessions.

You might want this for determining if you are making optimum use of the number of concurrent sessions that you are licensed for.

You may not have enough Connect:Direct nodes that you feel justifies the need for Sterling Control Center.

For Windows

The following script water-mark.vbs is an example of how to do this using VBScript and the Connect:Direct Windows SDK.

In fact this is a fairly simple example of how to use the Connect:Direct Windows SDK.

'
' Script to report on the high water mark for the number of concurrent Connect:Direct sessions 
'

Dim node        ' Represents the Connect Direct Node we are using. 
Dim stats       ' A collection of statistic records. 
Dim stat        ' An individual statistic record. 

' Make the OLE/COM connection to Connect:Direct 
Set node = CreateObject("CD.Node")

' Sign on using defaults from the Client Connection Utility
node.Connect "CD.NICKE","",""

' Get start and end sessions statistics records since yesterday
set stats = node.SelectStats("select statistics startt=(today) recids=(SSTR,SEND) ccode=(eq,0)")

currentNumberOfSessions = 0
highWaterMark  = 0

For Each stat in stats
       Select Case stat.RecId 
          Case "SSTR" currentNumberOfSessions=currentNumberOfSessions+1          Case "SEND" currentNumberOfSessions=currentNumberOfSessions-1       End Select
       If currentNumberOfSessions > highWaterMark Then
          highWaterMark = currentNumberOfSessions
       End If
Next

Wscript.echo "Concurrent sessions high water mark = " & highWaterMark

Set node  = nothing
Set stats = nothing
Set stat  = nothing

If you have the Connect:Direct Windows SDK installed and default sign on credentials registered with the Client Connection Utility you can run the script as follows:

C:\Users\nicke> cscript /nologo water-mark.vbs

Concurrent sessions high water mark = 5

The above script is minimal in that it is only meant to be run interactively as it does not itself check for every error.

You run the script on a Windows machine with the SDK installed, but the node in question can be a remote Windows or UNIX (I have not tested other platforms) Connect:Direct node as long as the node is registered with the Client Connection Utility.

For UNIX

If all you have are UNIX machines then you can use the following shell function for convenience:

function watermark
{
        # Usage: cd work/UNIX.NODE ; cat S201712* | watermark
        egrep "RECI=(SSTR|SEND)" | grep CCOD=0 | awk '
BEGIN{
        currentNumberOfSessions=0
        highWaterMark=0
}
/RECI=SSTR/ { currentNumberOfSessions++ }
/RECI=SEND/ { currentNumberOfSessions-- }
{
        if(currentNumberOfSessions > highWaterMark){
                highWaterMark=currentNumberOfSessions
        }
}
END{
        print "Concurrent sessions high water mark = " highWaterMark
}'
}

Put the above shell function definition either in your .profile or just paste it into a terminal session. You will need to be in the work directory for your UNIX node where the statistics files are. Then you can run it as follows by piping whatever period of statistics files you want through the watermark shell function:

[work/CD.UNIX] $ cat S201712* | watermark
Concurrent sessions high water mark = 8

Now you will know whether the number of sessions you use is appropriate for your licensing of these nodes.

generated by haroopad

Day 23 Translation what's the difference

Day 23

Translation what’s the difference

First let’s create some Connect:Direct translation tables using PowerShell:

PS C:\Users\nicke> conv ibm285 iso-8859-15 @(0..255) | Set-Content -Encoding byte ibm285-iso-8859-15.cdx

PS C:\Users\nicke> conv ibm01146 iso-8859-15 @(0..255) | Set-Content -Encoding byte ibm01146-iso-8859-15.cdx

Here we are converting from codepage ibm285 IBM EBCDIC (UK) to iso-8859-15 which has the Euro currency symbol, and converting all the byte values from 0 through to 255 (that is what the @(0..255) means), saving the result with the Connect:Direct Windows translation table file extension .cdx.

We then do the same thing for producing a translation table that will convert from IBM EBCDIC (UK-Euro) to iso-8859-15 .

Using the Powershell function below we can see the difference between these two translation tables.

# List difference between translation tables
function xlt_diff ([byte[]]$tbla,[byte[]]$tblb) {
    0..255 | %{
        if($tbla[$_] -ne $tblb[$_]) {
            "{0:x} : {1:x} | {2:x}" -f $_,$tbla[$_],$tblb[$_]
        }
    }
}

The above functions can be used as follows:

PS C:\Users\nicke> xlt_diff (cat -Encoding byte .\ibm285-iso-8859-15.cdx) (cat -Encoding byte .\ibm01146-iso-8859-15.cdx)
9f : 3f | a4

The output above shows that two translation tables differ when they map hex byte value 0x9f. In the first table it maps to hex value 0x3f, and in the other to 0xa4.
Now if we create the translation tables for translating back to either ibm285/ibm01146 from iso-8859-15, and then compare like so:

PS C:\Users\nicke> conv iso-8859-15 ibm01146 @(0..255) | Set-Content -Encoding byte iso-8859-15-ibm01146.cdx

PS C:\Users\nicke> conv iso-8859-15 ibm285 @(0..255) | Set-Content -Encoding byte iso-8859-15-ibm285.cdx

PS C:\Users\nicke> xlt_diff (cat -Encoding byte .\iso-8859-15-ibm01146.cdx) (cat -Encoding byte .\iso-8859-15-ibm285.cdx)
a4 : 9f | 6f

Here the translation tables differ in how they convert the Euro (€) symbol in iso-8859-15 (0xa4) to the two mainframe codepages.

This is not that surprising as ibm01146 has the Euro (€ 0x9f) and codepage ibm285 does not. In fact if you look up codepage 1146 on wikipedia you will see that ibm01146 was created to be ibm285 with the addition of the Euro (€) symbol.

I chose these two codepages as a simple example to showcase the finding the difference between translation tables.

These last two posts were about creating custom codepage translation tables for Connect:Direct, and spotting the differences between tables.

Next time we will look at displaying what maps to what more easily with these translation tables, and show a generally better way of translating from one codepage to another.

Power Translation

Day 22

Power Translation

While on assignment some time ago the only scripting language I had available to me was PowerShell. The PowerShell turned out to be a very useful tool. Today I will share some of its features that helped me with Connect:Direct custom translation tables.

I needed help in understanding the translation of certain characters between the Mainframe and Windows platforms, and understanding quickly what was different between a customised translation table, and the default table on a Windows machine.

I also wanted to experiment with the translation tables without actually always being on the machine that had Connect:Direct on it.

My solution was a set of PowerShell helper functions that allowed me to examine, compare, generate and test Connect:Direct translation tables without Connect:Direct necessarily.

Here are some simple PowerShell functions to illustrate. Please be aware that I have deliberately kept these minimal for brevity.

# List all codepages
function lsenc () {
    [System.Text.Encoding]::GetEncodings()
}

# Get an object representing the codepage
function getenc ($str) {
    [System.Text.Encoding]::GetEncoding($str)
}

# Simple filter that displays bytes as hex values
function hex {
    $input | %{ write-host -NoNewline ("{0:x2} " -f $_)}
    ""
}

# Converts a byte buffer to different codepage
function conv ($from,$to,$buf,$str=$false) {
    $from_enc=getenc $from
    $to_enc=getenc $to
    if($buf.gettype().BaseType.Name -ne "Array") {
        [System.Text.Encoding]::Convert($from_enc,$to_enc,$from_enc.getbytes($buf))
    } else {
        [System.Text.Encoding]::Convert($from_enc,$to_enc,$buf)
    }
}

The above functions can be used as follows:

# Test for well known EBCDIC value
PS C:\Users\nicke> conv 1252 37 " " | hex
40 
# Test for well known ASCII value
PS C:\Users\nicke> conv 37 1252 @(0x40) | hex
20 
# hex value for £ in Windows
PS C:\Users\nicke> conv 1252 1252 "£" | hex
a3 
# hex value for £ UTF-8
PS C:\Users\nicke> conv 1252 utf-8 "£" | hex
c2 a3 
# hex value for £ in UTF-16
PS C:\Users\nicke> conv 1252 utf-16 "£" | hex
a3 00 
# hex value of £ in IBM EBCDIC (UK-Euro)
PS C:\Users\nicke> conv 1252 1146 "£" | hex
5b

You can also do the same thing with longer strings and even contents of files.

So if I have a file that is encoded using the mainframe codepage IBM-1146 like so:

PS C:\Users\nicke> (cat -Encoding Byte ibm1146.1146) | hex
c9 c2 d4 60 f1 f1 f4 f6 

PS C:\Users\nicke>

I can translate it to Windows 1252 like so:

PS C:\Users\nicke> conv 1146 1252 (cat -Encoding Byte ibm1146.1146) | Set-Content -Encoding byte ibm1146.1252

PS C:\Users\nicke> type ibm1146.1252

IBM-1146

PS C:\Users\nicke>

So as you can see the 3rd paremeter to the conv function can be a string, a byte array, or the contents of a file which is then converted to a byte array.

I wouldn’t use this for large files, but with small files just to help understand the codepages and their translation between them.

Next time we will look at actual Connect:Direct translation tables, and how to create custom translation tables easily with some Powershell functions.

No Chit Chat, just Smalltalk

Day 20

This is something that you do not see every day.  Below is an extract from an old Smalltalk workspace listing I found where I was experimenting with Cincom's ObjectStudio Smalltalk programming environment and the OLE/COM interface to Connect:Direct.

It was just as easy if not easier than VBScript to control Connect:Direct using Smalltalk.

D := OLEDispatcher new: 'CD.NODE'.
N :=  D call: 'Connect' params: (Array with: 'MY.NODE' with: '' with: '').


TXT := 'TEST001 PROCESS'                           + CrLf
           + '     MAXDELAY=UNLIMITED'             + CrLf
           + '     REMOTE=CD.REMOTE'               + CrLf
           + '     HOLD = NO'                      + CrLf
           + 'STEP01 COPY  FROM ('                 + CrLf
           + '     FILE=C:\TEMP\INPUT.TXT '        + CrLf
           + '     LOCAL /*$Windows NT$*/)'        + CrLf
           + '     TO ('                           + CRLF
           + '     REMOTE /*$Windows NT$*/'        + CrLf
           + '     FILE=C:\TEMP\OUTPUT.TXT'        + CrLf
           + '     DISP=(RPL))'                    + CrLf
           + 'PEND'.

P := D call: 'Submit' params: (Array with: TXT).

P at: 'ProcessNumber'


S := D call: 'SelectStats' params: (Array with: 'select statistics pnumber=62').
I := S call: 'HasMore'.
I := S call: 'GetNext'.
I at: 'MsgId'

Smalltalk is one of my favourite programming languages, due to its' simple syntax and object orientation.

It might be interesting to see how this compares to using OLE/COM from TCL.  I'll save that for another post.

Mistaken Identity

Day 18 

Most problems with translation tables amount to a case of mistaken identity. Sometimes it is the source of the file that is assumed to be something, but turns out to be something else.

For example someone says they are having problems transferring a file from a VMS system to UNIX and the Excel spreadsheet is not arriving in the correct format.

Well in this case you can not just look at this problem from a VMS/UNIX perspective. The Excel spreadsheet probably originated from a Windows machine. So how was it transferred to the VMS machine? Was it transferred in binary mode? Is the spreadsheet file really an Excel file, or just a .csv file?

The answers to those questions have an impact on the problem and its' solution. If the file was truely an Excel spreadsheet then you would want to transfer it in binary mode so the file ends up at its' destination literally the same as at the source of the transfer, no matter how many hops there are in the transfer.

It all depends on what is being used to produce the file to be transferred and what will end up consuming/processing the file at the destination. In the case of an Excel spreadsheet it will be a piece of software expecting a file the same as would be on a Windows machine, hence the binary mode.

If the source of the transfer was . csv file (comma separated values), i.e. a text file and was to be consumed/processed on a UNIX machine by an application, then we would want the file to arrive on the UNIX machine as a UNIX text file with each line terminated with a newline character as opposed to a carriage-return and newline characters on a Windows platform.

For this to happen we want Connect:Direct to treat the file as a text file and not binary as in the previous example. So we would not specify DATATYPE binary as before but use the default DATATYPE which is TEXT.

Some times you are told which codepages are being used on both ends of the proposed Connect:Direct transfer with absolute certainty.

For example you be told that a text file transferred from a mainframe was produced using codepage IBM-1140 and that the application on a Windows machine receiving the file is using UTF-8, an encoding for Unicode.

It really does depend on the application that will consume/process the file. It might be assumed that the application can handle UTF-8, or that as ASCII is a subset of UTF-8 there should be no problem.

In this case an international character might be used within the file on the mainframe that is available to it within the IBM-1140 codepage and this will be translated to the corresponding UTF-8 encoding of Unicode.

For characters that map directly to a single ASCII/UTF-8 there will not be a problem, but international characters can be encoded as 1,2,3 or even 4 byte UTF-8 encoded Unicode characters.

This is because UTF-8 is a variable byte character encoding. If the application is written to use Windows codepage CP-1252, then it will only be expecting single byte characters and not multi-byte Unicode characters that UTF-8 can encode. It will then probably choke on the multi-byte encoding or just not recognise what it is supposed to represent, and not process the file properly.

Imagine that data entered into an application on the mainframe is using one codepage, but the application was programmed to use a field delimiter character from another codepage. The file the application produces on the mainframe will contain data from one codepage and delimiters from another, and then transferred to another machine with codepage translation specified for the destination.

You may not be surprised to find that the field delimiter characters were not translated correctly for the destination.

In this particular case I suggested that the application programmer on the mainframe use a particular hex value character for the field delimiter that was available within the IBM-1140 codepage, and it turned out that the application on the Windows machine was using CP-1252 and not UTF-8.

It turned out for this particular file and applications that there were no special codepage requirements, as the default translation was sufficient.

So next time some one is enfatically, absolutely certain about codepage requirements, it might just be a good idea to check the facts for yourself, as it is easy for people to get this wrong.

Nostalgia … NOT

Day 16



Today I was asked if I would dig out an example of automating a Connect:Direct transfer using a Windows batch script.  



I actually prefer to use WSH (Windows Script Host/VBScript) scripts for doing anything more complicated than a one line command when scripting with Connect:Direct on Windows.



I did find one however and have added a few comments to explain what it is doing.



send.bat

The other way around

Day 7

For my brother Lee he only needed to use his connection in one direction, from his Windows node to the remote UNIX node.

If he had needed to do it the other way around this is what I would have told him.

The concepts are the same, just presented differently as I said previously. We already have the netmap configuration, but we will need to make a configuration to allow the remote UNIX machine to send files to our local node.

The files that are received by our local node from the CD.REMOTE will have to be owned by a user account on our local node.  We could just use any local user account but it would be better to create a user specifically for this purpose and lock down the directories as mentioned previously when we set up the UNIX side of the configuration.

First let's set up Connect:Direct to allow a local user appusr1 to work with Connect:Direct.  Double-click the "User Authorities" in the left pane of the Connect:Direct Requester.  As you can see in the image below there are some entries that start with an asterisk which indicates they are templates.  In our case we are defining appusr1 as a non-administrative user of Connect:Direct.

Click the "New Genuser" and fill in as below. The controls will inherit from the templates you saw earlier and of course you can override those values here if you desire.

So how do we associate a incoming remote transfer with the local user we want to own the files?  This is done using the proxy definition.  Double-click "Proxies" in the left hand pane of the Requester and enter the fields as below.

So we are saying here that the remote user appusr1 from the remote node CD.REMOTE will be mapped to the local user appusr1.  It doesn't have to be the same name as I have done here.  Normally I create user accounts that reflect the name of the application that will consume the file if a suitable user account doesn't already exist.

If we didn't define a proxy for the remote Connect:Direct user, the remote Connect:Direct process would have to supply a SNODEID containing a local user account name on our local node together with a password.

This is not good for several reasons.  It is not a good idea to advertise local account information to anyone outside of your organisation.  Also hard coding passwords in scripts of any kind is also not a good idea because they might be seen, and if you have an information security department they will want you to change your passwords frequently which would mean amending all your scripts that contain passwords frequently.

The nice thing about Connect:Direct proxies is that the remote node doesn't care about what local user account you might be using, and they do not have to supply a password.  If their user is authorised to use Connect:Direct on the remote node they do not need to do anything regarding the user being used as you have covered that in the proxy definition.

For my brother's connection they were not using Secure+ to encrypt the data in transit or use it to authenticate using digital certificates.  If you read my post "Secure IT" then you will know that I always use Secure+.  In a later post I will give an example of doing just that.