Table of Contents

VMFPLC2 Case Study

One of the biggest limitations of working with VM/370 under hercules is the asymmetry in how CMS stores files, and how modern operating systems store files.

The namespace disparities present some hurdles in taking a 'host viewable' backup of the system.

VMFPLC2

The VM/370 Utility VMFPLC2 can be used to dump collections of files onto tape. Because tapes are of arbitrarily great lengths in the virtual world, it's an easy medium on which to transport files. It also turns out that there is a parallel utility created for the HERCULES project which can “unpack” these tape formats and transfer them from and to the standard format.

This case study covers the round-trip required to move files from CMS MDisks, to Virtual Tape, to Windows Host, to Virtual Tape, and then to CMS MDisks.

CMS to Virtual Tape

Dumping arbitrary files from CMS is trivial. In broad steps

  1. Attach a virtual file on a HERCULES emulated tape drive
  2. Attach the corresponding virtual tape drive on VM/370 Control Program to the USER's Tape Drive (CMS Uses 181 as the first tape drive).
  3. Issue the VMFPLC2 DUMP fn ft fm command
  4. Write two consecutive tape marks to signify end of tape.
  5. Detach the emulated tape drive from the virtual machine
  6. Detach the attached file from the HERCULES emulated tape drive.

Virtual Tape to Windows Host

This is where the fun begins. The HERCULES HOST version of VMFPLC2 doesn't operate exactly the same way as the VM/370 CMS version of the utility. This is because of the dissimilarities in how the files are named and a control file is employed to act as the bridge between the CMS naming and formatting of files.

Scanning is straightforward, but building the control file, is not.

  1. The VMFPLC2 command does a great job of producing a list of the tape's contents… but that's where the fun ends.
  2. Using the waterloo.aws tape as a working example, we issue the command vmfplc2 scan waterloo.aws > waterloo.scan.txt
  3. We see that the virtual tape file scan contains 135 subfiles.

General User
HHC02499I Hercules utility vmfplc2 - VM/CMS VMFPLC2/TAPE Utility - version 4.3.9999.35-SDL-g5f98961-modified HHC01414I (C) Copyright 1999-2020 by Roger Bowler, Jan Jaeger, and others HHC01417I ** The SoftDevLabs version of Hercules ** HHC01415I Build date: Oct 7 2020 at 14:15:24 HHC02637I Input tape format set to VMFPLC2 HHC02625I SCANNING... HHC02626I >>> ABSTRACT ABSTRACT A1 F 80 1982-05-25 15:58:21 recs 1437 blks 144 HHC02633I @TM HHC02626I >>> M0001V00 MEMO B1 F 80 1979-11-21 10:43:00 recs 6 blks 1 HHC02626I >>> M0001V00 DOCUMENT B1 F 80 1978-07-31 09:28:00 recs 22 blks 3 HHC02626I >>> NUCMAP EXEC B1 V 61 1978-07-31 09:28:00 recs 9 blks 1 HHC02626I >>> NUCMAP PLI B1 F 80 1978-07-31 09:28:00 recs 73 blks 8 HHC02626I >>> NUCMAP MODULE B1 V 9976 1978-07-31 09:28:00 recs 3 blks 15 HHC02626I >>> FASTREAD ASSEMBLE B1 F 80 1978-07-31 09:28:00 recs 89 blks 9 HHC02626I >>> FASTREAD MEMO B1 F 80 1978-07-31 09:28:00 recs 33 blks 4 HHC02626I >>> FASTREAD TEXT B1 F 80 1978-07-31 09:28:00 recs 9 blks 1 HHC02633I @TM HHC02626I >>> M0002V00 MEMO B1 F 80 1979-11-22 10:34:00 recs 7 blks 1 HHC02626I >>> M0002V00 DOCUMENT B1 V 60 1979-11-22 10:34:00 recs 28 blks 2 HHC02626I >>> INPSCAN ASSEMBLE B1 F 80 1979-11-22 10:34:00 recs 426 blks 43 HHC02626I >>> MTAM ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 24 blks 3 HHC02626I >>> MTAMACCT ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 225 blks 23 HHC02626I >>> MTAMATTX ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 21 blks 3 HHC02626I >>> MTAMCALL ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 28 blks 3 HHC02626I >>> MTAMCHOP ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 32 blks 4 HHC02626I >>> MTAMDIR ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 160 blks 16 HHC02626I >>> MTAMEND ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 5 blks 1 HHC02626I >>> MTAMEXT ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 46 blks 5 HHC02626I >>> MTAMHASH ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 20 blks 2 HHC02626I >>> MTAMINIT ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 11 blks 2 HHC02626I >>> MTAMIO ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 109 blks 11 HHC02626I >>> MTAMOFF ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 47 blks 5 HHC02626I >>> MTAMON ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 50 blks 5 HHC02626I >>> MTAMOPEN ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 84 blks 9 HHC02626I >>> MTAMSTOP ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 23 blks 3 HHC02626I >>> MTAMSTRT ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 117 blks 12 HHC02626I >>> MTAMWAIT ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 100 blks 10 HHC02626I >>> OUTSCAN ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 388 blks 39 HHC02626I >>> RDCHEK ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 168 blks 17 HHC02626I >>> TERMINTH ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 212 blks 22 HHC02626I >>> TERMIO ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 515 blks 52 HHC02626I >>> TOPNCLS ASSEMBLE B1 F 80 1978-09-12 09:23:00 recs 242 blks 25 HHC02626I >>> TREAD ASSEMBLE B1 F 80 1978-09-12 09:24:00 recs 255 blks 26 HHC02626I >>> TWRITE ASSEMBLE B1 F 80 1978-09-12 09:24:00 recs 209 blks 21 HHC02626I >>> WRCHEK ASSEMBLE B1 F 80 1978-09-12 09:24:00 recs 113 blks 12 HHC02626I >>> DMSIBM AUXUOW B1 F 80 1978-09-12 09:24:00 recs 1 blks 1 HHC02626I >>> SESSBLOK COPY B1 F 80 1978-09-12 09:24:00 recs 39 blks 4 HHC02626I >>> ATAMASM EXEC B1 V 51 1978-09-12 09:24:00 recs 14 blks 1 HHC02626I >>> MTAMASM EXEC B1 F 80 1978-09-12 09:24:00 recs 13 blks 2 HHC02626I >>> MTAMBLD EXEC B1 F 80 1978-09-12 09:24:00 recs 47 blks 5 HHC02626I >>> MTAMGEN EXEC B1 V 64 1978-09-12 09:24:00 recs 106 blks 4 HHC02626I >>> ATAM LISTING B1 V 73 1978-09-12 09:24:00 recs 575 blks 35 HHC02626I >>> MTAM LISTING B1 V 67 1978-09-12 09:24:00 recs 556 blks 29 HHC02626I >>> ATAMLIB MACLIB B1 F 80 1978-09-12 09:24:00 recs 721 blks 73 HHC02626I >>> MTAMGEN MACLIB B1 F 80 1978-09-12 09:24:00 recs 211 blks 22 HHC02626I >>> MTAMLIB MACLIB B1 F 80 1978-09-12 09:25:00 recs 620 blks 62 HHC02626I >>> MTAMGEN MODEL B1 F 80 1978-09-12 09:25:00 recs 18 blks 2 HHC02626I >>> MTAMDIR MODULE B1 V 3216 1978-09-12 09:25:00 recs 3 blks 5 HHC02626I >>> MTEST MODULE B1 V 20712 1978-09-12 09:25:00 recs 3 blks 29 HHC02626I >>> MTEST PLI B1 F 80 1978-09-12 09:25:00 recs 42 blks 5 HHC02626I >>> ATAM SCRIPT B1 V 80 1978-09-12 09:25:00 recs 784 blks 34 HHC02626I >>> ATAMMTAM SCRIPT B1 V 90 1978-09-12 09:25:00 recs 29 blks 2 HHC02626I >>> MTAM SCRIPT B1 V 79 1978-09-12 09:25:00 recs 676 blks 24 HHC02626I >>> MTEST TEXT B1 F 80 1978-09-12 09:25:00 recs 35 blks 4 HHC02626I >>> ATAMLIB TXTLIB B1 F 80 1978-09-12 09:25:00 recs 263 blks 27 HHC02626I >>> MTAMLIB TXTLIB B1 F 80 1978-09-12 09:26:00 recs 242 blks 25 HHC02626I >>> DMKHVC UPDTMTAM B1 F 80 1978-09-12 09:26:00 recs 53 blks 6 HHC02626I >>> DMSIBM UW155DMS B1 F 80 1978-09-12 09:26:00 recs 55 blks 6 HHC02633I @TM . . . HHC02633I @TM HHC02626I >>> M1132V00 MEMO B1 F 80 1981-12-15 14:59:45 recs 6 blks 1 HHC02626I >>> M1132V00 DOCUMENT B1 F 80 1981-11-20 16:48:27 recs 45 blks 5 HHC02626I >>> GENNS EXEC B1 V 64 1981-11-11 10:59:11 recs 7 blks 2 HHC02626I >>> GENNS PLI B1 F 80 1981-10-29 13:05:03 recs 308 blks 31 HHC02626I >>> GENNS TEXT B1 F 80 1981-11-20 16:51:59 recs 510 blks 51 HHC02626I >>> GENNS LISTING B1 V 121 1981-11-20 16:51:59 recs 597 blks 89 HHC02626I >>> GENNS MODULE B1 V 45816 1981-11-20 16:52:11 recs 3 blks 63 HHC02626I >>> SNTSP10 ASSEMBLE B1 F 80 1981-11-20 15:23:28 recs 123 blks 13 HHC02626I >>> SNTSP10 INPUT B1 F 80 1981-11-20 15:22:50 recs 69 blks 7 HHC02633I @TM HHC02626I >>> M1133V00 MEMO B1 F 80 1982-05-21 10:11:57 recs 2 blks 1 HHC02626I >>> M1133V00 DOCUMENT B1 F 80 1982-02-26 10:00:00 recs 45 blks 5 HHC02626I >>> EDTGEN EXEC B1 V 22 1982-02-26 09:57:00 recs 2 blks 2 HHC02626I >>> ED2 TEXT B1 F 80 1982-02-25 08:36:00 recs 106 blks 11 HHC02626I >>> $EDT EXEC B1 V 15 1982-01-26 14:53:00 recs 7 blks 2 HHC02633I @TM HHC02626I >>> M1134V00 MEMO B1 F 80 1982-05-21 10:54:20 recs 5 blks 1 HHC02626I >>> M1134V00 DOCUMENT B1 F 80 1982-02-17 15:42:00 recs 94 blks 10 HHC02626I >>> COMVME ASSEMBLE B1 F 80 1982-02-17 15:35:00 recs 155 blks 16 HHC02633I @TM HHC00204E 0:0000 Tape file E:\Emulation\operations.workspace\Tapes\waterloo.aws, type het: error in function het_read(), offset 0x00000000000042D2: end of file (uninitialized tape) HHC00007I Previous message from function 'read_het' at hettape.c(204)

Scan Data to Control File

The format of the vmfplc2 DUMP command is:

General User
HHC02499I Hercules utility vmfplc2 - VM/CMS VMFPLC2/TAPE Utility - version 4.3.9999.35-SDL-g5f98961-modified HHC01414I (C) Copyright 1999-2020 by Roger Bowler, Jan Jaeger, and others HHC01417I ** The SoftDevLabs version of Hercules ** HHC01415I Build date: Oct 7 2020 at 14:15:24 HHC02621E Invalid function "/?" HHC00007I Previous message from function 'parse_parms' at vmfplc2.c(263) HHC02620I Usage: vmfplc2 [options] VERB [<ctlfile>] <tapein> | <tapeout> HHC02620I VERB desired action (DUMP/LOAD/SCAN) HHC02620I ctlfile filename of control file (DUMP/LOAD only) HHC02620I tapein filename of input tape file (LOAD/SCAN only) HHC02620I tapeout filename of output tape file (DUMP only) HHC02620I Options: HHC02620I -c cp desired translation codepage HHC02620I -t read/write tape in CMS 'TAPE' DUMP format HHC02620I -v don't suppress certain informational messages HHC02620I -u create uncompressed .aws output HHC02620I -z create compressed .het using zlib (default) HHC02620I -b create compressed .het using bzip2 HHC02620I -4 desired compression level (-1, -2 ... -9)

The control file that needs to be derived from this tape, in preparation for loading is a somewhat mechanical transformation. However, there is another complexity. The representation of the on-disk file storage table (FST) looks like this: 

    X'00'+-----------------------------+    
         ×       F I L E N A M E       ×    
    X'08'×-----------------------------×    
         ×       F I L E T Y P E       ×    
    X'10'×-----------------------------×    
         ×  X'MMDDHHMM' ×WRT PTR×RD PTR×    
    X'18'×-----------------------------×    
         × MODE ×# ITEMS×  FCL  ×F/V×FL×    
    X'20'×-----------------------------×    
         ×     LRECL    ×DB CNT × YEAR ×    
    X'28'×-----------------------------×    
         ×#FULL TAPE BLK×LTH LSTBLK/800×    
    X'30'×-----------------------------×    
         × FILE ORIG PTR× ALT DATA BLK ×    
    X'38'×-----------------------------×    
         × ALT ITEM CNT ×#LV×PTRSZ×YYMM×    
    X'40'×-----------------------------×    
         ×   DDHHMMSS   ×  RESERVED    ×    
    X'48'×-----------------------------×

Note that the YEAR portion of the record is a two-digit representation. According to the listfile function (as implemented by DMSLST ASSEMBLE, all date representations PRIOR to the year 1960 are considered to be part of the next century. None of the utilities seem to understand this so even though the tape we've chosen waterloo.aws has dates after the year 1960, there are MANY files on the VM/370 system which have dates prior to 01 Jan 1960.

The Windows Host version of VMFPLC2 does NOT attempt to adjust dates and simply uses the system date and time at the time of the restore operation when transferring files to the host system. This makes sense because many so-called 'modern' filesystem formats do not understand dates prior to 01 Jan 1980. So preserving dates between systems is a two-step process, if at all.

There are additional challenges such as permissible characters in the hostfile which need to be managed carefully and therefore it makes sense use a utility program to perform this entire conversion.

Automatic Generation

To facilitate the transfer operation, a utility has been created to ingest AWS/HET tapes, and generate the control files and command files necessary to extract the AWS/HET Tape contents to a Windows (or Linux) host filesystem. The ProcAWSTape utility was created for this purpose.

For each tape, three files are generated by the utility:

  1. <TapeFileName>.files.txt
    a formatted/detailed enumeration of the tape's contents including the “host system safe filename” to which the contents can be written.
  2. <TapeFileName>.tapemap.ctl
    a formatted enumeration of the tape's contents according to the rules of the VMFPLC2 utility. This file is ONLY present if the tape contains VMFPLC2 or TAPE formatted output. The utility automatically detects which formats are contained within the tape.
  3. <TapeFileName.tapemap.cmd
    The generated command file which invokes the proper utilities for exporting the tape's content.

The Control File

The resulting control file looks like this:

General User
; * FileName FileType FM RecFM LRecL XFerType Host FileName ; --- ; --- Start of Tape File 000001 ; --- ABSTRACT ABSTRACT A1 Fixed 80 Text "F000001\ABSTRACT.ABSTRACT.A1.txt" ; --- ; --- Start of Tape File 000002 ; --- @TM M0001V00 MEMO B1 Fixed 80 Text "F000002\M0001V00.MEMO.B1.txt" M0001V00 DOCUMENT B1 Fixed 80 Text "F000002\M0001V00.DOCUMENT.B1.txt" NUCMAP EXEC B1 Variable Text "F000002\NUCMAP.EXEC.B1.txt" NUCMAP PLI B1 Fixed 80 Text "F000002\NUCMAP.PLI.B1.txt" NUCMAP MODULE B1 Variable Binary "F000002\NUCMAP.MODULE.B1.bin" FASTREAD ASSEMBLE B1 Fixed 80 Text "F000002\FASTREAD.ASSEMBLE.B1.txt" FASTREAD MEMO B1 Fixed 80 Text "F000002\FASTREAD.MEMO.B1.txt" FASTREAD TEXT B1 Fixed 80 Binary "F000002\FASTREAD.TEXT.B1.bin" ; --- ; --- Start of Tape File 000003 ; --- @TM . . . ; --- ; --- Start of Tape File 000133 ; --- @TM M1132V00 MEMO B1 Fixed 80 Text "F000133\M1132V00.MEMO.B1.txt" M1132V00 DOCUMENT B1 Fixed 80 Text "F000133\M1132V00.DOCUMENT.B1.txt" GENNS EXEC B1 Variable Text "F000133\GENNS.EXEC.B1.txt" GENNS PLI B1 Fixed 80 Text "F000133\GENNS.PLI.B1.txt" GENNS TEXT B1 Fixed 80 Binary "F000133\GENNS.TEXT.B1.bin" GENNS LISTING B1 Variable Text "F000133\GENNS.LISTING.B1.txt" GENNS MODULE B1 Variable Binary "F000133\GENNS.MODULE.B1.bin" SNTSP10 ASSEMBLE B1 Fixed 80 Text "F000133\SNTSP10.ASSEMBLE.B1.txt" SNTSP10 INPUT B1 Fixed 80 Text "F000133\SNTSP10.INPUT.B1.txt" ; --- ; --- Start of Tape File 000134 ; --- @TM M1133V00 MEMO B1 Fixed 80 Text "F000134\M1133V00.MEMO.B1.txt" M1133V00 DOCUMENT B1 Fixed 80 Text "F000134\M1133V00.DOCUMENT.B1.txt" EDTGEN EXEC B1 Variable Text "F000134\EDTGEN.EXEC.B1.txt" ED2 TEXT B1 Fixed 80 Binary "F000134\ED2.TEXT.B1.bin" $EDT EXEC B1 Variable Text "F000134\$EDT.EXEC.B1.txt" ; --- ; --- Start of Tape File 000135 ; --- @TM M1134V00 MEMO B1 Fixed 80 Text "F000135\M1134V00.MEMO.B1.txt" M1134V00 DOCUMENT B1 Fixed 80 Text "F000135\M1134V00.DOCUMENT.B1.txt" COMVME ASSEMBLE B1 Fixed 80 Text "F000135\COMVME.ASSEMBLE.B1.txt"

Because we have the file “in-hand”, it was also a straightforward process to calculate other statistics about the tape's content:

  1. The number of Internal Virtual Tape Files
    1. Sequence Number
    2. Number of Bytes
    3. Number of Blocks
      1. Minimum Block Size
      2. Maximum Block Size
      3. Average Block Size
  2. The Virtual Tape File Format and size (and creation date)
  3. If formatted, which format was used (VMFPLC2 or TAPE)
    1. Filename
    2. Filetype
    3. Filemode
    4. Fixed or Variable
      1. if Fixed, the logical record length
    5. The “automatically designated” export format (Text or Binary)
    6. The “automatically derived” host filename.

The entire waterloo.tapemap.ctl file is available for review.

Windows Host to Virtual Tape

Once the configuration file has been generated, it can be used to reverse the process in writing the files back to the Virtual Tape and then back to the host operating system. Using the same control file is a good starting point for this.

Virtual Tape to CMS

Transferring files back to CMS from the VMFPLC2 formatted tape is trivial (enough) if we got the file and record formats correct in the first place.