..  _`demo_cobol`:

##########################################################
Reading COBOL Files
##########################################################

For sample data, we're using data found here:
http://wonder.cdc.gov/wonder/sci_data/codes/fips/type_txt/cntyxref.asp

The data files are in two zip archives: http://wonder.cdc.gov/wonder/sci_data/datasets/zipctyA.zip 
and http://wonder.cdc.gov/wonder/sci_data/datasets/zipctyB.zip

Each of these archives contains five large files, with 2,310,000 rows of data, plus a header. 
The 10th file has 2,037,944 rows of data plus a header.

The member names of the ZIP archive are ``zipcty1`` to ``zipcty5``
and ``zipcty6`` to ``zipcty10``.

We'll work with two small subsets in the sample directory.

Here are the two record layouts.

..    parsed-literal::

    COUNTY CROSS-REFERENCE FILE - COBOL EXAMPLE

            BLOCK CONTAINS 0 RECORDS
            LABEL RECORDS ARE STANDARD
            RECORD CONTAINS 53 CHARACTERS
            RECORDING MODE IS F
            DATA RECORDS ARE
                   COUNTY-CROSS-REFERENCE-RECORD.

        01  COUNTY-CROSS-REFERENCE-RECORD.
            05   ZIP-CODE                                 PIC X(05).
            05   UPDATE-KEY-NO                            PIC X(10).
            05   ZIP-ADD-ON-RANGE.
                 10  ZIP-ADD-ON-LOW-NO.
                      15  ZIP-SECTOR-NO                   PIC X(02).
                      15  ZIP-SEGMENT-NO                  PIC X(02).
                 10  ZIP-ADD-ON-HIGH-NO.
                      15  ZIP-SECTOR-NO                   PIC X(02).
                      15  ZIP-SEGMENT-NO                  PIC X(02).
            05   STATE-ABBREV                             PIC X(02).
            05   COUNTY-NO                                PIC X(03).
            05   COUNTY-NAME                              PIC X(25).

..    parsed-literal::

    COPYRIGHT HEADER RECORD - COBOL EXAMPLE

               BLOCK CONTAINS 0 RECORDS
               LABEL RECORDS ARE STANDARD
               RECORD CONTAINS 53 CHARACTERS
               RECORDING MODE IS F
               DATA RECORDS ARE
                   COPYRIGHT-HEADER RECORD.

          01  COPYRIGHT-HEADER-RECORD.
              05  FILLER                                     PIC  X(05).
              05  FILE-VERSION-YEAR                          PIC  X(02).
              05  FILE-VERSION-MONTH                         PIC  X(02).
              05  COPYRIGHT-SYMBOL                           PIC  X(11).
              05  TAPE-SEQUENCE-NO                           PIC  X(03).
              05  FILLER                                     PIC  X(30).

First Steps
==============

The actual COBOL code for the schema is in :file:`sample/zipcty.cob`.
This file has both record layouts.
These are two ``01`` level items in a single file.
    
When working with unknown files, we sometimes need to preview a raw dump of the
records.

..  literalinclude:: ../../../demo/cobol_reader.py
    :lines: 87-89

This is a handy expedient for debugging.

Builder Functions
=================

As suggested in :ref:`using`, here are two builder functions.
The :py:func:`header_builder` function creates a header object from the
first row of each :file:`zipcty*` file. 

..  literalinclude:: ../../../demo/cobol_reader.py
    :lines: 92-98

  
The :py:func:`detail_builder` function creates a detail object from the subsequent
rows of each :file:`zipcty*` file. 

Because the names within the COBOL layout are not unique at the bottom-most element level, we must use path names.
The default path names include all levels of the DDE.
More clever path name components might be useful here.

COBOL uses an "of" to work up the hierarchy looking for a unique name.

Maybe we could build a fluent interface ``schema['ZIP-SECTOR-NO'].of('ZIP-ADD-ON-LOW-NO')``.

..  literalinclude:: ../../../demo/cobol_reader.py
    :lines: 108-127

Sheet Processing
=================

Here's the :py:func:`process_sheet` function which applies the builders to the various
rows in each sheet.
Currently, all that happens is a print of the object that was built.

Note that we've transformed the schema from a simple, flat list into a dictionary keyed by field name.
For COBOL processing, this is essential, since the numeric order of fields isn't often sensible.

Also note that we've put two versions of each name into the schema dictionary.

-   The lowest level name.

-   The entire path down to the lowest level name.

[For spreadsheets, where columns are numbered, the positional information may be useful.]

..  literalinclude:: ../../../demo/cobol_reader.py
    :lines: 130-148

Top-Level Script
================

The top-level script must do two things:

1.  Parse the :file:`"zipcty.cob"` data definition to create a schema.

2.  Open a data file as a :py:class:`cobol.Character_File`.
    This presumes the file is all character (no COMP-3) and already translated into ASCII.
    
    The :py:func:`process_sheet` is applied to each file.

Here's a function to parse arguments.

..  literalinclude:: ../../../demo/cobol_reader.py
    :lines: 154-167

Given this function to parse the command-lines
arguments, the ``main()`` function looks
like this:

..  literalinclude:: ../../../demo/cobol_reader.py
    :lines: 170-190


Running the demo
================

We can run this program like this:

..  code-block:: bash

    python demo/cobol_reader.py --schema sample/zipcty.cob sample/zipcty[1-2]

The output looks like this.

..  parsed-literal::

    {'file_version_year': '88', 'file_version_month': '09', 'copyright_symbol': ' (C)USPS   ', 'tape_sequence_no': '001'}
    {'zip_code': '00401', 'update_key_no': '0000000001', 'low_sector': '00', 'low_segment': '01', 'high_sector': '00', 'high_segment': '01', 'state_abbrev': 'NY', 'county_no': '119', 'county_name': 'WESTCHESTER              '}
    {'zip_code': '02186', 'update_key_no': '0000462001', 'low_sector': '52', 'low_segment': '66', 'high_sector': '52', 'high_segment': '66', 'state_abbrev': 'MA', 'county_no': '021', 'county_name': 'NORFOLK                  '}
    {'zip_code': '06111', 'update_key_no': '0000924001', 'low_sector': '49', 'low_segment': '01', 'high_sector': '49', 'high_segment': '01', 'state_abbrev': 'CT', 'county_no': '003', 'county_name': 'HARTFORD                 '}
    {'zip_code': '07901', 'update_key_no': '0001386001', 'low_sector': '22', 'low_segment': '08', 'high_sector': '22', 'high_segment': '08', 'state_abbrev': 'NJ', 'county_no': '039', 'county_name': 'UNION                    '}
    {'zip_code': '10463', 'update_key_no': '0001848001', 'low_sector': '17', 'low_segment': '05', 'high_sector': '17', 'high_segment': '05', 'state_abbrev': 'NY', 'county_no': '005', 'county_name': 'BRONX                    '}
    INFO:__main__:Counter({'read': 5})
    {'file_version_year': '88', 'file_version_month': '09', 'copyright_symbol': ' (C)USPS   ', 'tape_sequence_no': '002'}
    {'zip_code': '11789', 'update_key_no': '0002310001', 'low_sector': '25', 'low_segment': '43', 'high_sector': '25', 'high_segment': '43', 'state_abbrev': 'NY', 'county_no': '103', 'county_name': 'SUFFOLK                  '}
    {'zip_code': '14767', 'update_key_no': '0002772001', 'low_sector': '97', 'low_segment': '71', 'high_sector': '97', 'high_segment': '71', 'state_abbrev': 'NY', 'county_no': '013', 'county_name': 'CHAUTAUQUA               '}
    {'zip_code': '17201', 'update_key_no': '0003234001', 'low_sector': '90', 'low_segment': '33', 'high_sector': '90', 'high_segment': '33', 'state_abbrev': 'PA', 'county_no': '055', 'county_name': 'FRANKLIN                 '}
    {'zip_code': '19438', 'update_key_no': '0003696001', 'low_sector': '28', 'low_segment': '22', 'high_sector': '28', 'high_segment': '22', 'state_abbrev': 'PA', 'county_no': '091', 'county_name': 'MONTGOMERY               '}
    {'zip_code': '21740', 'update_key_no': '0004158001', 'low_sector': '53', 'low_segment': '05', 'high_sector': '53', 'high_segment': '05', 'state_abbrev': 'MD', 'county_no': '043', 'county_name': 'WASHINGTON               '}


Working with Archives
=====================

We don't need to unpack the archives to work with files inside them.
We can open a ``ZipFile`` member and process that.
This can be a helpful optimization when small extracts are pulled from ZIP archives.

The trick is this:

When we open the file with ``COBOL_Text_File(filename)``
we can pass the file object created by ``ZipFile.open()`` as the second argument.

It looks like this::

    with COBOL_Text_File(filename, file_object=archive.open(filename)) as wb:
        ...

This uses an already opened explicit file object rather
than opening the given file name.


