Using the Programmatic Python Interface to submit reference files
=================================================================

As an alternative to command-line and web submissions of reference files,
there is a programmatic interface that allows users to perform the tasks
of reference file submission using Python.  It requires crds version of at
least 7.3.1.

Required CRDS Account
.....................

Submitting files to CRDS requires obtaining a CRDS account authorizing you to
do so.  These permissions are managed by CRDS and checked automatically after
your MyST identity is verified.

To obtain a CRDS account,  please file a ticket with the `CRDS Jira`_
project requesting authorization so submit files.

.. _`CRDS Jira`:  https://jira.stsci.edu/projects/CRDS/issues

**IMPORTANT** Your MyST identity is connected to your CRDS permissions via your
MyST e-mail.  You should specify your MyST e-mail when applying for a CRDS
account.

Authenticating
..............

To utilize the programmatic API you are required to have a CRDS account and to
authenticate via MyST and auth.mast_.  MyST will ask you to enter your AD
password unless you have already authenticated.  auth.mast_ supports the
creation and revocation of secret token_ values.

.. _auth.mast: https://auth.mast.stsci.edu/
.. _token: https://auth.mast.stsci.edu/tokens

The string value of this token must be inserted into the environment
variable MAST_API_TOKEN::

    import os
    os.environ['MAST_API_TOKEN'] = 'LongStringObtainedFromAuth.mast.stsciPage'

A typical auth.mast token looks something like::

  1b73d91d0db55b1800847529a3b6f91e

**IMPORTANT:** Since CRDS now uses MyST and auth.mast to identify you, never
never directly enter your AD password (MyST password) on the CRDS server or in
response to a CRDS prompt.

**NOTE:** CRDS passwords for existing CRDS file submitters and operators are
obsolete and no longer work.

API Description
...............

To use this functionality, import the Submission class from crds::

    from crds.submit import Submission

The primary object that a user interacts with is a Submission
object.  This can be instantiated using the following syntax::

    s = Submission('jwst', 'ops')

The Submission object has several methods, not least a .help()
method that gives the full information on all the fields to be filled::

    s.help()
    deliverer (str)
    ---------
    Name of deliverer
    Who are you?

    other_email (str, optional)
    -----------
    Other e-mail adresses to send notifications
    Comma-delimited list (optional)

    instrument (str)
    ----------
    Instrument  (All submitted files should match this instrument.  This
    instrument will be locked for your submission exclusively)
    Valid choices:
      {'fgs', 'miri', 'nircam', 'niriss', 'nirspec', 'system'}

    file_type (str)
    ---------
    Type of files (Bias, Dark, etc.)

    history_updated (bool, optional)
    ---------------
    Has HISTORY section in the primary header been updated to describe in
    detail the reason for delivery and how the files were created?
    Valid choices:
      {'False', 'True'}

    pedigree_updated (bool, optional)
    ----------------
    Has PEDIGREE keyword been checked and updated as necessary?
    Valid choices:
      {'False', 'True'}

    keywords_checked (bool, optional)
    ----------------
    Have REFTYPE and AUTHOR been checked and updated as necessary?
    Valid choices:
      {'False', 'True'}

    descrip_updated (bool, optional)
    ---------------
    Was the DESCRIP keyword updated with a summary of why the files were
    updated or created?
    Valid choices:
      {'False', 'True'}

    useafter_updated (bool, optional)
    ----------------
    Has the USEAFTER keyword been checked, and if necessary, updated?
    Valid choices:
      {'False', 'True'}

    useafter_matches (str)
    ----------------
    If the reference files are replacing previous versions, do the new
    USEAFTER dates exactly match the old ones?
    N/A for ETC Files Only
    Valid choices:
      {'N/A', 'No', 'Yes'}

    compliance_verified (str)
    -------------------
    Verification for compliance complete (fits, json, asdf compliant,
    certify, etc. or N/A)
    optional
    Valid choices:
      {'N/A', 'No', 'Yes'}

    ingest_files (bool, optional)
    ------------
    Should the affected files be reprocessed?
    Valid choices:
      {'False', 'True'}

    etc_delivery (bool, optional)
    ------------
    Should the files be submitted to the ETC?
    Valid choices:
      {'False', 'True'}

    jwst_etc (bool, optional)
    --------
    Are these JWST ETC files?
    Valid choices:
      {'False', 'True'}

    calpipe_version (str)
    ---------------
    Files run through the current version of the calibration software
    being used by the pipeline or PYSYNPHOT and ETC (yes/no and version
    number)

    replacement_files (bool, optional)
    -----------------
    Are any files replacing old reference files (deliveries can be a mix
    of files that are or are not replacing old files) (yes/no)
    Valid choices:
      {'False', 'True'}

    old_reference_files (str, optional)
    -------------------
    If yes, list them here

    replacing_badfiles (str)
    ------------------
    If the files being replaced are bad (resulting in scientifically
    invalid results), and should not be used with any data, please
    indicate this here
    ... or crashing the system.
    Valid choices:
      {'N/A', 'No', 'Yes'}

    jira_issue (str, optional)
    ----------
    Any JIRA issues filed in regard to the references being delivered
    (e.g. "REDCAT-25")
    Comma-delimited list (optional)

    table_rows_changed (str, optional)
    ------------------
    If files are tables, please indicate exactly which rows have changed

    modes_affected (str)
    --------------
    Please indicate which modes (e.g. all the STIS, FUVMAMA, E140L modes)
    are affected by the changes in the files

    change_level (str)
    ------------
    Degree that new files are expected to impact science results.
    Valid choices:
      {'TRIVIAL', 'SEVERE', 'MODERATE'}

    correctness_testing (str)
    -------------------
    Description of how the files were tested for correctness

    additional_considerations (str, optional)
    -------------------------
    Additional considerations

    description (str)
    -----------
    Information about file changes and expected impacts, include
    instrument and type.  Formatting note: only alphanumeric, periods,
    commas, dashes, and underscores are allowed

The fields of the submission object can be filled using (key, value)
assignments as is done with Python dictionaries::

    s['deliverer'] = 'Scarlet Feline'
    s['other_email'] = 'redcat@stsci.edu'
    s['instrument'] = 'miri'

The value of s['instrument'] is limited to the set of valid instruments for the
observatory specified in the constructor for s.  If you try to set s['instrument']
to something other than one of these instruments, an exception will occur.
Similarly, if you try to set s['chicken'], or some other keyword not in the
set of allowed keywords, and exception will occur::

    s['file_type']                  = 'DARK'
    s['history_updated']            = True
    s['pedigree_updated']           = True
    s['keywords_checked']           = True
    s['descrip_updated']            = True
    s['useafter_updated']           = True
    s['useafter_matches']           = True
    s['compliance_verified']        = 'N/A'
    s['ingest_files']               = False
    s['etc_delivery']               = False
    s['jwst_etc']                   = False
    s['calpipe_version']            = '0.9.1'
    s['replacement_files']          = False
    #s['old_reference_files']       = ''
    s['replacing_badfiles']         = 'N/A'
    #s['jira_issue']                = ''
    s['table_rows_changed']         = 'All rows'
    s['modes_affected']             = 'All MIRI observations starting 2017-Jan-03'
    s['correctness_testing']        = 'DARK was run on all MIRI data and verified '
    #s['additional_considerations'] = ''
    s['change_level']               = 'MODERATE'
    s['description']                = 'Updating MIRI DARK starting 2017-Jan-03.'

The assignments that are commented out are unnecessary as the default values are empty strings.
To add files to the submission, use the add_file() method::

    s.add_file('miri_dark_file.fits')

You can also remove files::

    s.remove_file('miri_dark_file.fits')

And finally, when the files to be submitted have been added and the fields
of the Submission have been filled in, the Submission can be submitted::

    result = s.submit()
    2019-04-24 12:24:50,823 - CRDS - INFO -  =============================== setting up ===============================
    2019-04-24 12:24:51,038 - CRDS - INFO -  Symbolic context 'jwst-edit' resolves to 'jwst_0511.pmap'
    2019-04-24 12:24:51,038 - CRDS - INFO -  ########################################
    2019-04-24 12:24:51,038 - CRDS - INFO -  Certifying './miri_dark_test.fits' (1/1) as 'FITS' relative to context 'jwst_0511.pmap'
    2019-04-24 12:24:55,166 - CRDS - INFO -  FITS file 'miri_dark_test.fits' conforms to FITS standards.
    2019-04-24 12:24:56,219 - CRDS - WARNING -  Missing suggested keyword 'META.MODEL_TYPE [DATAMODL]'
    2019-04-24 12:24:56,298 - CRDS - INFO -  [0] DETECTOR MIRIMAGE 
    2019-04-24 12:24:56,298 - CRDS - INFO -  [0] EXP_TYPE MIR_IMAGE 
    2019-04-24 12:24:56,302 - CRDS - INFO -  META.AUTHOR [AUTHOR] = 'JANE MORRISON'
    2019-04-24 12:24:56,302 - CRDS - INFO -  META.DESCRIPTION [DESCRIP] = 'MIRI Dark Correction for MIRI'
    2019-04-24 12:24:56,302 - CRDS - INFO -  META.EXPOSURE.GROUPGAP [GROUPGAP] = 0
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.EXPOSURE.NFRAMES [NFRAMES] = 1
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.EXPOSURE.NGROUPS [NGROUPS] = 200
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.EXPOSURE.READPATT [READPATT] = 'FAST'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.EXPOSURE.TYPE [EXP_TYPE] = 'MIR_IMAGE'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.HISTORY [HISTORY] = 'SEE MIRI-TN-00007-UA-Mean-Dark.PDF for details\nFit was done correcting for nonlinearity and RSCD Correction\nOne solution rejected first 10 frames and last frame\nSecond solution corrected all the frames for nonlinearity and RSCD\nBecause of drifting zero points, first solution was used to find frame v\nframe value at time = 0 was subtracted off all the data\nresulting residuals were binned and average\n a linear fits was performed on the residuals to extrapolate the dark re\n the number of frames used in the fit is defined by NGVALID\n The mean dark residuals were used for frames with valid data, out side\nDATA USED: to create dark CV3  IMG_RAD_01, IMG_RAD_13\nDOCUMENT: MIRI-TR-00007-UA-Dark_06.00.pdf\nSOFTWARE: idl code create_dark_CDP6.pro merge_int_CDP6\nDIFFERENCES: darks now have two integrations\nCreated from: MiriDarkReferenceModel'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.BAND [BAND] = 'UNDEFINED'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.CHANNEL [CHANNEL] = 'UNDEFINED'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.CORONAGRAPH [CORONMSK] = 'UNDEFINED'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.DETECTOR [DETECTOR] = 'MIRIMAGE'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.FILTER [FILTER] = 'ANY'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.GRATING [GRATING] = 'UNDEFINED'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.NAME [INSTRUME] = 'MIRI'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.INSTRUMENT.PUPIL [PUPIL] = 'UNDEFINED'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.MODEL_TYPE [DATAMODL] = 'UNDEFINED'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.PEDIGREE [PEDIGREE] = 'GROUND'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.REFTYPE [REFTYPE] = 'DARK'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.SUBARRAY.FASTAXIS [FASTAXIS] = 1
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.SUBARRAY.NAME [SUBARRAY] = 'FULL'
    2019-04-24 12:24:56,303 - CRDS - INFO -  META.SUBARRAY.SLOWAXIS [SLOWAXIS] = 2
    2019-04-24 12:24:56,304 - CRDS - INFO -  META.SUBARRAY.XSIZE [SUBSIZE1] = 1032
    2019-04-24 12:24:56,304 - CRDS - INFO -  META.SUBARRAY.XSTART [SUBSTRT1] = 1
    2019-04-24 12:24:56,304 - CRDS - INFO -  META.SUBARRAY.YSIZE [SUBSIZE2] = 1024
    2019-04-24 12:24:56,304 - CRDS - INFO -  META.SUBARRAY.YSTART [SUBSTRT2] = 1
    2019-04-24 12:24:56,304 - CRDS - INFO -  META.TELESCOPE [TELESCOP] = 'JWST'
    2019-04-24 12:24:56,304 - CRDS - INFO -  META.USEAFTER [USEAFTER] = '2015-08-02T00:00:00'
    2019-04-24 12:24:56,304 - CRDS - INFO -  Running fitsverify.
    2019-04-24 12:24:56,315 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,315 - CRDS - INFO -  >>               fitsverify 4.18 (CFITSIO V3.440)              
    2019-04-24 12:24:56,315 - CRDS - INFO -  >>               --------------------------------              
    2019-04-24 12:24:56,315 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,315 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,315 - CRDS - INFO -  >> File: ./miri_dark_test.fits
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> 
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> 7 Header-Data Units in this file.
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> =================== HDU 1: Primary Array ===================
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  57 header keywords
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  Null data array; NAXIS = 0 
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> =================== HDU 2: Image Exten. ====================
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  15 header keywords
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> SCI 32-bit floating point pixels,  4 axes (1032 x 1024 x 200 x 2), 
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> =================== HDU 3: Image Exten. ====================
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  15 header keywords
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> ERR 32-bit floating point pixels,  4 axes (1032 x 1024 x 200 x 2), 
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,316 - CRDS - INFO -  >> =================== HDU 4: Image Exten. ====================
    2019-04-24 12:24:56,316 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  16 header keywords
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >> DQ 32-bit integer pixels,  4 axes (1032 x 1024 x 1 x 2), 
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >> =================== HDU 5: BINARY Table ====================
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  21 header keywords
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  DQ_DEF  (4 columns x 2 rows)
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  Col# Name (Units)       Format
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>    1 BIT                  J         
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>    2 VALUE                J         
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>    3 NAME                 40A       
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>    4 DESCRIPTION          80A       
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >> =================== HDU 6: Image Exten. ====================
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  8 header keywords
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >> METADATA 8-bit integer pixels,  1 axes (1605), 
    2019-04-24 12:24:56,317 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,317 - CRDS - INFO -  >> =================== HDU 7: Image Exten. ====================
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  9 header keywords
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,318 - CRDS - INFO -  >> FITERR 32-bit floating point pixels,  2 axes (1032 x 1024), 
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,318 - CRDS - INFO -  >> ++++++++++++++++++++++ Error Summary  ++++++++++++++++++++++
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  HDU#  Name (version)       Type             Warnings  Errors
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  1                          Primary Array    0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  2     SCI                  Image Array      0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  3     ERR                  Image Array      0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  4     DQ                   Image Array      0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  5     DQ_DEF               Binary Table     0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  6     METADATA             Image Array      0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  7     FITERR               Image Array      0         0     
    2019-04-24 12:24:56,318 - CRDS - INFO -  >>  
    2019-04-24 12:24:56,318 - CRDS - INFO -  >> **** Verification found 0 warning(s) and 0 error(s). ****
    2019-04-24 12:24:56,318 - CRDS - INFO -  Checking JWST datamodels.
    2019-04-24 12:24:56,347 - CRDS - WARNING -  NoTypeWarning : jwst.datamodels.util : model_type not found. Opening ./miri_dark_test.fits as a ReferenceQuadModel
    2019-04-24 12:24:56,396 - CRDS - INFO -  ########################################
    2019-04-24 12:24:56,436 - CRDS - INFO -  Logging in aquiring lock.
    2019-04-24 12:24:57,489 - CRDS - INFO -  =============================== wipe files ===============================
    2019-04-24 12:24:57,613 - CRDS - INFO -  Preparing server logging.
    2019-04-24 12:24:57,723 - CRDS - INFO -  ============================== ingest files ==============================
    2019-04-24 12:24:57,724 - CRDS - INFO -  Uploading 1 file(s) totalling   3.4 G bytes
    2019-04-24 12:24:57,825 - CRDS - INFO -  Upload started './miri_dark_test.fits' [ 1 / 1  files ] [   3.4 G /   3.4 G  bytes ]
    2019-04-24 12:26:07,683 - CRDS - INFO -  ===========================================================================
    2019-04-24 12:26:07,683 - CRDS - INFO -  Posting web request for '/submission_form/redcat_submit/'
    CRDS - INFO -  ======= monitoring server on 'e8e0f1d3-45d4-44cd-a2b8-1aa279f5dd33' =======
    CRDS - INFO -  >> Starting submission processing.
    CRDS - INFO -  >> Certifying 'miri_dark_test.fits'
    CRDS - INFO -  >> Resolved old rmap as 'jwst_miri_dark_0033.rmap' based on context 'jwst_0511.pmap'
    CRDS - INFO -  >> Doing trial insertion of 1 references into 'jwst_miri_dark_0033.rmap'
    CRDS - INFO -  >> Processing 'miri_dark_test.fits' [1 / 1 files] [  3.4 G /   3.4 G /   3.4 G bytes]
    CRDS - INFO -  >> Renaming 'miri_dark_test.fits' --> 'jwst_miri_dark_0075.fits'
    CRDS - INFO -  >> Linking miri_dark_test.fits --> jwst_miri_dark_0075.fits
    CRDS - INFO -  >> Adding file 'miri_dark_test.fits' to database.
    CRDS - INFO -  >> Generating new rmap 'jwst_miri_dark_0034.rmap' from 'jwst_miri_dark_0033.rmap'.
    CRDS - INFO -  >> Adding file 'jwst_miri_dark_0034.rmap' to database.
    CRDS - INFO -  >> Certifying 'jwst_miri_dark_0034.rmap'
    CRDS - INFO -  >> Checking for derivation collisions.
    CRDS - INFO -  >> Computing file differences.
    CRDS - INFO -  >> Differencing 'jwst_miri_dark_0033.rmap' vs. 'jwst_miri_dark_0034.rmap'
    CRDS - INFO -  >> COMPLETED: https://jwst-crds-test.stsci.edu//display_result/f01bdb8b-6d70-46a8-95e1-e2bdb6ce1f9c
    CRDS - INFO -  ========================= monitoring server done =========================
    CRDS - INFO -  0 errors
    CRDS - INFO -  2 warnings
    CRDS - INFO -  132 infos
    CRDS - INFO -  STARTED 2019-04-24 12:24:50.81
    CRDS - INFO -  STOPPED 2019-04-24 12:27:41.93
    CRDS - INFO -  ELAPSED 0:02:51.11

This will validate the submission by making sure all of the fields that need values
have them and by ensuring that there is at least 1 file to submit before performing
the submission.

The return value of `submit()` includes error and warning counts, as well as
a link to the Review/Confirm page for the submission::

    result.error_count
    result.warning_count
    result.ready_url

The `open_ready_url()` function will attempt to open the Review/Confirm page in
your system's default browser::

    result.open_ready_url()

Note that the page's *confirm*, *cancel*, and *force* buttons will not be available
unless authenticated.  If they seem to be missing, try logging in with the *Login*
button at the upper right-hand corner of the page.
