mdf4reader module documentation — mdfreader 2.8 documentation

# mdf4reader module documentation

Measured Data Format file reader module for version 4.x.

## Platform and python version

With Unix and Windows for python 2.6+ and 3.2+

Author: | Aymeric Rateau

Created on Thu Dec 10 12:57:28 2013

## Dependencies

- Python >2.6, >3.2 <http://www.python.org>
- Numpy >1.6 <http://numpy.scipy.org>
- bitarray to parse bits in not aligned bytes
- Sympy to convert channels with formula if needed
- zlib to uncompress data block if needed

## Attributes

PythonVersion : float
  Python version currently running, needed for compatibility of both python 2.6+ and 3.2+

## mdf4reader module

class `mdfreader.mdf4reader.` `DATA`() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)

  Bases: `dict`

  Methods

  `addRecord`(record) | Adds a new record in DATA class dict.
  `clear`() |
  `copy`() |
  `fromkeys`($type, iterable[, value]) | Returns a new dict with keys from iterable and values equal to value.
  `get`(k[,d]) |
  `items`() |
  `keys`() |
  `load`(record, info[, nameList, sortedFlag, vlsd]) | Reads data block from record definition
  `pop`(k[,d]) | If key is not found, d is returned if given, otherwise KeyError is raised
  `popitem`() | 2-tuple; but raise KeyError if D is empty.
  `read`(channelSet, info, filename) | Reads data block
  `readRecord`(recordID, info, buf[, channelSet]) | read record from a buffer
  `setdefault`(k[,d]) |
  `update`([E, ]**F) | If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
  `values`() |

  `addRecord`(record)

    Adds a new record in DATA class dict.

    Parameters: | record class channel group definition listing record channel classes

  `fid`
  `load`(record, info, nameList=None, sortedFlag=True, vlsd=False)

    Reads data block from record definition

    Parameters: | record class channel group definition listing record channel classes info class contains blocks nameList : list of str, optional list of channel names sortedFlag : bool, optional flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading vlsd : bool indicate a sd block, compressed (DZ) or not (SD)
    Returns: | numpy recarray of data

  `pointerTodata`
  `read`(channelSet, info, filename)

    Reads data block

    Parameters: | channelSet : set of str set of channel names info : info object contains blocks structures filename name of file ot read

  `readRecord`(recordID, info, buf, channelSet=None)

    read record from a buffer

    Parameters: | recordID : int record identifier info class contains blocks buf : str buffer of data from file to be converted to channel raw data channelSet : set of str setof channel names to be read

  `type`

`mdfreader.mdf4reader.` `DATABlock`(record, info, parent_block, channelSet=None, nrecords=None, sortedFlag=True, vlsd=False)

  DATABlock converts raw data into arrays

  Parameters: | record : class record class instance describing a channel group record parent_block : class MDFBlock class containing at least parent block header channelSet : set of str, optional defines set of channels to only read, can be slow but saves memory, for big files nrecords: int, optional number of records to read sortedFlag : bool, optional flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading vlsd : bool indicate a sd block, compressed (DZ) or not (SD)
  Returns: | a recarray containing the channels data

  Notes

  This function will read DTBlock, RDBlock, DZBlock (compressed), RDBlock (VLSD), sorted or unsorted

`mdfreader.mdf4reader.` `equalizeStringLength`(buf)

  Makes all strings in a list having same length by appending spaces strings.

  Parameters: | buf : list of str
  Returns: | list of str elements all having same length

`mdfreader.mdf4reader.` `formulaConv`(vect, formula)

  apply formula conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `linearConv`(vect, cc_val)

  apply linear conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

class `mdfreader.mdf4reader.` `mdf4`(fileName=None, channelList=None, convertAfterRead=True, filterChannelNames=False, noDataLoading=False, compression=False, convertTables=False, metadata=2)

  Bases: `mdfreader.mdf.mdf_skeleton`

  mdf file reader class from version 4.0 to 4.1.1

  Attributes

  fileName | (str) file name
  MDFVersionNumber | (int) mdf file version number
  masterChannelList | (dict) Represents data structure: a key per master channel with corresponding value containing a list of channels One key or master channel represents then a data group having same sampling interval.
  multiProc | (bool) Flag to request channel conversion multi processed for performance improvement. One thread per data group.
  convertAfterRead | (bool) flag to convert raw data to physical just after read
  filterChannelNames | (bool) flag to filter long channel names from its module names separated by ‘.’
  file_metadata | (dict) file metadata with minimum keys : author, organisation, project, subject, comment, time, date

  Methods

  read4( fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True) | Reads mdf 4.x file data and stores it in dict
  _getChannelData4(channelName) | Returns channel numpy array
  _convertChannel4(channelName) | converts specific channel from raw to physical data according to CCBlock information
  _convertAllChannel4() | Converts all channels from raw data to converted data according to CCBlock information

  `apply_invalid_bit`(channel_name)

    Mask data of channel based on its invalid bit definition if there is

    Parameters: | channel_name : str Name of channel

  `getChannelName4`(name, path)

    finds mdf channel name from name and path

    Parameters: | name : str channel name path: str source path or name, or channel group name, source name or path

  `read4`(fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True, compression=False, metadata=2)

    Reads mdf 4.x file data and stores it in dict

    Parameters: | fileName : str, optional file name info : mdfinfo4.info4 class info4 class containing all MDF Blocks multiProc : bool flag to activate multiprocessing of channel data conversion channelList : list of str, optional list of channel names to be read If you use channelList, reading might be much slower but it will save you memory. Can be used to read big files convertAfterRead : bool, optional flag to convert channel after read, True by default If you use convertAfterRead by setting it to false, all data from channels will be kept raw, no conversion applied. If many float are stored in file, you can gain from 3 to 4 times memory footprint To calculate value from channel, you can then use method .getChannelData() compression : bool, optional flag to activate data compression with blosc metadata: int, optional, default = 2 Reading metadata has impact on performance, especially for mdf 4.x using xml. 2: minimal metadata reading (mostly channel blocks) 1: used for noDataLoading 0: all metadata reading, including Source Information, Attachment, etc..

  `write4`(fileName=None, compression=False)

    Writes simple mdf 4.1 file

    Parameters: | fileName : str, optional Name of file If file name is not input, written file name will be the one read with appended ‘_new’ string before extension compression : bool flag to store data compressed

    Notes

    All channels will be converted to physical data, so size might be bigger than original file

`mdfreader.mdf4reader.` `rationalConv`(vect, cc_val)

  apply rational conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `readUnsorted`(record, info, parent_block, channelSet=None)

`mdfreader.mdf4reader.` `read_sdblock`(signal_data_type, sdblock, sdblock_length)

  Reads vlsd channel from its SD Block bytes

  Parameters: | signal_data_type : int sdblock : bytes SD Block bytes sdblock_length: int SD Block data length (header not included)
  Returns: | array

class `mdfreader.mdf4reader.` `record`(dataGroup, channelGroup)

  Bases: `list`

  Methods

  `addChannel`(info, channelNumber) | add a channel in class
  `append`(object) |
  `clear`() |
  `copy`() |
  `count`(value) |
  `extend`(iterable) |
  `generate_chunks`() | calculate data split
  `index`(value, [start, [stop]]) | Raises ValueError if the value is not present.
  `initialise_recarray`(info, channelSet, nrecords) | Initialise recarray
  `insert` | L.insert(index, object) – insert object before index
  `loadInfo`(info) | gathers records related from info class
  `pop`([index]) | Raises IndexError if list is empty or index is out of range.
  `readRecordBuf`(buf, info[, channelSet]) | read stream of record bytes
  `readSortedRecord`(fid, info[, channelSet]) | reads record, only one channel group per datagroup
  `read_all_channels_sorted_record`(fid) | reads all channels from file using numpy fromstring, chunk by chunk
  `read_channels_from_bytes`(bita, info[, …]) | reads stream of record bytes using dataRead module if available otherwise bitarray
  `read_channels_from_bytes_fallback`(bita, info) | reads stream of record bytes using bitarray in case no dataRead available
  `read_not_all_channels_sorted_record`(fid, …) | reads channels from file listed in channelSet
  `remove`(value) | Raises ValueError if the value is not present.
  `reverse` | L.reverse() – reverse IN PLACE
  `sort`([key, reverse]) |

  `CANOpen`
  `CGrecordLength`
  `Flags`
  `MLSD`
  `VLSD`
  `VLSD_CG`
  `addChannel`(info, channelNumber)

    add a channel in class

    Parameters: | info : mdfinfo4.info4 class channelNumber : int channel number in mdfinfo4.info4 class

  `byte_aligned`
  `channelGroup`
  `channelNames`
  `dataGroup`
  `dataRecordName`
  `generate_chunks`()

    calculate data split

    Returns: | (nrecord_chunk, chunk_size)

  `hiddenBytes`
  `initialise_recarray`(info, channelSet, nrecords, dtype=None, channels_indexes=None)

    Initialise recarray

    Parameters: | info: info class channelSet : set of str, optional set of channel to read nrecords: int number of records dtype: numpy dtype, optional channels_indexes: list of int, optional
    Returns: | rec : numpy recarray contains a matrix of raw data in a recarray (attributes corresponding to channel name)

  `invalid_channel`
  `loadInfo`(info)

    gathers records related from info class

    Parameters: | info : mdfinfo4.info4 class

  `master`
  `numberOfRecords`
  `numpyDataRecordFormat`
  `readRecordBuf`(buf, info, channelSet=None)

    read stream of record bytes

    Parameters: | buf : stream stream of bytes read in file info class contains blocks structure channelSet : set of str, optional set of channel to read
    Returns: | rec : dict returns dictionary of channel with its corresponding values

  `readSortedRecord`(fid, info, channelSet=None)

    reads record, only one channel group per datagroup

    Parameters: | fid : file identifier pointer position in file of data block beginning channelSet : set of str, optional set of channel to read
    Returns: | rec : numpy recarray contains a matrix of raw data in a recarray (attributes corresponding to channel name)

    Notes

    If channelSet is None, read data using numpy.core.records.fromfile that is rather quick. However, in case of large file, you can use channelSet to load only interesting channels or only one channel on demand, but be aware it might be much slower.

  `read_all_channels_sorted_record`(fid)

    reads all channels from file using numpy fromstring, chunk by chunk

    Parameters: | fid : file identifier
    Returns: | rec : numpy recarray contains a matrix of raw data in a recarray (attributes corresponding to channel name)

  `read_channels_from_bytes`(bita, info, channelSet=None, nrecords=None, dtype=None, channels_indexes=None)

    reads stream of record bytes using dataRead module if available otherwise bitarray

    Parameters: | bita : stream stream of bytes info: info class channelSet : set of str, optional set of channel to read nrecords: int number of records dtype: numpy dtype channels_indexes: list of int
    Returns: | rec : numpy recarray contains a matrix of raw data in a recarray (attributes corresponding to channel name)

  `read_channels_from_bytes_fallback`(bita, info, channelSet=None, nrecords=None, dtype=None, channels_indexes=None)

    reads stream of record bytes using bitarray in case no dataRead available

    Parameters: | bita : stream stream of bytes info: info class channelSet : set of str, optional set of channel to read nrecords: int number of records dtype: numpy dtype channels_indexes: list of int
    Returns: | rec : numpy recarray contains a matrix of raw data in a recarray (attributes corresponding to channel name)

  `read_not_all_channels_sorted_record`(fid, info, channelSet)

    reads channels from file listed in channelSet

    Parameters: | fid : file identifier info: info class channelSet : set of str, optional set of channel to read
    Returns: | rec : numpy recarray contains a matrix of raw data in a recarray (attributes corresponding to channel name)

  `recordID`
  `recordIDCFormat`
  `recordIDsize`
  `recordLength`
  `recordToChannelMatching`

`mdfreader.mdf4reader.` `textToTextConv`(vect, cc_ref)

  apply text to text conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `textToValueConv`(vect, cc_val, cc_ref)

  apply text to value conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `valueRangeToTextConv`(vect, cc_val, cc_ref)

  apply value range to text conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `valueRangeToValueTableConv`(vect, cc_val)

  apply value range to value table conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `valueToTextConv`(vect, cc_val, cc_ref)

  apply value to text conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `valueToValueTableWInterpConv`(vect, cc_val)

  apply value to value table with interpolation conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value

`mdfreader.mdf4reader.` `valueToValueTableWOInterpConv`(vect, cc_val)

  apply value to value table without interpolation conversion to data

  Parameters: | vect : numpy 1D array raw data to be converted to physical value cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
  Returns: | converted data to physical value