encutils module — cssutils 0.9.8 documentation

### Navigation

- index
- modules |
- next |
- previous |
- cssutils 0.9.8 documentation »

# `encutils` module

encutils - encoding detection collection for Python

Version: | 0.9.8
Author: | Christof Hoeke, see http://cthedot.de/encutils/
Contributor: | Robert Siemer, Fredrik Hedman <fredrik.hedman@me.com> ported to python3
Copyright: | 2005-2011: Christof Hoeke
License: | encutils has a dual-license, please choose whatever you prefer: encutils is published under the LGPL 3 or later encutils is published under the Creative Commons License. encutils is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. encutils is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with encutils. If not, see <http://www.gnu.org/licenses/>.

A collection of helper functions to detect encodings of text files (like HTML, XHTML, XML, CSS, etc.) retrieved via HTTP, file or string.

`getEncodingInfo()` is probably the main function of interest which uses other supplied functions itself and gathers all information together and supplies an `EncodingInfo` object.

example:

    >>> import encutils
    >>> info = encutils.getEncodingInfo(url='http://cthedot.de/encutils/')

    >>> str(info)
    utf-8

    >>> repr(info)
    <encutils.EncodingInfo object encoding='utf-8' mismatch=False at...>

    >>> info.logtext
    HTTP media_type: text/html
    HTTP encoding: utf-8
    Encoding (probably): utf-8 (Mismatch: False)

references
  XML

    RFC 3023 (http://www.ietf.org/rfc/rfc3023.txt)

    easier explained in
      - http://feedparser.org/docs/advanced.html
      - http://www.xml.com/pub/a/2004/07/21/dive.html
  HTML
    http://www.w3.org/TR/REC-html40/charset.html#h-5.2.2
TODO
  - parse @charset of HTML elements?
  - check for more texttypes if only text given

`encutils.` `buildlog`(logname='encutils', level='INFO', stream=<open file '<stderr>', mode 'w' at 0x0000000001CAE1E0>, filename=None, filemode='w', format='%(levelname)s\t%(message)s')

  Helper to build a basic log

  - if filename is given returns a log logging to filename with mode filemode
  - else uses a log streaming to stream which defaults to sys.stderr
  - level defines the level of the log
  - format defines the formatter format of the log

  Returns: | a log with the name logname

`encutils.` `encodingByMediaType`(media_type, log=None)

  Parameters: | media_type – a media type like “text/html”
  Returns: | a default encoding for given media_type. For example `"utf-8"` for `media_type="application/xml"`. If no default encoding is available returns `None`. Refers to RFC 3023 and HTTP MIME specification.

`encutils.` `getHTTPInfo`(response, log=None)

  Parameters: | response – a HTTP response object
  Returns: | `(media_type, encoding)` information from the response Content-Type HTTP header. (Case of headers is ignored.) May be `(None, None)` e.g. if no Content-Type header is available.

`encutils.` `getMetaInfo`(text, log=None)

  Parameters: | text – a byte string
  Returns: | `(media_type, encoding)` information from (first) X/HTML Content-Type `<meta>` element if available in text. XHTML format:
      <meta http-equiv="Content-Type"
            content="media_type;charset=encoding" />

`encutils.` `detectXMLEncoding`(fp, log=None, includeDefault=True)

  Attempt to detect the character encoding of the xml file given by a file object fp. fp must not be a codec wrapped file object! fp may be a string or unicode string though.

  Based on a recipe by Lars Tiede: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/363841 which itself is based on Paul Prescotts recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52257

  Returns: | if detection of the BOM succeeds, the codec name of the corresponding unicode charset is returned if BOM detection fails, the xml declaration is searched for the encoding attribute and its value returned. the “<” character has to be the very first in the file then (it’s xml standard after all). if BOM and xml declaration fail, utf-8 is returned according to XML 1.0.

`encutils.` `getEncodingInfo`(response=None, text=u'', log=None, url=None)

  Find all encoding related information in given text.

  Information in headers of supplied HTTPResponse, possible XML declaration and X/HTML `<meta>` elements are used.

  Parameters: | response – HTTP response object, e.g. via `urllib.urlopen('url')` text – a byte string to guess encoding for. XML prolog with encoding pseudo attribute or HTML meta element will be used to detect the encoding url – When given fetches document at url and all needed information. No reponse or text parameters are needed in this case. log – an optional logging logger to which messages may go, if no log given all log messages are available from resulting `EncodingInfo`
  Returns: | instance of `EncodingInfo`.

  How the resulting encoding is retrieved:

  XML

    RFC 3023 states if media type given in the Content-Type HTTP header is application/xml, application/xml-dtd, application/xml-external-parsed-entity, or any one of the subtypes of application/xml such as application/atom+xml or application/rss+xml etc then the character encoding is determined in this order:

    1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. the encoding given in the encoding attribute of the XML declaration within the document, or 3. utf-8.

    Mismatch possibilities:
      - HTTP + XMLdecla
      - HTTP + HTMLmeta
      application/xhtml+xml ?
        XMLdecla + HTMLmeta

    If the media type given in the Content-Type HTTP header is text/xml, text/xml-external-parsed-entity, or a subtype like text/Anything+xml, the encoding attribute of the XML declaration is ignored completely and the character encoding is determined in the order: 1. the encoding given in the charset parameter of the Content-Type HTTP header, or 2. ascii.

    No mismatch possible.

    If no media type is given the XML encoding pseuso attribute is used if present.

    No mismatch possible.

  HTML
    For HTML served as text/html:
      http://www.w3.org/TR/REC-html40/charset.html#h-5.2.2
    1. An HTTP “charset” parameter in a “Content-Type” field. (maybe defaults to ISO-8859-1, but should not assume this)
    2. A META declaration with “http-equiv” set to “Content-Type” and a value set for “charset”.
    3. The charset attribute set on an element that designates an external resource. (NOT IMPLEMENTED HERE YET)
    Mismatch possibilities:
      - HTTP + HTMLmeta
  TEXT
    For most text/* types the encoding will be reported as iso-8859-1. Exceptions are XML formats send as text/* mime type (see above) and text/css which has a default encoding of UTF-8.

`encutils.` `tryEncodings`(text, log=None)

  If installed uses chardet http://chardet.feedparser.org/ to detect encoding, else tries different encodings on text and returns the one that does not raise an exception which is not very advanced or may be totally wrong. The tried encoding are in order ‘ascii’, ‘iso-8859-1’, ‘windows-1252’ (which probably will never happen as ‘iso-8859-1’ can decode these strings too) and lastly ‘utf-8’.

  Parameters: | text – a byte string
  Returns: | Working encoding or `None` if no encoding does work at all. The returned encoding might nevertheless be not the one intended by the author as it is only checked if the text might be encoded in that encoding. Some texts might be working in “iso-8859-1” and “windows-1252” and “ascii” and “utf-8” and ...

class `encutils.` `EncodingInfo`

  All encoding related information, returned by `getEncodingInfo()`.

  Attributes filled:
    - `encoding`: The guessed encoding

        Encoding is the explicit or implicit encoding or None and always lowercase.

    - from HTTP response
        - `http_encoding`
        - `http_media_type`
    - from HTML <meta> element
        - `meta_encoding`
        - `meta_media_type`
    - from XML declaration
        - `xml_encoding`
    - `mismatch`: True if mismatch between XML declaration and HTTP

        header. Mismatch is True if any mismatches between HTTP header, XML declaration or textcontent (meta) are found. More detailed mismatch reports are written to the optional log or `logtext`

        Mismatches are not necessarily errors as preferences are defined. For details see the specifications.

    - `logtext`: if no log was given log reports are given here

### Navigation

- index
- modules |
- next |
- previous |
- cssutils 0.9.8 documentation »