File: //lib/python2.7/site-packages/bs4/dammit.pyo
�
o�\[c @ sO d Z d Z d d l Z d d l m Z d d l Z d d l Z d d l Z d Z y d d l
Z
d � Z WnG e k
r� y d d l
Z
d � Z Wq� e k
r� d � Z q� Xn Xy d d l Z Wn e k
r� n Xe j d j � e j � Z e j d j � e j � Z d
e f d � � YZ d d d
� � YZ d d d � � YZ d S( sB Beautiful Soup bonus library: Unicode, Dammit
This library converts a bytestream to Unicode through any means
necessary. It is heavily based on code from Mark Pilgrim's Universal
Feed Parser. It works best on XML and HTML, but it does not rewrite the
XML or HTML to reflect a new encoding; that's the tree builder's job.
t MITi����N( t codepoint2namec C s t j | � d S( Nt encoding( t cchardett detect( t s( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt chardet_dammit s c C s t j | � d S( NR ( t chardetR ( R ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyR ! s c C s d S( N( t None( R ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyR ' s s! ^<\?.*encoding=['"](.*?)['"].*\?>s0 <\s*meta[^>]+charset\s*=\s*["']?([^>]*?)[ /;'">]t EntitySubstitutionc B s� e Z d Z d � Z e � \ Z Z Z i d d 6d d 6d d 6d d 6d
d 6Z e j d � Z
e j d
� Z e d � � Z
e d � � Z e d � � Z e e d � � Z e e d � � Z e d � � Z RS( sA Substitute XML or HTML entities for the corresponding characters.c C s� i } i } g } x\ t t j � � D]H \ } } t | � } | d k rc | j | � | | | <n | | | <q% Wd d j | � } | | t j | � f S( Ni" s [%s]t ( t listR t itemst unichrt appendt joint ret compile( t lookupt reverse_lookupt characters_for_ret codepointt namet charactert
re_definition( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt _populate_class_variables9 s
t apost 't quott "t ampt &t ltt <t gtt >s& ([<>]|&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;))s ([<>&])c C s# | j j | j d � � } d | S( Ni s &%s;( t CHARACTER_TO_HTML_ENTITYt gett group( t clst matchobjt entity( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt _substitute_html_entityZ s c C s | j | j d � } d | S( sm Used with a regular expression to substitute the
appropriate XML entity for an XML special character.i s &%s;( t CHARACTER_TO_XML_ENTITYR& ( R' R( R) ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt _substitute_xml_entity_ s c C sN d } d | k rB d | k r9 d } | j d | � } qB d } n | | | S( s* Make a value into a quoted XML attribute, possibly escaping it.
Most strings will be quoted using double quotes.
Bob's Bar -> "Bob's Bar"
If a string contains double quotes, it will be quoted using
single quotes.
Welcome to "my bar" -> 'Welcome to "my bar"'
If a string contains both single and double quotes, the
double quotes will be escaped, and the string will be quoted
using double quotes.
Welcome to "Bob's Bar" -> "Welcome to "Bob's bar"
R R s "( t replace( t selft valuet
quote_witht replace_with( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt quoted_attribute_valuef s c C s4 | j j | j | � } | r0 | j | � } n | S( s Substitute XML entities for special XML characters.
:param value: A string to be substituted. The less-than sign
will become <, the greater-than sign will become >,
and any ampersands will become &. If you want ampersands
that appear to be part of an entity definition to be left
alone, use substitute_xml_containing_entities() instead.
:param make_quoted_attribute: If True, then the string will be
quoted, as befits an attribute value.
( t AMPERSAND_OR_BRACKETt subR, R2 ( R' R/ t make_quoted_attribute( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt substitute_xml� s
c C s4 | j j | j | � } | r0 | j | � } n | S( s� Substitute XML entities for special XML characters.
:param value: A string to be substituted. The less-than sign will
become <, the greater-than sign will become >, and any
ampersands that are not part of an entity defition will
become &.
:param make_quoted_attribute: If True, then the string will be
quoted, as befits an attribute value.
( t BARE_AMPERSAND_OR_BRACKETR4 R, R2 ( R' R/ R5 ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt" substitute_xml_containing_entities� s
c C s | j j | j | � S( s� Replace certain Unicode characters with named HTML entities.
This differs from data.encode(encoding, 'xmlcharrefreplace')
in that the goal is to make the result more readable (to those
with ASCII displays) rather than to recover from
errors. There's absolutely nothing wrong with a UTF-8 string
containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that
character with "é" will make it more readable to some
people.
( t CHARACTER_TO_HTML_ENTITY_RER4 R* ( R' R ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt substitute_html� s ( t __name__t
__module__t __doc__R R$ t HTML_ENTITY_TO_CHARACTERR9 R+ R R R7 R3 t classmethodR* R, R2 t FalseR6 R8 R: ( ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyR 5 s&
%t EncodingDetectorc B s\ e Z d Z d e d d � Z d � Z e d � � Z e d � � Z
e e e d � � Z RS( s^ Suggests a number of possible encodings for a bytestring.
Order of precedence:
1. Encodings you specifically tell EncodingDetector to try first
(the override_encodings argument to the constructor).
2. An encoding declared within the bytestring itself, either in an
XML declaration (if the bytestring is to be interpreted as an XML
document), or in a <meta> tag (if the bytestring is to be
interpreted as an HTML document.)
3. An encoding detected through textual analysis by chardet,
cchardet, or a similar external library.
4. UTF-8.
5. Windows-1252.
c C s} | p g | _ | p g } t g | D] } | j � ^ q% � | _ d | _ | | _ d | _ | j | � \ | _ | _
d S( N( t override_encodingst sett lowert exclude_encodingsR t chardet_encodingt is_htmlt declared_encodingt strip_byte_order_markt markupt sniffed_encoding( R. RJ RB RG RE t x( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt __init__� s ( c C sO | d k rK | j � } | | j k r+ t S| | k rK | j | � t Sn t S( N( R RD RE R@ t addt True( R. R t tried( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt _usable� s
c c s t � } x+ | j D] } | j | | � r | Vq q W| j | j | � rW | j Vn | j d k r� | j | j | j � | _ n | j | j | � r� | j Vn | j d k r� t
| j � | _ n | j | j | � r� | j Vn x( d D] } | j | | � r� | Vq� q� Wd S( s< Yield a number of encodings that might work for this markup.s utf-8s windows-1252N( s utf-8s windows-1252( RC RB RQ RK RH R t find_declared_encodingRJ RG RF R ( R. RP t e( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt encodings� s$
c C s" d } t | t � r | | f St | � d k rg | d d k rg | d d !d k rg d } | d } n� t | � d k r� | d d k r� | d d !d k r� d } | d } ni | d d k r� d
} | d } nF | d d k r� d } | d } n# | d d
k rd } | d } n | | f S( sM If a byte-order mark is present, strip it and return the encoding it implies.i i s ��s s utf-16bes ��s utf-16lei s s utf-8s ��s utf-32bes �� s utf-32leN( R t
isinstancet unicodet len( R' t dataR ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyRI s*
"
"
c C s� | r t | � } } n% d } t d t t | � d � � } d } t j | d | �} | r~ | r~ t j | d | �} n | d k r� | j � d j d d � } n | r� | j � Sd S( s� Given a document, tries to find its declared encoding.
An XML encoding is declared at the beginning of the document.
An HTML encoding is declared in a <meta> tag, hopefully near the
beginning of the document.
i i g�������?t endposi t asciiR- N(
RW t maxt intR t xml_encoding_ret searcht html_meta_ret groupst decodeRD ( R' RJ RG t search_entire_documentt
xml_endpost html_endposRH t declared_encoding_match( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyRR + s
N( R; R< R= R R@ RM RQ t propertyRT R? RI RR ( ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyRA � s
!t
UnicodeDammitc B s� e Z d Z i d d 6d d 6Z d d d g Z g de g d � Z d � Z d
d � Z d
d � Z
e d
� � Z d � Z
d � Z i dd 6d d 6dd 6dd 6dd 6d d 6d
d# 6d d& 6d d) 6d
d, 6dd/ 6dd2 6dd5 6d6 d7 6dd: 6d6 d; 6d6 d< 6dd? 6ddB 6ddE 6ddH 6ddK 6ddN 6ddQ 6ddT 6ddW 6ddZ 6d d] 6d d` 6d6 da 6d dd 6d dg 6Z i� dh d 6d d 6di d 6dj d 6dk d 6dl d 6dm d# 6dn d& 6do d) 6dp d, 6dq d/ 6dr d2 6ds d5 6d6 d7 6dt d: 6d6 d; 6d6 d< 6du d? 6du dB 6dv dE 6dv dH 6dw dK 6dx dN 6dy dQ 6dz dT 6d{ dW 6d| dZ 6d} d] 6d~ d` 6d6 da 6d dd 6d� dg 6d d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6dq d� 6d� d� 6df d� 6d� d� 6d� d� 6d� d� 6d d� 6d� d� 6dx d� 6d� d� 6d� d� 6d� d� 6d� d� 6d d� 6d� d� 6d� d� 6dw d� 6di d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d6 d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6dw d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d� 6d� d 6d� d6d� d6d� d6d� d6dd6d� d6dd6Z iz d d
6d d 6d
d6dd6dd6dd6dd6dd6dd6dd 6d d 6d d 6d!d"6d#d$6d%d&6d'd(6d)d*6d+d,6d-d.6d/d06d1d26d3d46d5d66d7d86d9d:6d;d<6d=d>6d?d@6dAdB6dCdD6dEdF6dGdH6dIdJ6dKdL6dMdN6dOdP6dQdR6dSdT6dUdV6dWdX6dYdZ6d[d\6d]d^6d_d`6dadb6dcdd6dedf6dgdh6didj6dkdl6dmdn6dodp6dqdr6dsdt6dudv6dwdx6dydz6d{d|6d}d~6dd�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d� d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6d�d�6Z d!d"d#g Z e d�d�Z e d dZ e dd d� � Z RS($ s� A class for detecting the encoding of a *ML document and
converting it to a Unicode string. If the source encoding is
windows-1252, can replace MS smart quotes with their HTML or XML
equivalents.s mac-romant macintoshs shift-jiss x-sjiss windows-1252s
iso-8859-1s
iso-8859-2c C sn | | _ g | _ t | _ | | _ t j t � | _ t | | | | � | _
t | t � si | d k r� | | _
t | � | _ d | _ d S| j
j
| _
d } x? | j
j D]1 } | j
j
} | j | � } | d k r� Pq� q� W| sOxa | j
j D]P } | d k r | j | d � } n | d k r� | j j d � t | _ Pq� q� Wn | | _ | sjd | _ n d S( NR
RZ R- sS Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.( t smart_quotes_tot tried_encodingsR@ t contains_replacement_charactersRG t loggingt getLoggerR; t logRA t detectorRU RV RJ t unicode_markupR t original_encodingRT t
_convert_fromt warningRO ( R. RJ RB Ri RG RE t uR ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyRM X s> c C s� | j d � } | j d k r9 | j j | � j � } n� | j j | � } t | � t k r� | j d k r� d j � | d j � d j � } q� d j � | d j � d j � } n | j � } | S( s[ Changes a MS smart quote character to an XML or HTML
entity, or an ASCII character.i RZ t xmls &#xt ;R i ( R& Ri t MS_CHARS_TO_ASCIIR% t encodet MS_CHARSt typet tuple( R. t matcht origR4 ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt _sub_ms_char� s '' t strictc C s� | j | � } | s+ | | f | j k r/ d S| j j | | f � | j } | j d k r� | | j k r� d } t j | � } | j | j
| � } n y+ | j | | | � } | | _ | | _ Wn t
k
r� } d SX| j S( Ns ([�-�])( t
find_codecRj R R RJ Ri t ENCODINGS_WITH_SMART_QUOTESR R R4 R~ t _to_unicodeRq t Exception( R. t proposedt errorsRJ t smart_quotes_ret smart_quotes_compiledRt RS ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyRr � s"
c C s t | | | � S( s Given a string and its encoding, decodes the string into Unicode.
%encoding is a string recognized by encodings.aliases( RV ( R. RX R R� ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyR� � s c C s | j s
d S| j j S( N( RG R Ro RH ( R. ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt declared_html_encoding� s c C s� | j | j j | | � � pu | r? | j | j d d � � pu | r` | j | j d d � � pu | rr | j � pu | } | r� | j � Sd S( Nt -R
t _( t _codect CHARSET_ALIASESR% R- RD R ( R. t charsetR/ ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyR� � s !!
c C sE | s
| Sd } y t j | � | } Wn t t f k
r@ n X| S( N( R t codecsR t LookupErrort
ValueError( R. R� t codec( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyR� � s
t eurot 20ACs �t s �t sbquot 201As �t fnoft 192s �t bdquot 201Es �t hellipt 2026s �t daggert 2020s �t Daggert 2021s �t circt 2C6s �t permilt 2030s �t Scaront 160s �t lsaquot 2039s �t OEligt 152s �t ?s �s #x17Dt 17Ds �s �s �t lsquot 2018s �t rsquot 2019s �t ldquot 201Cs �t rdquot 201Ds �t bullt 2022s �t ndasht 2013s �t mdasht 2014s �t tildet 2DCs �t tradet 2122s �t scaront 161s �t rsaquot 203As �t oeligt 153s �s �s #x17Et 17Es �t YumlR
s �t EURt ,t fs ,,s ...t +s ++t ^t %t SR! t OEt ZR R t *R� s --t ~s (TM)R R# t oet zt Ys �t !s �t cs �t GBPs �t $s �t YENs �t |s �s �s ..s �s �s (th)s �s <<s �s �s �s (R)s �s �t os �s +-s �t 2s �t 3s �t acutes �Rt s �t Ps �s �s �t 1s �s �s >>s �s 1/4s �s 1/2s �s 3/4s �s �t As �s �s �s �s �s �t AEs �t Cs �t Es �s �s �s �t Is �s �s �s �t Ds �t Ns �t Os �s �s �s �s �s �s �t Us �s �s �s �s �t bs �t Bs �t as �s �s �s �s �s �t aes �s �RS s �s �s �s �t is �s �s �s �s �t ns �s �s �s �s �s �t /s �s �s �s �s �s �t ys �s �s �s €i� s ‚i� s ƒi� s „i� s …i� s †i� s ‡i� s ˆi� s ‰i� s Ši� s ‹i� s Œi� s Ži� s ‘i� s ’i� s “i� s ”i� s •i� s –i� s —i� s ˜i� s ™i� s ši� s ›i� s œi� s ži� s Ÿi� s i� s ¡i� s ¢i� s £i� s ¤i� s ¥i� s ¦i� s §i� s ¨i� s ©i� s ªi� s «i� s ¬i� s i� s ®i� s ¯i� s °i� s ±i� s ²i� s ³i� s ´i� s µi� s ¶i� s ·i� s ¸i� s ¹i� s ºi� s »i� s ¼i� s ½i� s ¾i� s ¿i� s Ài� s Ái� s Âi� s Ãi� s Äi� s Åi� s Æi� s Çi� s Èi� s Éi� s Êi� s Ëi� s Ìi� s Íi� s Îi� s Ïi� s Ði� s Ñi� s Òi� s Ói� s Ôi� s Õi� s Öi� s ×i� s Øi� s Ùi� s Úi� s Ûi� s Üi� s Ýi� s Þi� s ßi� s ài� i� s âi� s ãi� s äi� s åi� s æi� s çi� s èi� s éi� s êi� s ëi� s ìi� s íi� s îi� s ïi� s ði� s ñi� s òi� s ói� s ôi� s õi� s öi� s ÷i� s øi� s ùi� s úi� s ûi� s üi� s ýi� s þi� i i i i i����i t utf8c C s� | j d d � j � d
k r- t d � � n | j � d k rN t d � � n g } d } d } x| t | � k rd| | } t | t � s� t | � } n | | j k r| | j k rx� | j D]5 \ } } }
| | k r� | | k r� | |
7} Pq� q� Wqc | d
k rW| | j
k rW| j | | | !� | j | j
| � | d 7} | } qc | d 7} qc W| d k ru| S| j | | � d j | � S( s� Fix characters from one encoding embedded in some other encoding.
Currently the only situation supported is Windows-1252 (or its
subset ISO-8859-1), embedded in UTF-8.
The input must be a bytestring. If you've already converted
the document to Unicode, you're too late.
The output is a bytestring in which `embedded_encoding`
characters have been converted to their `main_encoding`
equivalents.
R� R� s windows-1252t windows_1252sP Windows-1252 and ISO-8859-1 are the only currently supported embedded encodings.R� s utf-8s4 UTF-8 is the only currently supported main encoding.i i� i R
( s windows-1252R� ( R� s utf-8(
R- RD t NotImplementedErrorRW RU R\ t ordt FIRST_MULTIBYTE_MARKERt LAST_MULTIBYTE_MARKERt MULTIBYTE_MARKERS_AND_SIZESt WINDOWS_1252_TO_UTF8R R ( R' t in_bytest
main_encodingt embedded_encodingt byte_chunkst chunk_startt post bytet startt endt size( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt detwingle s<
N( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( s #x17DR� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( R� R� ( s #x17ER� ( R� R
( R R� ( i� i� i ( i� i� i ( i� i� i ( R; R< R= R� R� R R@ RM R~ Rr R� Rf R� R� R� Ry Rw R� R� R� R� R? R ( ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyRg E sd
1
( ( ( R= t __license__R� t htmlentitydefsR R Rl t stringR t chardet_typeR R t ImportErrorR t iconv_codecR Rx R� R] R_ t objectR RA Rg ( ( ( s. /usr/lib/python2.7/site-packages/bs4/dammit.pyt <module> s6
�