ó ìŠ'Gc@sjdZddgZdZddlZddlmZdd d„ƒYZd „Zed krfeƒndS( s8 This class is based on the Python recipe titled "Language detection using character trigrams" http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/326576 by Douglas Bagnall. It has been (slightly) adapted by Eric van der Vlist to support Unicode and accept a method to parse strings. sDouglas Bagnalls%Eric van der Vlist tPythoniÿÿÿÿN(turlopentTrigramcBsneZdZdZd d„Zdd„Zd„Zdd„Zd„Z d „Z d „Z d „Z d „Z RS(s% From one or more text files, the frequency of three character sequences is calculated. When treated as a vector, this information can be compared to other trigrams, and the difference between them seen as an angle. The cosine of this angle varies between 1 for complete similarity, and 0 for utter difference. Since letter combinations are characteristic to a language, this can be used to determine the language of a body of text. For example: >>> reference_en = Trigram('/path/to/reference/text/english') >>> reference_de = Trigram('/path/to/reference/text/german') >>> unknown = Trigram('url://pointing/to/unknown/text') >>> unknown.similarity(reference_de) 0.4 >>> unknown.similarity(reference_en) 0.95 would indicate the unknown text is almost cetrtainly English. As syntax sugar, the minus sign is overloaded to return the difference between texts, so the above objects would give you: >>> unknown - reference_de 0.6 >>> reference_en - unknown # order doesn't matter. 0.05 As it stands, the Trigram ignores character set information, which means you can only accurately compare within a single encoding (iso-8859-1 in the examples). A more complete implementation might convert to unicode first. As an extra bonus, there is a method to make up nonsense words in the style of the Trigram's text. >>> reference_en.makeWords(30) My withillonquiver and ald, by now wittlectionsurper, may sequia, tory, I ad my notter. Marriusbabilly She lady for rachalle spen hat knong al elf Beware when using urls: HTML won't be parsed out. Most methods chatter away to standard output, to let you know they're still there. icCs)i|_|dk r%|j|ƒndS(N(tluttNonet parseFile(tselftfn((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt__init__As  s cCsRxK|D]C}|jj|iƒ}|j|dƒd||<|d|}qW|S(Nii(Rt setdefaulttget(Rtlinetpairtlettertd((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt_parseAFragmentFs  cCs|j|ƒ|jƒdS(N(Rtmeasure(Rtstring((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt parseStringMs s iso-8859-1cCs„d}d|kr!t|ƒ}n t|ƒ}x<t|ƒD].\}}|j|jƒj|ƒdƒ}q:W|jƒ|jƒdS(Ns s://t (Rtopent enumerateRtstriptdecodetcloseR(RRtencodingR tftzR ((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyRQs  & cCs^d}xD|jjƒD]3}|tg|jƒD]}||^q/ƒ7}qW|d|_dS(sXcalculates the scalar length of the trigram vector and stores it in self.length.igà?N(Rtvaluestsumtlength(Rttotaltytx((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyRas1c Cs¾t|tƒstdƒ‚n|j}|j}d}xm|jƒD]_}||krC||}||}x6|D]+}||krp|||||7}qpqpWqCqCWt|ƒ|j|jS(s—returns a number between 0 and 1 indicating similarity. 1 means an identical ratio of trigrams; 0 means no trigrams in common. s&can't compare Trigram with non-Trigrami(t isinstanceRt TypeErrorRtkeystfloatR( Rtothertlut1tlut2RtktatbR!((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt similarityis       $cCsd|j|ƒS(scindicates difference between trigram sets; 1 is entirely different, 0 is entirely the same.i(R,(RR&((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt__sub__}scCsig}d}xM|r[|j|ƒ}|j|ƒ|d|}|dkr|d8}qqWdj|ƒS(s:returns a string of made-up words based on the known text.s is t(tlikelytappendtjoin(RtcountttextR)tn((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt makeWordsƒs   cCsj||jkrdSg}x2|j|jƒD]\}}|j||ƒq-Wdj|ƒ}tj|ƒS(ssReturns a character likely to follow the given string two character string, or a space if nothing is found.RR.(RtitemsR0R1trandomtchoice(RR)tletterstv((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyR/s N(t__name__t __module__t__doc__RRRRRRRR,R-R5R/(((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyRs,        cCs9tdƒ}tdƒ}tdƒ}tdƒ}tdƒ}tdƒ}tdƒ}tdƒ}d GHd ||GHd ||GHd ||GHd ||GHd||GHd||GHd||GHd||GHd||GHd||GHd||GHd||GHd||GHd||GHd||GHd||GHdS(Ns-http://gutenberg.net/dirs/etext97/lsusn11.txts-http://gutenberg.net/dirs/etext03/candi10.txts3http://gutenberg.net/dirs/1/0/4/9/10492/10492-8.txts3http://gutenberg.net/dirs/1/2/8/4/12844/12844-8.txts3http://gutenberg.net/dirs/1/0/1/1/10117/10117-8.txts3http://gutenberg.net/dirs/1/3/0/4/13041/13041-8.txts-http://gutenberg.net/dirs/etext05/cfgsh10.txts3http://gutenberg.net/dirs/1/3/7/0/13704/13704-8.txtscalculating difference:s en - fr is %ss fr - en is %ssen - en2 is %ssen - fr2 is %ssfr - en2 is %ssfr - fr2 is %ssfr2 - en2 is %ssfi - fr is %ssfi - en is %ssfi - se is %ssno - se is %ssen - no is %ssno - no2 is %ssse - no2 is %ssen - no2 is %ssfr - no2 is %s(R(tentfrtfitnotsetno2ten2tfr2((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyttests2                       t__main__(( R=t __authors__t __license__R7turllibRRRFR;(((sX/home/sa3ruby/intertwingly.net/code/venus-bzr/examples/filters/guess-language/trigram.pyt s  Œ