python - How to modify lxml autolink to be more liberal? -

-

i using autolink function of great lxml library documented here: http://lxml.de/api/lxml.html.clean-module.html

my problem detects urls start http://. use broader url detection regex one: http://daringfireball.net/2010/07/improved_regex_for_matching_urls

i tried make regex work lxml autolink function without success. end a:

lxml\html\clean.py", line 571, in _link_text host = match.group('host') indexerror: no such group

any python/regex gurus out there know how make work?

there 2 things in order adapt regexp lxml's autolink. first wrap entire url pattern match in group (?p<body> .. ) - lets lxml know goes inside href="" attribute.

next, wrap host part in (?<host> .. ) group , pass avoid_hosts=[] parameter when call autolink function. reason regexp pattern you're using doesn't find host (sometimes host part none) since matches partial urls , ambiguous url-like patterns.

i've modified regexp include above changes , given snippet test case:

import re import lxml.html import lxml.html.clean  url_regexp = re.compile(r"""(?i)\b(?p<body>(?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|(?p<host>[a-z0-9.\-]+[.][a-z]{2,4}/))(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))""")  doc = """<html><body>     http://foo.com/blah_blah     http://foo.com/blah_blah/.     http://www.extinguishedscholar.com/wpglob/?p=364.     http://✪df.ws/1234     rdar://1234     rdar:/1234     message://%3c330e7f840905021726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e     &lt;mailto:gruber@daringfireball.net?subject=test&gt; (including brokets).     bit.ly/foo </body></html>"""  tree = lxml.html.fromstring(doc) body = tree.find('body') lxml.html.clean.autolink(body, [url_regexp], avoid_hosts=[]) print lxml.html.tostring(tree)

output:

<html><body>     <a href="http://foo.com/blah_blah">http://foo.com/blah_blah</a>     <a href="http://foo.com/blah_blah/">http://foo.com/blah_blah/</a>.     <a href="http://www.extinguishedscholar.com/wpglob/?p=364">http://www.extinguishedscholar.com/wpglob/?p=364</a>.     <a href="http://%c3%a2%c2%9c%c2%aadf.ws/1234">http://&#226;&#156;&#170;df.ws/1234</a>     <a href="rdar://1234">rdar://1234</a>     <a href="rdar:/1234">rdar:/1234</a>     <a href="message://%3c330e7f840905021726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e">message://%3c330e7f840905021726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e</a>     &lt;<a href="mailto:gruber@daringfireball.net?subject=test">mailto:gruber@daringfireball.net?subject=test</a>&gt;     (including brackets).     <a href="bit.ly/foo">bit.ly/foo</a> </body></html>

Comments

Post a Comment

Popular posts from this blog

php - What is the difference between $_SERVER['PATH_INFO'] and $_SERVER['ORIG_PATH_INFO']? -

-
what’s difference between $_server['path_info'] , $_server['orig_path_info'] ? how use them? when run print_r($_server) , path_info , orig_path_info not present in array. why not? how can enable them? i have read php manual on them, still don’t understand them. the path_info variable present if invoke php script this: http://www.example.com/phpinfo.php/hello_there it's /hello_there part after .php script. if don't invoke url that, there won't $_server["path_info"] environment variable. the porig_ prefix uncommon. path_info standard cgi-environment variable, , should never prefixed. did read that? (there issues around php3/php4 if invoked php interpreter via cgi-bin/ - hardly has such setups today.) for reference: http://www.ietf.org/rfc/rfc3875
Read more

fortran - Function return type mismatch -

-
i'm attempting recode old c++ program in fortran make use of lapack (i'm aware c++ have lapack++, i'm having lot of trouble installing it, gave up). i didn't have problems compilation, when had variables declared real . when started coding section of program required lapack, found parameters passed dsyev need double precision . tried change double precision (including changing hard coded numbers double precision counterparts, i.e. 0.0 -> 0.0d0) when try compile, following error of functions , subroutines i've written: error: return type mismatch of function <function> @ (1) (real(4)/real(8)) i'm not sure coming from, in program has been changed double precision. for example, i've declared following: double precision :: alpha(3),d(3),zeta1,zeta2 double precision :: a1(3),a2(3),d1(3),d2(3) double precision :: pi pi = 3.14159265359d0 alpha = (/0.109818d0, 0.405771d0, 2.22766d0/) d = (/0.444635d0, 0.535328d0, 0.154329d0 /) 10 i=1,3 ...
Read more

queue - mq_receive: message too long -

-
i implementing communication between 2 processes using queue. problem when call function mq_receive, error: message long. i have done following: struct mq_attr attr; long size = attr.mq_msgsize; .... // initializing queue "/gateway" int rc = mq_receive(gateway, buffer, size, &prio); if print size value, size=1, while when print same size program (got same mechanism), not long integer ( -1217186280 )... how can solve error?....so while size = 1, believe it's right "message long" why 1? p.s. have tried put : int rc = mq_receive(gateway, buffer, sizeof(buffer), &prio); but no result. it seems need read docs more carefully. when call mq_receive should pass size of destination buffer. this size must greater mq_msgsize attribute of queue . in addition, seems have error in queue attributes initialisation makes proper mq_receive call impossible. here standard message queue session: fill mq_attr struct ( doc ): struct mq_a...
Read more