python - Unicode and UTF-8 encoding issue with Scrapy XPath selector text -

-

i'm using scrapy , python (as part of django project) scrape site german content. have libxml2 installed backend scrapy selectors.

if extract word 'hüftsitz' (this how displayed on site) through selectors, get: u'h\ufffd\ufffdftsitz' (scrapy xpath selectors return unicode strings).

if encode utf-8, get: 'h\xef\xbf\xbd\xef\xbf\xbdftsitz'. , if print that, 'h??ftsitz' isn't correct. wondering why may happening.

the character-set on site set utf-8. testing above on python shell sys.getdefaultencoding set utf-8. using django application data xpath selectors written mysql database utf-8 character set, see same behaviour.

am overlooking obvious here? clues or appreciated.

u'\ufffd' "unicode replacement character", printed question mark inside black triangle. not u umlaut. problem must somewhere upstream. check encoding web page headers being returned , verify in fact, says is.

the unicode replacement character inserted replacement illegal or unrecognized character, caused several things, likeliest encoding not claims be.


Comments

Post a Comment

Popular posts from this blog

how to build hyperlink for query string in php -

-
i have question, how can add variable in current url book.php?action=addressbook i want add book.php?action=addressbook&page=2 how generate hyperlink this, have tried using $_server['php_self'] query string not included in url showing that book.php?page=2 i want append other variables query string please you can use http_build_query(); append more params url $params = $_get; $params["item"] = 45; $new_query_string = http_build_query($params); for instance: $data = array('page'=> 34, 'item' => 45); echo http_build_query($data); //page=34&item=45 or include amp echo http_build_query($data, '', '&'); //page=34&&item=45
Read more

php - What is the difference between $_SERVER['PATH_INFO'] and $_SERVER['ORIG_PATH_INFO']? -

-
what’s difference between $_server['path_info'] , $_server['orig_path_info'] ? how use them? when run print_r($_server) , path_info , orig_path_info not present in array. why not? how can enable them? i have read php manual on them, still don’t understand them. the path_info variable present if invoke php script this: http://www.example.com/phpinfo.php/hello_there it's /hello_there part after .php script. if don't invoke url that, there won't $_server["path_info"] environment variable. the porig_ prefix uncommon. path_info standard cgi-environment variable, , should never prefixed. did read that? (there issues around php3/php4 if invoked php interpreter via cgi-bin/ - hardly has such setups today.) for reference: http://www.ietf.org/rfc/rfc3875
Read more

queue - mq_receive: message too long -

-
i implementing communication between 2 processes using queue. problem when call function mq_receive, error: message long. i have done following: struct mq_attr attr; long size = attr.mq_msgsize; .... // initializing queue "/gateway" int rc = mq_receive(gateway, buffer, size, &prio); if print size value, size=1, while when print same size program (got same mechanism), not long integer ( -1217186280 )... how can solve error?....so while size = 1, believe it's right "message long" why 1? p.s. have tried put : int rc = mq_receive(gateway, buffer, sizeof(buffer), &prio); but no result. it seems need read docs more carefully. when call mq_receive should pass size of destination buffer. this size must greater mq_msgsize attribute of queue . in addition, seems have error in queue attributes initialisation makes proper mq_receive call impossible. here standard message queue session: fill mq_attr struct ( doc ): struct mq_a...
Read more