fix(url): canonicalize_url('http://您好.中国:80/') failed

[hidva]

expect: 'http://xn--5usr0o.xn--fiqs8s:80/'
actual: 'http://xn--5usr0o.xn--:80-u68dy61b/'

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (master@34435d0). Learn more about missing BASE report.
Report is 279 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master      #98   +/-   ##
=========================================
  Coverage          ?   95.61%           
=========================================
  Files             ?        7           
  Lines             ?      593           
  Branches          ?      132           
=========================================
  Hits              ?      567           
  Misses            ?       17           
  Partials          ?        9           

Files with missing lines	Coverage Δ
w3lib/url.py	97.21% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hidva]

ERROR: pypy: InterpreterNotFound: pypy

[redapple]

A good catch @pp-qq !
This also affects safe_url_string() (and would be fixed too with your change to _safe_ParseResult())
This change needs an update to the tests. Could you add 1 or 2 for this case?

[redapple]

What do you think of using .hostname and .port attributes from the SplitResult instead of looking for a port number in netloc?

[redapple]

@pp-qq , the issue with pypy on Travis is unrelated. I'm trying to fix it in #99

[hidva]

ERROR: pypy: InterpreterNotFound: pypy
ERROR: The command "pip install -U tox twine wheel codecov" failed and exited with 2 during .

[redapple]

Thanks for the tests.
My #98 (comment) still holds though.
Have you considered using SplitResult .hostname and .port attributes?

[hidva]

Current behavior:

# Calling this function on an already "safe" URL will return the URL unmodified.
>>> w3lib.url.safe_url_string('https://www.baIdu.com')
'https://www.baIdu.com'  # I
>>> w3lib.url.canonicalize_url('http://www.baidu.com:80000000')
'http://www.baidu.com:80000000/'

when _encode_netloc() using SplitResult .hostname and .port attributes:

>>> w3lib.url.safe_url_string('https://www.baIdu.com')
'https://www.baidu.com'
>>> w3lib.url.canonicalize_url('http://www.baidu.com:80000000')
'http://www.baidu.com/'

because python 3.5 says:

hostname is Host name (lower case)...

Is this what we want?

[redapple]

I'm fine with lowercasing the hostname. But it is indeed a change in behavior.

[redapple]

>>> w3lib.url.canonicalize_url('http://www.baidu.com:80000000')
'http://www.baidu.com/'

what's the reason for the "lost" port number?

[hidva]

python 3.5 says: port is None when value if not valid:

Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from six.moves.urllib.parse import (urljoin, urlsplit, urlunsplit,
...                                     urldefrag, urlencode, urlparse,
...                                     quote, parse_qs, parse_qsl,
...                                     ParseResult, unquote, urlunparse)
>>> 
>>> t = urlparse('http://www.BaidU.com:80')  # valid port
>>> t.hostname, t.port
('www.baidu.com', 80)
>>> t = urlparse('http://www.BaidU.com:65536')  # invalid port
>>> t.hostname, t.port
('www.baidu.com', None)

And the new _encode_netloc():

def _encode_netloc(parts):
    """
    :type parts: ParseResult
    :rtype: unicode
    """
    try:
        hostname = to_unicode(parts.hostname.encode('idna'))
        netloc = hostname if parts.port is None else '%s:%s' % (hostname, parts.port)
        # lost the port number if parts.port is None
    except UnicodeError:
        netloc = parts.netloc
    return netloc

[redapple]

It's a shame that urlsplit and urlparse in Python 2.7 and 3.5 simply swallow an invalid port number.
Python 3.6 does report an error:

$ python
Python 3.6.2 (default, Aug 24 2017, 10:48:24) 
[GCC 6.3.0 20170406] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import urlsplit
>>> t = urlsplit('http://www.BaidU.com:654444')
>>> t.port
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/urllib/parse.py", line 169, in port
    raise ValueError("Port out of range 0-65535")
ValueError: Port out of range 0-65535

[nandgator]

Bumping to close outdated PR.