Moving PTL to Python 3

• Links that might be useful

  • https://docs.python.org/3.0/whatsnew/3.0.html - Changes from Python 2 to Python 3

  • https://docs.python.org/2/library/2to3.html - Doc for Py2to3

  It is interesting to note that the tool Py2to3, always returns the same object that the function used in Python 2 did, even when it is unnecessary. A few examples of this are given below - 

  
  
  

  •   - dict(map(map_fun, cols, row))
     + dict(list(map(map_fun, cols, row)))

    • Python 2 returns a list when map is used and Python 3 returns a map object. The map object can be directly cast to a dictionary if needed.

  
  
  

  •   - data.iteritems():
     + data.items():

    • Python 2 returns a generator for dict.iteritems() so the tool converts this in Python 3 to dict.items()

  
  
  

  •   - data.items():
     + list(data.items()):

    • In this case however, dict.items() returns a list in Python 2 which is why the tool converted the generator returned by dict.items() explicitly to a list.

  
  
  

  •   - data.keys()
     + list(data.keys())

    • Similar to the previous case. Explicit conversion to a list in case of dict.keys()

  
  
  

  • base64bytes = base64.b64encode("**Random String**".encode('utf-8'))

    base64bytes.decode('utf-8')

    • This change is manual. Py2to3 does not take this into account. The b64encode function requires a byte-like object and returns one as well in Python 3, this was not the case for Python 2.

  
  
  

  •   - urllib2.Request(url, json.dumps(data))
     + urllib.request.Request(url, json.dumps(data))

  •  

    • urllib2 has changed to urllib and it's functions have been moved to sub packages.

       

  •   - httplib.HTTPConnection(self.host, timeout=25)
     + http.client.HTTPConnection(self.host, timeout=25)

    • httplib has changed to http and it's functions have been moved to sub packages.

  
  
  

  •   - filter(lambda x: x[0] == '**match**', filters):
     + [x for x in filters if x[0] == '**match**']:

    • Filter like map now returns a filter object. 

  
  
  

  • Exceptions in Python 3 no longer have a message attribute. This has been replaced with str(e). This change is also manual.

Specifically from a PTL stand point - 

• The defusedxml pypi page says it's for Python 2.7 but it is available through pip.

• The binaries have to have the .py extension, otherwise they will not be considered while running the tool for the workspace.

• There are issues while converting pbs_as and the conversion fails. This has to be done manually. 
  RefactoringTool: Skipping optional fixer: buffer
  RefactoringTool: Skipping optional fixer: idioms
  RefactoringTool: Skipping optional fixer: set_literal
  RefactoringTool: Skipping optional fixer: ws_comma
  RefactoringTool: Can't parse pbs_dev/pbspro/test/fw/bin/pbs_as: ParseError: bad input: type=1, value='elif', context=('\n ', (341, 4))
  RefactoringTool: No files need to be modified.
  RefactoringTool: There was 1 error:
  RefactoringTool: Can't parse pbs_dev/pbspro/test/fw/bin/pbs_as: ParseError: bad input: type=1, value='elif', context=('\n ', (341, 4))

Link to Forum - http://community.pbspro.org/t/moving-to-python-3/1421