Moves from optparse to argparse

[thisisdhaas]

See https://docs.python.org/2.7/library/argparse.html#upgrading-optparse-code for a checklist of things that should have been done (which I tried to follow), and http://stackoverflow.com/questions/3217673/why-use-argparse-rather-than-optparse for a discussion of why this is a good idea in general.

[nchammas]

I'm generally in favor of cleaning up old libraries like this, but there is at least one prerequisite that needs to be addressed: spark-ec2 currently supports Python 2.6.

I think that support should be dropped, but we need to agree on that and communicate it before we start using Python 2.7+ libraries like argparse. An alternative is to automatically download argparse like we do with boto.

[thisisdhaas]

Thanks for the response! I'm comfortable with either option--would the latter just involve adding argparse to the external_libs dict if the python version is < 2.7?

[shivaram]

Yeah if we can handle this using external_libs I'd be fine with that. However I also think that with Spark 2.0 we get a chance to announce some breaking changes, so we could use that as an opportunity to stop supporting Python 2.6

[nchammas]

I'd personally vote to drop Python 2.6 support, which would make future contributions to spark-ec2 just a bit easier to make. Python 2.6 was EOLed some years ago, so it's an easy deprecation IMO.

[shivaram]

Dropping support for Python 2.6 sounds good to me -- I just think its better to do it at a new version rather than in the middle of an existing branch.

Here is one proposed working model: We can create branch-2.0 in the spark-ec2 repo, but keep branch-1.6 as the default branch on github (for users who are just cloning the repo and launching clusters). We merge new PRs into branch-2.0 and flip the default branch when Spark 2.0 is released.

[nchammas]

Agreed. Dropping support for Python 2.6 with Spark 2.0 sounds good to me.

[thisisdhaas]

Okay, so I'll leave this alone until either a branch-2.0 exists or Spark 2.0 is released.

[etrain]

FWIW it looks like Spark 2.0 is going to drop support for python 2.6.

It looks like what you're planning here aligns with that, but you may want
to chime in on this JIRA: https://issues.apache.org/jira/browse/SPARK-12661
with these plans

On Wed, Apr 20, 2016 at 12:57 PM, Daniel Haas [email protected]
wrote:

Okay, so I'll leave this alone until either a branch-2.0 exists or Spark
2.0 is released.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#26 (comment)

[shivaram]

I just created a branch-2.0 to address #33 -- Lets open this PR against that branch.

[thisisdhaas]

Ok, sounds good. See #34 for the new version.