Python


Last updated: Nov 15, 2016

If you are not interested in a short introduction on how I began using Python and the resources that helped me to learn it, you can jump directly to the Table of Contents

I would like to say a few words about Python. Hopefully this will not bore you too much nor turn into a "my programming language can beat up your programming language" style rant. I just picked up Python this year (2011) after being a die hard Perl fanboy for the past 10 or so years. I had learning Python (and Ruby) on my TODO list for awhile now, but since I was able to do everything I wanted with Perl, I kept it on the back burner. It wasn't until this year I wanted to help a friend with a Python program he wrote that I finally had reason to learn it. I am very happy I did. I really like Python. It's easy to write bad code in Perl, it is difficult in Python. Also I was finally able to learn Object Oriented programming by using Python. I tried it a few times in the past with Perl, and always quit in frustration. I am not sure I will go back to Perl. I will consider it for every project, but I really like the way Python does things. I am very grateful to have learned Perl because it allowed me to gain a good grasp of Python in about a solid week of reading and using it, but I think someone said it best that the name of Perl 6 should be called Python.

The below document is a result of the notes that I took while learning Python and the document that I reference frequently. If you are an experienced Python programmer, much of it will seem trivial to you. Also there may be some obvious things that you feel should not be here (ie. use sys.exit() instead of exit) but this is due to my years of using Perl and I need these reminders. I also tried to keep the document short on explanation and long on examples, since I learn best by example.

Finally, I would like to point out the resources that I have used to learn Python:

 



Table of Contents


  1. Reserved Keywords

  2. Documentation

  3. Comments

  4. Quotes

  5. Strings

  6. Lists

  7. Tuples

  8. Dictionaries

  9. Input/Output

  10. Iteration

  11. Functions
  12. Regular Expressions

  13. Exceptions

  14. Object Oriented Programming

  15. System Commands

  16. Date/Time

  17. Debugging

  18. Style

  19. Miscellaneous

  20. Differences Between Python 2 and 3

*Note1 - all examples where tested using Python 2.6.5 and Python 3 syntax is used as much as possible

*Note2 - I am not a professional programmer, so if I am doing anything horribly wrong in any of this document, please contact me so that I can fix it as soon as possible. I do not want to be teaching any Python newbies bad habits nor do I want to be known as the Python version of Matt's Scripts


 

 

Reserved Keywords


and as assert break class continue
def del elif else except exec
finally for from global if import
in is lambda not or pass
print raise return try while with
yield          

[Table of Contents]

 

Documentation

[Table of Contents]

 

Comments

[Table of Contents]

 

Quotes

[Table of Contents]

 

Strings
[Table of Contents]

 

Lists

Notes:

Examples:


[Table of Contents]

 

Tuples

Notes:

Examples:

[Table of Contents]

 

Dictionaries

Notes: Examples:

Advanced:

 

[Table of Contents]

 

 

Input/Output


[Table of Contents]

 

 

Iteration

[Table of Contents]

 

Functions

[Table of Contents]

 

Regular Expressions


[Table of Contents]

 

Exceptions

  • Tutorial: http://docs.python.org/tutorial/errors.html

  • Use a try..except block to catch errors and print them out in a friendly message. For example, this will print out a new error message if someone does not enter a required option on the command line:
  •   try:
        site = sys.argv[1]
      except IndexError :
        print ("\nYou need to enter something!)
      sys.exit()
      
  • An except cause may have multiple exceptions: except (RuntimeError, TypeError, NameError):

  • If you catch an exception, but do not want to do anything about it, you can use 'exit' (or 'return' if in a function)
      except someException:
          exit
      
  • It's a good idea to always catch KeyboardInterrupt (when someone press Ctrl+C or sends a kill signal):
      except KeyboardInterrupt:
          print("You interrupted the script! No exiting.\n")
      
  • There is an optional "else" clause which must follow all except clauses (when present). It is useful for code that must be executed if the try clause does not raise an exception

  • You can use "final" clause that will always execute whether an exception was raised or not. I typically use it to close any open files:
      try:
        do_something()
      except:
        exit
      finally:
        f.close()
      
[Table of Contents]

 

Object Oriented Programming

Object Oriented programming (OOP) is the opposite of procedural programming (a program that executes from top to bottom, usually in order). In OOP the object is the focus of the program. An object is usually a noun (animal, person, thing), but really it can be anything. These objects have properties (height, weight, age, etc).

Here are the general steps for OOP:
  • Define the properties of the object by using a class: class fruit:

  • Add properties to the class:
      >>> class fruit:
      ...  number_of_barrels = 0
      
  • Since you can not manipulate a class directly, you need to create an instance of the class, stored in a variable: apple = fruit()

  • You can access/manipulate the properties of an object by using a dot in between the instance and property: fruit.number_of_barrels = 3

  • The above can be printed like so:
      >>> print("Apple has {0} barrels.".format(apple.number_of_barrels))
      Apple has 3 barrels.
      
  • You can use functions in a class, but they are called "methods" and they are limited to the class. Also, they have to be indented within the class. To call them, you use the dot notation just like above with the variables

  • A method always has to have an argument between it's parentheses called 'self' (actually it can be called anything, but python convention is to use self): def pick(self):

  • Python uses 'self' to pass this argument to the method, so the following is passing 'apple' to the sleep method: apple.pick()

  • The following program will print out "ouch"
      >>> class fruit:
      ...  number_of_barrels = 0
      ...  def pick(self):
      ...    print("Ouch!")
      ...
      >>> apple = fruit()
      >>> apple.pick()
      Ouch!
      
  • Here is an example has two objects (apple & orange), assigns them the number of barrels, and then prints them out
      >>> class fruit:
       ...  number_of_barrels = 0 
       ...  def pick(self):
       ...    print("Ouch!")
       ...  def count_barrels(self):
       ...    print("I have {0} barrels".format(self.number_of_barrels))
       ...
       >>> apple = fruit()
       >>> apple.number_of_barrels = 7
       >>> apple.count_barrels()
       I have 7 barrels
    
       >>> orange = fruit()
       >>> orange.number_of_barrels = 9
       >>> orange.count_barrels()
       I have 9 barrels
      
  • There is a special method call __init__() that is used to initialize the properties when the object is created (everytime a new object of the class is created). You pass the arguments (variables) to init that you want to initialize
     
      >>> class fruit:
      ... def __init__(self, color, size, source):
      ...   self.color = color
      ...   self.size = size
      ...   self.source = source
      ... def __str__(self):
      ...   msg = "I'm a " + self.size + " " + self.color + " apple " + "from " 
      ...          +  self.source
      ...   return msg
      ...
      
  • Another special method is __str__() which is used to print an object (it overrides the "print" command). If you wanted to print out the arguments from the above class, here's how you would do it:
      >>> apple = fruit("red", "small", "Washington")
      >>> print(apple)
      I'm a small red apple from Washington
      
  • And a final special method is __repr__() which will execute when you print the class object:
       class BankAccount:
        balance = 0
        def __init__(self, name):
          self.name = name
          
        def __repr__(self):
          return("\nThis account belongs to {0} and the balance is {1:.2f}\n"
            .format(self.name, self.balance))
        
       acct = BankAccount("Joe")
       print(acct)
      
  • Polymorphism means that you can have two (or more) methods with the same name for different classes. Below is an example. Both classes have the method "totalBarrels"
      >>> class Apples:
      ...  def __init__(self, type, barrels):
      ...    self.type = type
      ...    self.barrels = barrels
      ...  def totalBarrels(self):
      ...    total_barrels = self.barrels + 1
      ...    return total_barrels
      ...
      >>> class Oranges:
      ...  def __init__(self, barrels):
      ...    self.barrels = barrels
      ...  def totalBarrels(self):
      ...    total_barrels = self.barrels + 2
      ...    return total_barrels
      ...
      >>> myApples = Apples('delicious', 10)
      >>> myOranges = Oranges(15)
      >>>
      >>> myApples.totalBarrels()
      11
      >>> myOranges.totalBarrels()
      17
      
  • Inheritance is making a new class (also called 'child class') based on the parent class (also called 'base class') and inheriting all of it's properties, methods, etc Here is a child class (apple) that inherits everything from it's parent class (fruit) and some examples of how they work together:
      >>> class fruit:
      ...  number_of_barrels = 0
      ...  def pick(self):
      ...    print("Ouch!")
      ...  def count_barrels(self):
      ...    print("I have {0} barrels".format(self.number_of_barrels))
      ...
      >>> class apple(fruit):
      ...  def pack(self):
      ...    print("It's crowded in here!")
      ...
      >>> delicious = apple()
      >>> delicious.number_of_barrels = 8
      >>> delicious.count_barrels()
      I have 8 barrels
      >>> delicious.pack()
      It's crowded in here!
      
  • One thing to note in the above example. The "delicious" object belongs to both the apple AND fruit class.

  • You can call a method from within another method by using the following syntax: self.method2()

  • A good way to structure an OOP program is to use the following layout:
      class myClass:
        def __init__(self, arg1, arg2):
          self.arg1 = arg1
          self.arg2 = arg2
    
        def __str__(self):
          return '(ARG1: " + str(self.arg1) + ", ARG2: " + str(self.arg2) + ")'
    
        def my_mod1(self):
          sum = arg1 + arg2
          return sum
    
        def my_mod2(self, arg3, arg4)
          if (arg3 == arg4):
            return True
          else:
            return False
    
      def main(arg3, arg4):
        myClassObj = myClass(arg1, arg2)
        status = myClassObj.my_mod2(arg3, arg4)
    
      if __name__ == '__main__':
        arg3 = 'some text'
        arg4 = 'some more text'
        main(arg3, arg4)
      
  • Example of a simple banking program (this was a Codecademy project for pro users):
       class BankAccount:
        balance = 0
        def __init__(self, name):
          self.name = name
          
        def __repr__(self):
          return("\nThis account belongs to {0} and the balance is {1:.2f}\n"
            .format(self.name, self.balance))
        
        def show_balance(self):
          print("\nThe balance is: {0:.2f}\n".format(self.balance))
          
        def deposit(self, amount):
          if amount <= 0:
            print("\nYou can not deposit 0 or less amount\n")
            return
          else:
            print("\nYou are depositing {0:.2f}\n".format(amount))
            self.balance += amount
            self.show_balance()
            
        def withdraw(self, amount):
          if amount > self.balance:
            print("\nYou can not withdrawl more then your balance\n")
            return
          else:
            print("\nYou are withdrawing {0:.2f}\n".format(amount))
            self.balance -= amount
            self.show_balance()
        
       acct = BankAccount("Joe")
       print(acct)
       acct.show_balance()
       acct.deposit(3000)
       acct.withdraw(700)
       print(acct)
      
    This should print:
        This account belongs to Joe and the balance is 0.00
    
        The balance is: 0.00
    
        You are depositing 3000.00
    
        The balance is: 3000.00
    
        You are withdrawing 700.00
    
        The balance is: 2300.00
    
        This account belongs to Joe and the balance is 2300.00
      
[Table of Contents]

 

System Commands

This section is to discuss how to execute system commands via python

Docs:

Notes:

  • Setting "shell=True" causes python to spawn an intermediate shell to execute the command vs the default of running the command directly. Using this intermediate shell causes variables, etc to be processed before the command is run

  • Uses "/bin/sh" by default. If you set "shell=True" you can change this (ie. executable="/bin/bash")

  • Use "process.qpoll()" to check if a child process has terminated then "print process.returncode" to get the return code (0 = good, anything else bad).
    Similarly "process.wait()" is very similar in that it waits for the child process to terminate

  • If you experience performance issues (ie: you are passing lots of data through a pipe), it is recommended that you try to enable buffering by setting bufsize to either -1 or a large enough positive value (such as 4096).

  • The only reason you would need to specify shell=True on Windows is when the command you wish to execute is actually built in to the shell, eg dir, copy. You don't need shell=True to run a batch file, nor to run a console-based executable.

Examples:

  • A few basic examples:
      >>>import subprocess
      >>>subprocess.call(['ls', '-l'], shell=True)
      >>>import subprocess
      >>>subprocess.Popen('echo "Hello world!"', shell=True)
      Hello world!
      
  • You can use pipes and redirects:
      >>>subprocess.Popen('echo "Hello world!" \
          | tr a-z A-Z 2> errors.txt', shell=True)
      HELLO WORLD!
      
  • You can use variables like so:
      >>> def print_string(string):
      ...  print(string)
      >>> print_string('Hello world!')
      Hello world!
    
      >>> def print_string(string):
      ...  subprocess.Popen('echo "{0}"'.format(string), shell=True)
      ... 
      >>> print_string('Hello world!')
      >>> Hello world!
      
  • To read stdout, you need to use the "subprocess.PIPE" option, so in the above example you would do this
      >>> p = subprocess.Popen('Hello World!', shell=True,
      ...  stdout=subprocess.PIPE)
      
  • To get the output you use "process.communicate()" The value returned from a call to communicate() is a tuple with two values, the first is the data from stdout and the second is the data from stderr.
      >>> print (p.communicate())
      ('Hello world!\n', None)
      
  • The reason the 2nd value was none, was because we did not tell it to redirect stderr. Here's how you can do that
      >>> string = "Hello World"
      >>> p = subprocess.Popen('echo "{0}"'.format(string), shell=True,
      ...  stdout=subprocess.PIPE, stderr=subprocess.PIPE)
      >>> print (p.communicate())
      ('Hello world!\n', '')
      
  • To print all stdout you can use:
      >>> print (p.communicate()[0])
      Hello World!
      
  • If you want to process something over each line of output:
      >>> cmd1 = "wmic qfe list brief /format:csv"
      >>> p = subprocess.Popen(cmd1, bufsize=-1, shell=True,
      ...  stdout=subprocess.PIPE, stderr=subprocess.PIPE)
      >>> for line in p.stdout.readlines():
      ...  print("Line: " + line)
      ... 
      
[Table of Contents]

 

Date/Time I use dates A LOT in my scripts. Here are some different ways of retrieving the date in Python.
  • Print a calendar using the month and year:
      >>> import calendar
      >>> calendar.prmonth(2013, 01)
          January 2013
      Mo Tu We Th Fr Sa Su
          1  2  3  4  5  6
       7  8  9 10 11 12 13
      14 15 16 17 18 19 20
      21 22 23 24 25 26 27
      28 29 30 31
      
  • Print all calendars for a given year
      >>> calendar.prcal(2013)
      
  • To parse a date string, use strftime:
       >>> from time import strftime
       >>> print(strftime("%a, %d %b %Y %H:%M:%S"))
       Thu, 10 Nov 2016 11:29:02
       
  • Save the various parts of a date to it's own variable:
        >>> day = (strftime("%b"))
        >>> print(day)
        Nov
        >>> day = (strftime("%d"))
        >>> print(day)
        10
        >>> month = (strftime("%b"))
        >>> print(month)
        Nov
        >>> year = (strftime("%Y"))
        >>> print(year)
        2016
       
  • [Table of Contents]

     

    Debugging

    Some debuggers:

    • This one integrates with vim http://jaredforsyth.com/projects/vim-debug/ Note, I have a section on using Vim as a Python IDE here

      A few notes on this debugger:
      • Type ":Dbg ." within vim to debug the current file (notice the period and the "D" has to be capitalized
      • Type ":Dbg" to get the help. Note, when you view the help, notice the shortcut commands (ie. \r for "run"). Using these will save you time debugging
      • Type ":Dbg quit" to quit
      • If you receive an error while debugging use ":e <filename>"
      • Move around the windows with Ctrl + (standard vi navigation key) (ie. to move to the window to the right, use Ctrl+l). These are mapped in the .vimrc file so if you prefer different keys to do this, change them there
      • For a list of help commands: ":Dbg help" or just ":Dbg"

    • This is a a good standalone debugger: http://pypi.python.org/pypi/pudb (*note - to run a file in pudb "python -m pudb.run temp.py"). Note, With pudb you use the arrow keys to move between windows.

    • A good intro to general debugging http://www.learncpp.com/cpp-tutorial/a4-debugging-your-program-stepping-and-breakpoints/

    General debugging terminology:

    • step into: executes code line by line. If it comes to a function, it will enter the function
    • step over: like step into, but if it's a function, executes function and returns (vs step into which enters the function and continues to proceed)
    • step out: executes the rest of the function (that you stepped into), and then returns control. Good for when you know a function is working properly
    • run to cursor: just like it sounds, it will execute the code until it reaches the cursor, useful in large programs
    • continue (also known as "run" or "go" in some debuggers) - runs into it hits the end of the program and terminate
    • breakpoints: stop execution at a certain point
    • watch: keep track of a variable or function as you step through your code (ie. if you are in a loop it will show the variable incrementing in the watch window)
    • call stack: shows you which line is responsible for calling the function

    [Table of Contents]

     

    Style

    Style guidelines per PEP8

    • Function names & variables should be lowercase, with words separated by underscores as necessary to improve readability.
    • Do not mix tabs and spaces (use vi to convert all tabs to spaces)
    • Use a space between two parameters of a function
    • Class names use the CapWords/CamelCase convention. Classes for internal use should include a leading underscore
    • You can install a pep8 checker with the following command: sudo easy_install pep8 and then execute: pep8 file.py to check the file against the pep8 guidelines.

    [Table of Contents]

     

    Miscellaneous

    • PyCIT - a collection of python templates for creating command line tools

    • PyInstaller - a program that converts Python programs into stand-alone executables

    • Using Vim as a Python IDE

    • How to install a package after downloading and uncompressing it:
      python setup.py install

    • How to get a list of installed modules: pydoc modules >> modules.txt

    • Python Package Index - a repository of software

    • Pip is a very usefull tool for installing and managing packages (pip is similar to Perl's CPAN). Here are some useful commands:

      • Install a package such as python-dateutil: pip install python-dateutil

      • Uninstall python-dateutil: pip uninstall python-dateutil

      • Check if there is a module for a certain keyword such as "date"
        pip search date

      • Pip help: pip -h

    • How to get the version of python installed: python -V

    • To strip new lines, use rstrip: url = url.rstrip('\n')

    • List comprehension is a good method of removing whitespace from a list of strings
        >>> freshfruit = ['  banana', ' loganberry  ', 'passion fruit  ']
        >>> [weapon.strip() for weapon in freshfruit]
        ['banana', 'loganberry', 'passion fruit']
        
    • To get command line options, use optparse for pre 2.7 and argparse for post 2.7

    • A good package to work with dns: dnspython

    • You can use the "time" module to benchmark your programs:
        import time
        t0 = time.clock()
        do_something()
        t1 = time.clock
        result = t0 - t1
        print("Result: {0:.3f}".format(result))
        
    • To create your own module, just save your program as my_program.py and place it in the path where Python can find it (see the next item for help with this) and simply call this program with import in your code "import my_program" and access the functions with "my_program.my_function()"

    • To find out where Python modules are located, you need to figure out your system's prefix (/usr for linux). You can do this with the following command:
        Linux:
        >>> import sys
        >>> sys.prefix
        '/usr'
      
        Windows:
        >>> import sys
        >>> sys.prefix
        'C:\\Python27'
        
    • To print your data structures (similar to Perls' Data::Dumper and VERY useful)
        import pprint
        pp = pprint.PrettyPrinter(indent=4)
        pp.pprint(<data_structure_name_here>)
        
    • To execute a program and drop you to a python shell, run your program with the "-i" switch: python -i myprogram.py

    [Table of Contents]

     

    Differences Between Python 2 and 3

    These are some of the common differences (note there is a 2to3 conversion tool available). Most of these are supported in 2.6 so it's a good idea to begin using them:

    • Print is now a function: print("Print something here")

    • dictionary methods: dict.keys(), dict.items() and dict.values() return "views" instead of lists. For example, this no longer works: k = d.keys(); k.sort(). Use k = sorted(d) instead

    • dict.iterkeys(), dict.iteritems() and dict.itervalues() methods are no longer supported

    • The "%" print style formatting will be supported, but will be going away in later versions, so it is a good idea to stop using it now. See this doc for more info on formatting: http://docs.python.org/tutorial/inputoutput.html

    Many more can be found here: http://docs.python.org/py3k/whatsnew/3.0.html


    [Table of Contents]