EffectivePyPrgramming/EffectivePyEn

Effective Python Programming – OSCON 2005 Effective Python Programming Page 2 Effective Python Programming – OSCON 2005 Effective Programming ● Get the job done ● Better, more maintainable code ● Use the language's strengths ● Python is not: C++ Java Perl ... Page 3 Effective Python Programming – OSCON 2005 Laziness ● In development, laziness can be good ● Do things right the first time ● Don't re-invent every wheel ● Death to NIH Page 4 Effective Python Programming – OSCON 2005 Effective != Efficient ● Effective does not necessarily mean efficient ● Optimise for development time ● Then worry about performance ● We'll touch on efficient code, later Page 5 Effective Python Programming – OSCON 2005 Target ● Python 2.4 ● CPython Page 6 Effective Python Programming – OSCON 2005 Python Fundamentals ● Learning by tinkering ● Everything's an object ● Namespaces ● EAFP ● Duck Typing Page 7 Effective Python Programming – OSCON 2005 Short Attention Span Theatre ● Key things to remember ● Even if you sleep through the rest, you'll still get something Page 8 Effective Python Programming – OSCON 2005 S.A.S. Theatre ● Rule #1: Dictionaries ● Rule #2: See Rule #1 Page 9 Effective Python Programming – OSCON 2005 Programming with the hood up ● Introspection ● Experimentation Page 10 Effective Python Programming – OSCON 2005 Interactive Python bonanza% python Python 2.4.1 (#2, Mar 30 2005, 21:51:10) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" ... >>> Page 11 Effective Python Programming – OSCON 2005 help and dir ● help(obj) – formats docstrings ● dir(obj) – shows attributes Page 12 Effective Python Programming – OSCON 2005 help >>> import os >>> help(os.popen) Help on built-in function popen in module posix: popen(...) popen(command [, mode='r' [, bufsize]]) -> pipe Open a pipe to/from a command returning a file object. Page 13 Effective Python Programming – OSCON 2005 help... >>> help(8) Help on int object: class int(object) | int(x[, base]) -> integer | | Convert a string or number to an integer, if | possible. A floating point argument will be | truncated towards zero (this does not include a | string representation of a floating point | number!) When converting a string, use | the optional base. It is an error to supply a Page 14 Effective Python Programming – OSCON 2005 dir >>> import popen2 >>> dir(popen2) ['MAXFD', 'Popen3', 'Popen4', 'all', 'builtins', 'doc', 'file', 'name', '_active', '_cleanup', '_test', 'os', 'popen2', 'popen3', 'popen4', 'sys'] Page 15 Effective Python Programming – OSCON 2005 Everything is an object ● ints, strings, files ● functions, modules, classes ● Variables are just labels Page 16 Effective Python Programming – OSCON 2005 Objects vs Names ● Variables are references ● Just a name referring to an object ● Stored in a namespace (defaults to the local namespace) Page 17 Effective Python Programming – OSCON 2005 Namespaces... ● Namespaces are dictionaries! >>> import sys >>> def foo(a=1, b=2): ... c = "hello" ... print sys._getframe().f_locals ... >>> >>> foo(a=4) {'a': 4, 'c': 'hello', 'b': 2} Page 18 Effective Python Programming – OSCON 2005 Namespace lookup – Classic ● locals ● module globals ● built-ins Page 19 Effective Python Programming – OSCON 2005 Assignment ● Assignment goes to local namespace ● Unless 'global' statement Page 20 Effective Python Programming – OSCON 2005 global ● global only for assignment ● not needed for getting a value ● globals are slower! Page 21 Effective Python Programming – OSCON 2005 Namespaces – nested scopes ● statically nested scopes (nested function definitions) ● useful for lambda: def createUpdater(self): return lambda foo, bar: self.update(foo,bar) ● nested function calls example later of this Page 22 Effective Python Programming – OSCON 2005 EAFP ● Easier to Ask Forgiveness than Permission ● Very basic Python concept Page 23 Effective Python Programming – OSCON 2005 Permission... ● Permission: if hasattr(obj, 'value'): value = obj.value else: value = None ● Forgiveness: try: read = obj.value except AttributeError: value = None Page 24 Effective Python Programming – OSCON 2005 EAFP ● Exceptions are expensive ● Checks can also be expensive ● Case-by-case – how often is it expected to fail? Page 25 Effective Python Programming – OSCON 2005 Python Typing ● Weak vs Strong ● Static vs Dynamic ● C++/Java: strong static typing ● Python: strong dynamic typing Page 26 Effective Python Programming – OSCON 2005 Duck-Typing ● Walks like a duck ● ... quacks like a duck ● ... it's a duck Page 27 Effective Python Programming – OSCON 2005 Duck-Typing ● File-like objects ● Might only need 'read()' Page 28 Effective Python Programming – OSCON 2005 Duck-Typing – File objects def getData(obj): data = obj.read() print data f = open('file.txt') getData(f) ● Actually, that data file was gzipped: import gzip f = gzip.GzipFile('file.txt.gz') getData(f) Page 29 Effective Python Programming – OSCON 2005 More Duck-Typing ● The mapping interface (dictionary) ● Start with a dictionary ● Slot in a different implementation ● e.g. network, database, ... Page 30 Effective Python Programming – OSCON 2005 Interfaces ● zope.interface ● PyProtocols ● Assert that an object implements an interface ● Documentation ● Adaptation ● Future Python Page 31 Effective Python Programming – OSCON 2005 Structured Programming Page 32 Effective Python Programming – OSCON 2005 Control Flow ● Iterators ● Generators ● for/else ● try/finally ● try/except/else ● switch statement Page 33 Effective Python Programming – OSCON 2005 S.A.S. Theatre! ● enumerate for n in range(len(sequence)): element = sequence[n] ● instead: for n, element in enumerate(sequence): ● enumerate returns an iterator >>> print enumerate([]) <enumerate object at 0xb7df418c> Page 34 Effective Python Programming – OSCON 2005 Basic control flow ● while ● for ● try/except Page 35 Effective Python Programming – OSCON 2005 Iterators ● Returns the next item each time ● No need to have all items in memory ● More flexible Page 36 Effective Python Programming – OSCON 2005 Files are iterators ● Returns a line >>> for line in open('/etc/resolv.conf'): ... print "got line '%s'"%(line.strip()) ... got line 'nameserver 210.15.254.240' got line 'nameserver 210.15.254.241' got line 'nameserver 203.10.110.101' got line 'nameserver 203.17.103.1' Page 37 Effective Python Programming – OSCON 2005 Creating iterators ● iter() built-in ● turns a sequence into an iterator ● classes can have an iter method that returns an iterator Page 38 Effective Python Programming – OSCON 2005 More flexible for loops ● Call .next() to get the next item iterobj = iter(sequence) for item in iterobj: if item == 'extended': item = item + iterobj.next() Page 39 Effective Python Programming – OSCON 2005 Token streams tokens=['text:hello','style:bold','text:world', 'text:goodbye','style:italic','text:world'] tokens = iter(tokens) for tok in tokens: what,value = tok.split(':',1) if what == 'text': handlePlainToken(value) elif what == 'style': what, value = tok.next().split(':', 1) if style == 'bold': handleBoldToken(value) elif style == 'italic': handleItalicToken(value) Page 40 Effective Python Programming – OSCON 2005 Push Iterator ● Sometimes you want to check the next item, but not always consume it ● Push it back onto the iterator! ● Like stdio's getc()/ungetc() Page 41 Effective Python Programming – OSCON 2005 Push Iterator class PushIterator: def init(self, iter): self.iter = iter self.pushed = [] def push(self, value): self.pushed.append(value) def next(self): if self.pushed: return self.pushed.pop() else: return self.iter.next() def iter(self): return self Page 42 Effective Python Programming – OSCON 2005 Peek Iterator ● Sibling to PushIterator ● Peek at the next result (without consuming it) Page 43 Effective Python Programming – OSCON 2005 itertools ● high performance iterator manipulation ● functional programming Page 44 Effective Python Programming – OSCON 2005 Generators ● “Good artists copy, great artists steal” - Picasso ● Stolen from Icon ● functions containing 'yield' are a generator ● When called, returns a generator object ● ... which is an iterator Page 45 Effective Python Programming – OSCON 2005 Generators ● 'yield' passes a result to the caller ● execution is suspended ● when next() is called again, resumes where it left off Page 46 Effective Python Programming – OSCON 2005 Generators >>> def squares(start=1): ... while True: ... yield start * start ... start += 1 ... >>> sq = squares(4) >>> sq <generator object at 0xb7df440c> >>> sq.next() 16 >>> sq.next() 25 Page 47 Effective Python Programming – OSCON 2005 Generators ● finish when they fall off the end ● or 'return' ● Generators can't 'return' a value ● Generators can be called multiple times Page 48 Effective Python Programming – OSCON 2005 Multiple Generator Calls >>> s1 = squares(5) >>> s2 = squares(15) >>> print s1.next(), s2.next(), s1.next(), s2.next() 25 225 36 256 Page 49 Effective Python Programming – OSCON 2005 Generator Example ● DB-API ● cursor.fetchone() - get one row Inefficient! ● cursor.fetchall() - get all rows Could consume a lot of memory ● cursor.fetchmany(N) – get N rows Slightly fiddly Page 50 Effective Python Programming – OSCON 2005 Possible Solutions for row in cursor.fetchall(): processResult(row) row = cursor.fetchone() while row: processResult(row) row = cursor.fetchone() while True: rows = cursor.fetchmany(100) if not rows: break for row in rows: processResult(row) Page 51 Effective Python Programming – OSCON 2005 Generator Version def ResultIter(cursor, arraysize=1000): while True: results = cursor.fetchmany(arraysize) if not results: break for result in results: yield result ● Using this: for res in ResultIter(cursor): processRow(res) Page 52 Effective Python Programming – OSCON 2005 for/else ● for statements can have an else: clause ● executed when the for loop exhausts it's loop (no 'break', return or exception) Page 53 Effective Python Programming – OSCON 2005 for/else example for element in sequence: if elementMatched(element): correct = element break else: print "no elements matched" correct = None Page 54 Effective Python Programming – OSCON 2005 try/finally ● finally: clause is always executed ● resource cleanup lock = acquireLock() try: val = doSomeStuff() if val is None: raise ValueError('got None') elif val < 0: return finally: lock.release() Page 55 Effective Python Programming – OSCON 2005 try/finally ● Great for preventing those nightmare bugs ● “This should never happen” ● Chances are it will ● Program Defensively The Universe Hates You. ● Woefully underused Page 56 Effective Python Programming – OSCON 2005 try/except/else ● try/except can have an else: statement ● executed when no exception occurred Page 57 Effective Python Programming – OSCON 2005 try/except/else ● Put only the important bits in the try: block try: import gtk except: MyWindow = None else: MyWindow = gtk.Window() Page 58 Effective Python Programming – OSCON 2005 minimising code in a try: block ● This code has a problem: try: data = obj.read() except AttributeError: data = ● This masks any AttributeErrors in the read() method ● Source of hard-to-find bugs Page 59 Effective Python Programming – OSCON 2005 switch statement ● python has no switch/case ● if/elif/elif/elif/else ● use a dictionary Page 60 Effective Python Programming – OSCON 2005 Dispatch via dict if indata == 'FOZZIE': showFozzie() elif indata == 'KERMIT': showKermit() ... else: showUnknownMuppet() ● becomes: callDict = { 'FOZZIE': showFozzie, 'KERMIT': showKermit, ... } func = callDict.get(indata, showUnknownMuppet) func() Page 61 Effective Python Programming – OSCON 2005 Object Oriented Programming Page 62 Effective Python Programming – OSCON 2005 Using Classes Pythonically ● New-style vs Old-style ● More Ducks! ● isinstance ● inheritance, mixins ● access control ● Simplifying your APIs Page 63 Effective Python Programming – OSCON 2005 S.A.S. Theatre ● del is not your friend Page 64 Effective Python Programming – OSCON 2005 del often considered harmful ● C++/Java-ism ● del breaks garbage collector ● non-deterministic ● doesn't always get called usefully when Python is exiting ● use a weakref in some cases Page 65 Effective Python Programming – OSCON 2005 New-style vs Old-style ● Python has two types of class ● Old-style classes are the original ones ● New-style (in 2.2) fix issues with these Difference between C types and Python classes Can inherit from built-in types (e.g. dict) Some shiny new features Page 66 Effective Python Programming – OSCON 2005 New-style vs Old-style ● New style derive from 'object' ● New style classes in 2.2 ● Fix for implementation issues in original classes ● Many new features properties descriptors new ..... Page 67 Effective Python Programming – OSCON 2005 Use New Style Classes ● Most new code should use them ● Will become the default in Python 3.0 Page 68 Effective Python Programming – OSCON 2005 More Ducks! ● Objects supporting the mapping interface: dictionaries *dbm database files shelves db_row instances Page 69 Effective Python Programming – OSCON 2005 Sometimes you care ● There are times when you care what things are passed ● Check for the methods you need if hasattr(obj, 'read'): obj.read() else: raise ValueError ● Or use an Interface Page 70 Effective Python Programming – OSCON 2005 Interfaces ● Documentation ● Assertions def getAudioFromSource(obj): if not IAudio(obj): raise ValueError('expected audio, got %r'% obj) return audio.read() ● Adaptation Automatically adapt an IFoo to an IBar Page 71 Effective Python Programming – OSCON 2005 Interface Example class IAudio(Interface): Lowlevel interface to audio source/sink. def close(): Close the underlying audio device def reopen(): Reopen a closed audio device def isOpen(): Return True if underlying audio open def read(): Return a packet of audio from device. def write(data): Write audio to device. Page 72 Effective Python Programming – OSCON 2005 Using an Interface class ALSAAudio(object): implements(IAudio) def reopen(self): .... def close(self): .... def isOpen(self): .... alsa = ALSAAudio() IAudio.implementedBy(alsa) IAudio(alsa) Page 73 Effective Python Programming – OSCON 2005 Interfaces are dynamic ● Interfaces can be changed at runtime ● Assert that a 3 rd party object implements an interface ● Register an adapter from one Interface to another Page 74 Effective Python Programming – OSCON 2005 isinstance ● Checking obj.class is evil ● Breaks subclassing ● type(foo) is ancient, and clunky ● Use isinstance: if isinstance(num, basestring): num = int(num) if not isinstance(thing, (int, float)): raise ValueError('expected a number') Page 75 Effective Python Programming – OSCON 2005 inheritance, mixins ● Multiple inheritance is useful ● Mixin classes – assemble components into a class ● Select from multiple implementations Page 76 Effective Python Programming – OSCON 2005 Inheritance ● “Flat is better than nested” ● Excessive inheritance trees are painful ● Use inheritance when you need it Page 77 Effective Python Programming – OSCON 2005 Design for subclassing ● Use self.class instead of hardcoded class names class Item: def init(self, value): self.value = value def getNextItem(self): newItem = self.class() newItem.value = self.value + 1 Page 78 Effective Python Programming – OSCON 2005 Base class methods ● Call base class methods from subclassed methods! ● Protects you if parent class changes ● Particularly important for init class Monkey(Simian): def init(self, bananas=2, *args, **kwargs): self.bananas = bananas Simian.init(self, *args, **kwargs) Page 79 Effective Python Programming – OSCON 2005 Don't Assume You Know Best ● You don't know how people will need to use your code ● Think about how someone could extend your classes ● Design classes accordingly ● Document the methods ● Factor out the useful bits Page 80 Effective Python Programming – OSCON 2005 access control ● Another C++/Java-ism friend, public, private, ... ● Python: “Everyone's a consenting adult” ● Convention: leading underscore signals “implementation detail” ● Better yet: document the API of the class Page 81 Effective Python Programming – OSCON 2005 private names ● leading double underscore mangles names ● In theory useful to stop people stepping on implementation details ● In practice, annoying Personal goal for 2.5 is to remove all use from the stdlib Page 82 Effective Python Programming – OSCON 2005 Simplifying your APIs ● Damien Conway's “Sufficiently Advanced Magic” ● But not quite that advanced... Page 83 Effective Python Programming – OSCON 2005 Container object ● Get the objects in a container ● First attempt: for child in container.getChildren(): doSomethingWithObject(child) ● Unnecessary API ● Python already has a good way to spell this: for child in container: doSomethingWithObject(child) Page 84 Effective Python Programming – OSCON 2005 Container example ● Using the iterator protocol class Container(object): def init(self, *children): self.children = children def iter(self): return iter(self.children) cont = Container(1,2,3,4,5) for c in cont: print c Page 85 Effective Python Programming – OSCON 2005 special methods ● Used to implement operators ● Examples: str - string representation setitem - obj[key] = val ==> obj.setitem(key,val) add - add something to this object getattr - get an attribute of this object eq - object is being compared to another Page 86 Effective Python Programming – OSCON 2005 special methods, examples ● A + B First tries A.add(B) Then B.radd(A) (We're ignoring coerce) ● A == B In order: A.eq(B), B.eq(A), A.cmp(B), and then B.cmp(A) Page 87 Effective Python Programming – OSCON 2005 Make the API Intuitive ● Every new method name is something that has to be remembered documented ● Finite amount of brain space ● Use intuitive operators ● But don't get too clever Page 88 Effective Python Programming – OSCON 2005 C++-style cout ● A bit too magical to be recommended: class CppFile: def init(self, fp): self.fp = fp def lshift(self, someobj): self.fp.write(str(someobj)) return self import sys cout = CppFile(sys.stdout) cout << "hello” << “world\n" Page 89 Effective Python Programming – OSCON 2005 Demonstrating special methods class chatty: def init(self, name): self.name = name def getattr(self, what): print self.name, "asked for", what raise AttributeError(what) Page 90 Effective Python Programming – OSCON 2005 demonstration... >>> A = chatty('A') >>> B = chatty('B') >>> A + B A asked for coerce A asked for add B asked for coerce B asked for radd Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: unsupported operand type(s) for +: 'instance' and 'instance' >>> print A A asked for str A asked for repr <main.chatty instance at 0xb7df47cc> Page 91 Effective Python Programming – OSCON 2005 getattr warning ● Special-case special names ● Pain, otherwise ● Python uses lots of special methods ● e.g. repr to print def getattr(self, name): if name.startswith('') and name.endswith(''): raise AttributeError(name) return self.remoteObject.lookupName(name) Page 92 Effective Python Programming – OSCON 2005 Get/Set methods ● instead of: print someobject.getValue() someobject.setValue(5) ● use: print someobject.value someobject.value = 5 Page 93 Effective Python Programming – OSCON 2005 Get/Set methods ● Sometimes, attributes need to be computed ● Use a property! class Magic: def _getMagicNum(self): return findTheMagicNumber() def _setMagicNum(self, value) setTheMagicNumber(value) magicNumber = property(_getMagicNum, _setMagicNum) Page 94 Effective Python Programming – OSCON 2005 Properties ● Property takes 1-4 arguments: attribute = property(getmeth, setmeth, delmeth, doc=) ● set method and del method optional ● doc sets a docstring – use it monkeyCount = property(getMonkeys, doc='The count of monkeys in this barrel') ● Only use property when needed Attributes are a lot faster Page 95 Effective Python Programming – OSCON 2005 Descriptors ● property returns an object called a descriptor ● extremely powerful, but you'll have to read up on them yourself ● descriptors are how 'self' gets inserted as the first argument of a method ● classmethod and staticmethod are also descriptors Page 96 Effective Python Programming – OSCON 2005 staticmethod ● Very limited use ● Nearly always better to use a function Page 97 Effective Python Programming – OSCON 2005 classmethod ● Alternate constructors class TimeStamp(object): def init(self, h, m, s): self.h, self.m, self.s = h, m, s @classmethod def fromString(cls, string): h,m,s = [int(x) for x in string.split(':')] return cls(h, m, s) ● Alternate constructors make nicer API ● Not many other uses Page 98 Effective Python Programming – OSCON 2005 @classmethod !? ● Decorators ● Replaces this: def alternateCtor(cls, argumentList): ... alternateCtor = classmethod(alternateCtor) ● With this: @classmethod def alternateCtor(cls, argumentList): ... Page 99 Effective Python Programming – OSCON 2005 Decorators ● Decorators are passed the function object, return a new function object ● Remember to copy func_name and doc if you create a new function object! Page 100 Effective Python Programming – OSCON 2005 Laziness ● Python makes things easy ● But only if you let it ● Anytime you feel like you're doing too much, take a step back Page 101 Effective Python Programming – OSCON 2005 S.A.S. Theatre ● dict.setdefault(key, defaultval) ● sets a value (if not already set), returns value Page 102 Effective Python Programming – OSCON 2005 setdefault ● Turns this: if key in dictobj: dictobj[key].append(val) else: dictobj[key] = [val] ● Into this: dictobj.setdefault(key, []).append(val) Page 103 Effective Python Programming – OSCON 2005 Sequence Unpacking ● Most things Just Work: a, b, c = threetuple ● Swap two values: a, b = b, a ● Get and clear a value: val, self.val = self.val, None Page 104 Effective Python Programming – OSCON 2005 Stamping License Plates ● Any time you find yourself writing the same/similar code repeatedly, you're working too hard ● Use a template function (a closure) Page 105 Effective Python Programming – OSCON 2005 Boring repetitive code class Monkey: def displayNameAsHTML(self): cssClass = self.cssClass html='<div class=”%s”>%s</div>'%(cssClass, self.name) return html def displayBananasAsHTML(self): cssClass = self.cssClass html='<div class=”%s”>%s</div>'%(cssClass, self.bananas) return html Page 106 Effective Python Programming – OSCON 2005 Templated Monkey def _displayer(what): def template(self): val = getattr(self, what) cssClass = self.cssClass return '<div class=”%s”>%s</div>'%(cssClass, val) return html return template class Monkey: displayNameAsHTML = _displayer('name') displayBananasAsHTML = _displayer('bananas') Page 107 Effective Python Programming – OSCON 2005 Making Templates saner def _displayer(what): def template(self, what=what): ... template.func_name = 'display%sAsHtml'%( what.capitalize()) template.doc = 'Returns %s as HTML'%what return template Page 108 Effective Python Programming – OSCON 2005 Universal Newline Mode ● Windows, Mac, Unix have different line endings ● Annoying ● Open files with 'rU', and it just works fp = open('somefile.txt', 'rU') Page 109 Effective Python Programming – OSCON 2005 codecs.open ● codecs.open provides a file-like object in a particular encoding ● useful for reading and writing ● more duck typing Page 110 Effective Python Programming – OSCON 2005 Plan for Unicode ● Much easier if you start out allowing unicode ● Even if not, it's not too hard to retrofit Page 111 Effective Python Programming – OSCON 2005 basestring ● Parent class for str and unicode ● Useful in isinstance() checks >>> isinstance('moose', basestring) True >>> isinstance(u'møøse', basestring) True Page 112 Effective Python Programming – OSCON 2005 The Instant Guide to i18n ● i18n your strings: from gettext import gettext as _ _('Enter your name') ● Decode strings on input: bytes = "naïve" unistr = bytes.decode('iso8859-1') ● Encode unicode on output: print unistr.encode('iso8859-1', 'replace') Page 113 Effective Python Programming – OSCON 2005 Batteries Included ● Python ships with a large std library ● Don't re-invent the wheel Page 114 Effective Python Programming – OSCON 2005 When Things Go Pear-Shaped ● Debugging ● Testing Page 115 Effective Python Programming – OSCON 2005 S.A.S. Theatre ● The single most useful Python trick I know: import pdb; pdb.set_trace() ● When executed, drops into the Python debugger Page 116 Effective Python Programming – OSCON 2005 unittest ● Python standard unittest module ● Port of Junit ● Makes writing tests not-too-horrible ● API is still a bit cumbersome Page 117 Effective Python Programming – OSCON 2005 doctest ● More Pythonic ● Perfect for the lazy programmer ● Cut-n-paste from an interactive session ● Doubles as documentation and examples Page 118 Effective Python Programming – OSCON 2005 Running Tests ● A good test runner makes it more likely people will actually run the tests ● test.py (in Zope3) Page 119 Effective Python Programming – OSCON 2005 Dealing with tracebacks ● Sometimes, the default traceback isn't what you want ● traceback module try: .... except: e, v, tb = sys.exc_info() traceback.print_tb(t) Page 120 Effective Python Programming – OSCON 2005 cgitb ● Debugging CGI scripts can be horrible import cgitb; cgitb.enable() ● Displays nicely formatted tracebacks context variables arguments ● Can log in text or html, to the browser or files Page 121 Effective Python Programming – OSCON 2005 Making Life Hard For Yourself ● Things to avoid ● Recognising refactoring targets ● Learning from (bad) examples Page 122 Effective Python Programming – OSCON 2005 S.A.S. Theatre ● Bare except: clauses will nearly always bite you later ● Silently eating exceptions is bad ● except: should either log the exception, or re-raise it ● Just 'raise' will re-raise the current exception Page 123 Effective Python Programming – OSCON 2005 Consistent Coding Style ● Pick a style, and stick to it PEP 008 has one, or choose your own ● Use tools where necessary emacs python mode reindent.py ● Pick a consistent indent style! Page 124 Effective Python Programming – OSCON 2005 from module import * ● Don't ● Figuring out where a name comes from should not be painful Page 125 Effective Python Programming – OSCON 2005 Circular Imports ● Circular imports lead to subtle bugs: ● Module A: import moduleB ● Module B: import moduleA ● Defer imports until you need them to fix this Page 126 Effective Python Programming – OSCON 2005 more on exceptions ● Raising a new exception from an except: clause often makes debugging harder ● The new traceback will point at the except: block, not the original exception ● A bare 'raise' will re-raise the current exception Page 127 Effective Python Programming – OSCON 2005 Refactoring Targets ● Learn to spot potential problems Page 128 Effective Python Programming – OSCON 2005 Long Argument Lists ● Once a function gets past a couple of arguments, either: refactor it or use keyword arguments def someHideousFunction(firstname, surname, addr1, addr2, zipcode, city, state): .... someHideousFunction(firstname=”Anthony”, surname=”Baxter”, .... Page 129 Effective Python Programming – OSCON 2005 Format Strings ● Similar problem with format strings using % operator ● Use the dict form: addrInfo = dict(firstname='Anthony', surname='Baxter', ...) label = “””%(firstname)s %(surname)s %(addr1)s %(addr2)s %(city)s, %(state)s %(zipcode)s””” % addrInfo Page 130 Effective Python Programming – OSCON 2005 string.Template ● Even simpler >>> from string import Template >>> s = Template('$cheese is from ${country}') >>> print s.substitute(cheese='Gouda', country='the Netherlands') Gouda is from the Netherlands >>> print s.substitute(cheese='Gouda') Traceback (most recent call last): [...] KeyError: 'country' >>> print s.safe_substitute(cheese='Gouda') Gouda is from ${country} Page 131 Effective Python Programming – OSCON 2005 $ == Perl!?! ● The $ sign is only accepted in the strings passed to string.Template ● Lots of other languages use $ - it's the obvious choice Page 132 Effective Python Programming – OSCON 2005 key in sequence ● Any time you use 'in', check the RHS ● If a sequence, how big will it get? ● O(N) ● Use a dict, or a set Page 133 Effective Python Programming – OSCON 2005 Too many globals ● Overuse of 'global' usually indicates poor code ● Refactor into a class, storing the global values as attributes Page 134 Effective Python Programming – OSCON 2005 Learning by (bad) example ● Lots of Python code to learn from ● Some of it is old ● People pick up bad habits from this Page 135 Effective Python Programming – OSCON 2005 map/filter/reduce ● map and filter using a lambda should be replaced with listcomps or genexprs ● reduce: just don't use sum() replaced 90% of use cases other use cases usually leads to headscratching Page 136 Effective Python Programming – OSCON 2005 string module ● string methods are better >>> import string >>> name = 'anthony' >>> print string.upper(name) ANTHONY >>> print name.upper() ANTHONY ● Remember: strings are immutable, methods return a copy Page 137 Effective Python Programming – OSCON 2005 backslash line continuations ● Almost never needed ● Not needed if there's open braces ● So wrap in ( ) ● Works for import, too! (in 2.4) from package.subpkg.module import (Aname, Bname, Cname, Dname) Page 138 Effective Python Programming – OSCON 2005 has_key() ● key in dict ● dict.get(key, default) Page 139 Effective Python Programming – OSCON 2005 Circular references ● Not so much of a problem now (garbage collector) ● Remember: del stops GC! Page 140 Effective Python Programming – OSCON 2005 Regular Expressions ● Should not be the first hammer in your toolbox ● Sometimes, a real parser is better ● If you must use them... Page 141 Effective Python Programming – OSCON 2005 re.compile() ● re.compile() returns a compiled expression ● much, much faster than re-compiling each time ● nicer API, too Page 142 Effective Python Programming – OSCON 2005 named RE groups ● Use named RE groups rather than numbered ones r = re.compile('(?P<name>[ ^{]+) (?P<surname>[} ]+)') match = r.match('Anthony Baxter') match.group('surname') Page 143 Effective Python Programming – OSCON 2005 Defensive RE programming ● Check the string matches what you expect, first ● Debugging complex RE failures is a world of hurt Page 144 Effective Python Programming – OSCON 2005 Efficient Python Programming ● Making Python Go Fast Page 145 Effective Python Programming – OSCON 2005 S.A.S. Theatre ● function calls are slow Page 146 Effective Python Programming – OSCON 2005 Making your Python code fast ● dicts, list.sort() are highly tuned ● globals are slower than locals ● list.pop(0), list.insert(0, value) are slow reverse your list, or use collections.deque ● move fixed code out of the critical path Page 147 Effective Python Programming – OSCON 2005 profile/hotshot ● Figure out where the slow bits are ● Get the code right first (with unit tests!) and then optimise Page 148 Effective Python Programming – OSCON 2005 numeric/numarray ● Insanely optimised array operations ● If dealing with numbers, use them Page 149 Effective Python Programming – OSCON 2005 Pyrex ● Python dialect that's compiled to C ● Start with straight Python code, add declarations to use C types ● Only optimise the hot spots ● Much easier than trying to write straight C code extensions Page 150 Effective Python Programming – OSCON 2005 slots ● Optimisation trick ● Python objects store attributes in a dictionary ● slots is a fixed list ● reduces memory consumption when you have many small objects Page 151 Effective Python Programming – OSCON 2005 slots ● Don't use slots as some sort of type-checking hack ● slots and subclassing == hurt Page 152 Effective Python Programming – OSCON 2005 Tying it all back together ● Java DOM Node ● Good example of a bad Python API Page 153 Effective Python Programming – OSCON 2005 Node.getChildNodes() ● Returns the child nodes of the current Node. ● Either: Node.childNodes ● or Make Node an iterator that iterates through it's children Page 154 Effective Python Programming – OSCON 2005 getValue()/setValue() ● Instead, just use node.value ● Or use a property if value is computed, or needs to be checked when set Page 155 Effective Python Programming – OSCON 2005 isSameNode() ● Implement a eq method thisNode == otherNode ● Note that 'is' might not be right if you can have two Node instances pointing at the same part of the DOM Page 156 Effective Python Programming – OSCON 2005 compareDocumentPosition() ● Compares Node's document position to another Node. ● Instead, implement a .position attribute that can be compared: thisNode.position < otherNode.position ● Or even Node.lt, Node.gt Could be a bit too clever/magic Page 157 Effective Python Programming – OSCON 2005 parent/child reference cycle ● Remember, a parent child reference cycle isn't a problem (So long as Node doesn't implement a del method!) ● Still, don't create them unnecessarily References keep objects alive Page 158 Effective Python Programming – OSCON 2005 Wrapping Up ● Write Python code in Python ● Might involve un-learning some habits from other languages ● More information in the notes for these slides... Page 159 Effective Python Programming – OSCON 2005 ● Beautiful is better than ugly. ● Explicit is better than implicit. ● Simple is better than complex. ● Complex is better than complicated. ● Flat is better than nested. ● Sparse is better than dense. ● Readability counts. ● Special cases aren't special enough to break the rules. ● Although practicality beats purity. ● Errors should never pass silently. ● Unless explicitly silenced. ● In the face of ambiguity, refuse the temptation to guess. ● There should be one – and preferably only one – obvious way to do it. ● Although that way may not be obvious at first unless you're Dutch. ● Now is better than never. ● Although never is often better than *right* now. ● If the implementation is hard to explain, it's a bad idea. ● If the implementation is easy to explain, it may be a good idea. ● Namespaces are one honking great idea -- let's do more of those!