Copyright © 2012 Rafe Kettler
Version 1.17
-A PDF version of this guide can be obtained from my site or Github. The magic methods guide has a git repository at http://www.github.com/RafeKettler/magicmethods. Any issues can be reported +
A PDF version of this guide can be obtained from Github. The magic methods guide has a git repository at http://www.github.com/RafeKettler/magicmethods. Any issues can be reported there, along with comments, (or even contributions!).
__new__ and __init__ formed the constructor of the object, __del__ is the destructor. It doesn't implement behavior for the statement del x (so that code would not translate to x.__del__()). Rather, it defines behavior for when an object is garbage collected. It can be quite useful for objects that might require extra cleanup upon deletion, like sockets or file objects. Be careful, however, as there is no guarantee that __del__ will be executed if the object is still alive when the interpreter exits, so __del__ can't serve as a replacement for good coding practices (like always closing a connection when you're done with it. In fact, __del__ should almost never be used because of the precarious circumstances under which it is called; use it with caution!
Putting it all together, here's an example of __init__ and __del__ in action:
from os.path import join
+from os.path import join
-class FileObject:
- '''Wrapper for file objects to make sure the file gets closed on deletion.'''
+class FileObject:
+ '''Wrapper for file objects to make sure the file gets closed on deletion.'''
- def __init__(self, filepath='~', filename='sample.txt'):
- # open a file filename in filepath in read and write mode
- self.file = open(join(filepath, filename), 'r+')
+ def __init__(self, filepath='~', filename='sample.txt'):
+ # open a file filename in filepath in read and write mode
+ self.file = open(join(filepath, filename), 'r+')
- def __del__(self):
- self.file.close()
- del self.file
+ def __del__(self):
+ self.file.close()
+ del self.file
+
Making Operators Work on Custom Classes
One of the biggest advantages of using Python's magic methods is that they provide a simple way to make objects behave like built-in types. That means you can avoid ugly, counter-intuitive, and nonstandard ways of performing basic operators. In some languages, it's common to do something like this:
-if instance.equals(other_instance):
- # do something
+if instance.equals(other_instance):
+ # do something
+
You could certainly do this in Python, too, but this adds confusion and is unnecessarily verbose. Different libraries might use different names for the same operations, making the client do way more work than necessary. With the power of magic methods, however, we can define one method (__eq__, in this case), and say what we mean instead:
-if instance == other_instance:
- #do something
+if instance == other_instance:
+ #do something
+
That's part of the power of magic methods. The vast majority of them allow us to define meaning for operators so that we can use them on our own classes just like they were built in types.
@@ -95,25 +98,26 @@ Comparison magic methods
- Defines behavior for the greater-than-or-equal-to operator,
>=.
For an example, consider a class to model a word. We might want to compare words lexicographically (by the alphabet), which is the default comparison behavior for strings, but we also might want to do it based on some other criterion, like length or number of syllables. In this example, we'll compare by length. Here's an implementation:
-class Word(str):
- '''Class for words, defining comparison based on word length.'''
-
- def __new__(cls, word):
- # Note that we have to use __new__. This is because str is an immutable
- # type, so we have to initialize it early (at creation)
- if ' ' in word:
- print "Value contains spaces. Truncating to first space."
- word = word[:word.index(' ')] # Word is now all chars before first space
- return str.__new__(cls, word)
-
- def __gt__(self, other):
- return len(self) > len(other)
- def __lt__(self, other):
- return len(self) < len(other)
- def __ge__(self, other):
- return len(self) >= len(other)
- def __le__(self, other):
- return len(self) <= len(other)
+class Word(str):
+ '''Class for words, defining comparison based on word length.'''
+
+ def __new__(cls, word):
+ # Note that we have to use __new__. This is because str is an immutable
+ # type, so we have to initialize it early (at creation)
+ if ' ' in word:
+ print "Value contains spaces. Truncating to first space."
+ word = word[:word.index(' ')] # Word is now all chars before first space
+ return str.__new__(cls, word)
+
+ def __gt__(self, other):
+ return len(self) > len(other)
+ def __lt__(self, other):
+ return len(self) < len(other)
+ def __ge__(self, other):
+ return len(self) >= len(other)
+ def __le__(self, other):
+ return len(self) <= len(other)
+
Now, we can create two Words (by using Word('foo') and Word('bar')) and compare them based on length. Note, however, that we didn't define __eq__ and __ne__. This is because this would lead to some weird behavior (notably that Word('foo') == Word('bar') would evaluate to true). It wouldn't make sense to test for equality based on length, so we fall back on str's implementation of equality.
@@ -174,11 +178,13 @@ Normal arithmetic operators
Reflected arithmetic operators
You know how I said I would get to reflected arithmetic in a bit? Some of you might think it's some big, scary, foreign concept. It's actually quite simple. Here's an example:
-some_object + other
+some_object + other
+
That was "normal" addition. The reflected equivalent is the same thing, except with the operands switched around:
-other + some_object
+other + some_object
+
So, all of these magic methods do the same thing as their normal equivalents, except the perform the operation with other as the first operand and self as the second, rather than the other way around. In most cases, the result of a reflected operation is the same as its normal equivalent, so you may just end up defining __radd__ as calling __add__ and so on. Note that the object on the left hand side of the operator (other in the example) must not define (or return NotImplemented) for its definition of the non-reflected version of an operation. For instance, in the example, some_object.__radd__ will only be called if other does not define __add__.
@@ -214,8 +220,9 @@ Reflected arithmetic operators
Augmented assignment
Python also has a wide variety of magic methods to allow custom behavior to be defined for augmented assignment. You're probably already familiar with augmented assignment, it combines "normal" operators with assignment. If you still don't know what I'm talking about, here's an example:
-x = 5
-x += 1 # in other words x = x + 1
+x = 5
+x += 1 # in other words x = x + 1
+
Each of these methods should return the value that the variable on the left hand side should be assigned to (for instance, for a += b, __iadd__ might return a + b, which would be assigned to a). Here's the list:
@@ -303,39 +310,41 @@ Controlling Attribute Access
- After all this,
__getattribute__ fits in pretty well with its companions __setattr__ and __delattr__. However, I don't recommend you use it. __getattribute__ can only be used with new-style classes (all classes are new-style in the newest versions of Python, and in older versions you can make a class new-style by subclassing object. It allows you to define rules for whenever an attribute's value is accessed. It suffers from some similar infinite recursion problems as its partners-in-crime (this time you call the base class's __getattribute__ method to prevent this). It also mainly obviates the need for __getattr__, which, when __getattribute__ is implemented, only gets called if it is called explicitly or an AttributeError is raised. This method can be used (after all, it's your choice), but I don't recommend it because it has a small use case (it's far more rare that we need special behavior to retrieve a value than to assign to it) and because it can be really difficult to implement bug-free.
You can easily cause a problem in your definitions of any of the methods controlling attribute access. Consider this example:
-def __setattr__(self, name, value):
- self.name = value
- # since every time an attribute is assigned, __setattr__() is called, this
- # is recursion.
- # so this really means self.__setattr__('name', value). Since the method
- # keeps calling itself, the recursion goes on forever causing a crash
+def __setattr__(self, name, value):
+ self.name = value
+ # since every time an attribute is assigned, __setattr__() is called, this
+ # is recursion.
+ # so this really means self.__setattr__('name', value). Since the method
+ # keeps calling itself, the recursion goes on forever causing a crash
-def __setattr__(self, name, value):
- self.__dict__[name] = value # assigning to the dict of names in the class
- # define custom behavior here
+def __setattr__(self, name, value):
+ self.__dict__[name] = value # assigning to the dict of names in the class
+ # define custom behavior here
+
Again, Python's magic methods are incredibly powerful, and with great power comes great responsibility. It's important to know the proper way to use magic methods so you don't break any code.
So, what have we learned about custom attribute access in Python? It's not to be used lightly. In fact, it tends to be excessively powerful and counter-intuitive. But the reason why it exists is to scratch a certain itch: Python doesn't seek to make bad things impossible, but just to make them difficult. Freedom is paramount, so you can really do whatever you want. Here's an example of some of the special attribute access methods in action (note that we use super because not all classes have an attribute __dict__):
-class AccessCounter(object):
- '''A class that contains a value and implements an access counter.
- The counter increments each time the value is changed.'''
+class AccessCounter(object):
+ '''A class that contains a value and implements an access counter.
+ The counter increments each time the value is changed.'''
- def __init__(self, val):
- super(AccessCounter, self).__setattr__('counter', 0)
- super(AccessCounter, self).__setattr__('value', val)
+ def __init__(self, val):
+ super(AccessCounter, self).__setattr__('counter', 0)
+ super(AccessCounter, self).__setattr__('value', val)
- def __setattr__(self, name, value):
- if name == 'value':
- super(AccessCounter, self).__setattr__('counter', self.counter + 1)
- # Make this unconditional.
- # If you want to prevent other attributes to be set, raise AttributeError(name)
- super(AccessCounter, self).__setattr__(name, value)
+ def __setattr__(self, name, value):
+ if name == 'value':
+ super(AccessCounter, self).__setattr__('counter', self.counter + 1)
+ # Make this unconditional.
+ # If you want to prevent other attributes to be set, raise AttributeError(name)
+ super(AccessCounter, self).__setattr__(name, value)
- def __delattr__(self, name):
- if name == 'value':
- super(AccessCounter, self).__setattr__('counter', self.counter + 1)
- super(AccessCounter, self).__delattr__(name)
+ def __delattr__(self, name):
+ if name == 'value':
+ super(AccessCounter, self).__setattr__('counter', self.counter + 1)
+ super(AccessCounter, self).__delattr__(name)
+
Making Custom Sequences
@@ -365,55 +374,56 @@ The magic behind containers
An example
For our example, let's look at a list that implements some functional constructs that you might be used to from other languages (Haskell, for example).
-class FunctionalList:
- '''A class wrapping a list with some extra functional magic, like head,
- tail, init, last, drop, and take.'''
-
- def __init__(self, values=None):
- if values is None:
- self.values = []
- else:
- self.values = values
-
- def __len__(self):
- return len(self.values)
-
- def __getitem__(self, key):
- # if key is of invalid type or value, the list values will raise the error
- return self.values[key]
-
- def __setitem__(self, key, value):
- self.values[key] = value
-
- def __delitem__(self, key):
- del self.values[key]
-
- def __iter__(self):
- return iter(self.values)
-
- def __reversed__(self):
- return reversed(self.values)
-
- def append(self, value):
- self.values.append(value)
- def head(self):
- # get the first element
- return self.values[0]
- def tail(self):
- # get all elements after the first
- return self.values[1:]
- def init(self):
- # get elements up to the last
- return self.values[:-1]
- def last(self):
- # get last element
- return self.values[-1]
- def drop(self, n):
- # get all elements except first n
- return self.values[n:]
- def take(self, n):
- # get first n elements
- return self.values[:n]
+class FunctionalList:
+ '''A class wrapping a list with some extra functional magic, like head,
+ tail, init, last, drop, and take.'''
+
+ def __init__(self, values=None):
+ if values is None:
+ self.values = []
+ else:
+ self.values = values
+
+ def __len__(self):
+ return len(self.values)
+
+ def __getitem__(self, key):
+ # if key is of invalid type or value, the list values will raise the error
+ return self.values[key]
+
+ def __setitem__(self, key, value):
+ self.values[key] = value
+
+ def __delitem__(self, key):
+ del self.values[key]
+
+ def __iter__(self):
+ return iter(self.values)
+
+ def __reversed__(self):
+ return reversed(self.values)
+
+ def append(self, value):
+ self.values.append(value)
+ def head(self):
+ # get the first element
+ return self.values[0]
+ def tail(self):
+ # get all elements after the first
+ return self.values[1:]
+ def init(self):
+ # get elements up to the last
+ return self.values[:-1]
+ def last(self):
+ # get last element
+ return self.values[-1]
+ def drop(self, n):
+ # get all elements except first n
+ return self.values[n:]
+ def take(self, n):
+ # get first n elements
+ return self.values[:n]
+
There you have it, a (marginally) useful example of how to implement your own sequence. Of course, there are more useful applications of custom sequences, but quite a few of them are already implemented in the standard library (batteries included, right?), like Counter, OrderedDict, and NamedTuple.
@@ -434,24 +444,26 @@ Callable Objects
- Allows an instance of a class to be called as a function. Essentially, this means that
x() is the same as x.__call__(). Note that __call__ takes a variable number of arguments; this means that you define __call__ as you would any other function, taking however many arguments you'd like it to.
__call__ can be particularly useful in classes with instances that need to often change state. "Calling" the instance can be an intuitive and elegant way to change the object's state. An example might be a class representing an entity's position on a plane:
-class Entity:
- '''Class to represent an entity. Callable to update the entity's position.'''
+class Entity:
+ '''Class to represent an entity. Callable to update the entity's position.'''
- def __init__(self, size, x, y):
- self.x, self.y = x, y
- self.size = size
+ def __init__(self, size, x, y):
+ self.x, self.y = x, y
+ self.size = size
- def __call__(self, x, y):
- '''Change the position of the entity.'''
- self.x, self.y = x, y
+ def __call__(self, x, y):
+ '''Change the position of the entity.'''
+ self.x, self.y = x, y
- # snip...
+ # snip...
+
Context Managers
In Python 2.5, a new keyword was introduced in Python along with a new method for code reuse: the with statement. The concept of context managers was hardly new in Python (it was implemented before as a part of the library), but not until PEP 343 was accepted did it achieve status as a first-class language construct. You may have seen with statements before:
-with open('foo.txt') as bar:
- # perform some action with bar
+with open('foo.txt') as bar:
+ # perform some action with bar
+
Context managers allow setup and cleanup actions to be taken for objects when their creation is wrapped with a with statement. The behavior of the context manager is determined by two magic methods:
@@ -462,39 +474,41 @@ Context Managers
- Defines what the context manager should do after its block has been executed (or terminates). It can be used to handle exceptions, perform cleanup, or do something always done immediately after the action in the block. If the block executes successfully,
exception_type, exception_value, and traceback will be None. Otherwise, you can choose to handle the exception or let the user handle it; if you want to handle it, make sure __exit__ returns True after all is said and done. If you don't want the exception to be handled by the context manager, just let it happen.
__enter__ and __exit__ can be useful for specific classes that have well-defined and common behavior for setup and cleanup. You can also use these methods to create generic context managers that wrap other objects. Here's an example:
-class Closer:
- '''A context manager to automatically close an object with a close method
- in a with statement.'''
+class Closer:
+ '''A context manager to automatically close an object with a close method
+ in a with statement.'''
- def __init__(self, obj):
- self.obj = obj
+ def __init__(self, obj):
+ self.obj = obj
- def __enter__(self):
- return self.obj # bound to target
+ def __enter__(self):
+ return self.obj # bound to target
- def __exit__(self, exception_type, exception_val, trace):
- try:
- self.obj.close()
- except AttributeError: # obj isn't closable
- print 'Not closable.'
- return True # exception handled successfully
+ def __exit__(self, exception_type, exception_val, trace):
+ try:
+ self.obj.close()
+ except AttributeError: # obj isn't closable
+ print 'Not closable.'
+ return True # exception handled successfully
+
Here's an example of Closer in action, using an FTP connection to demonstrate it (a closable socket):
->>> from magicmethods import Closer
->>> from ftplib import FTP
->>> with Closer(FTP('ftp.somesite.com')) as conn:
-... conn.dir()
-...
-# output omitted for brevity
->>> conn.dir()
-# long AttributeError message, can't use a connection that's closed
->>> with Closer(int(5)) as i:
-... i += 1
-...
-Not closable.
->>> i
-6
+>>> from magicmethods import Closer
+>>> from ftplib import FTP
+>>> with Closer(FTP('ftp.somesite.com')) as conn:
+... conn.dir()
+...
+# output omitted for brevity
+>>> conn.dir()
+# long AttributeError message, can't use a connection that's closed
+>>> with Closer(int(5)) as i:
+... i += 1
+...
+Not closable.
+>>> i
+6
+
See how our wrapper gracefully handled both proper and improper uses? That's the power of context managers and magic methods. Note that the Python standard library includes a module contextlib that contains a context manager, contextlib.closing(), that does approximately the same thing (without any handling of the case where an object does not have a close() method).
@@ -512,29 +526,30 @@ Building Descriptor Objects
- Define behavior for when the descriptor's value is deleted.
instance is the instance of the owner object.
Now, an example of a useful application of descriptors: unit conversions.
-class Meter(object):
- '''Descriptor for a meter.'''
+class Meter(object):
+ '''Descriptor for a meter.'''
- def __init__(self, value=0.0):
- self.value = float(value)
- def __get__(self, instance, owner):
- return self.value
- def __set__(self, instance, value):
- self.value = float(value)
+ def __init__(self, value=0.0):
+ self.value = float(value)
+ def __get__(self, instance, owner):
+ return self.value
+ def __set__(self, instance, value):
+ self.value = float(value)
-class Foot(object):
- '''Descriptor for a foot.'''
+class Foot(object):
+ '''Descriptor for a foot.'''
- def __get__(self, instance, owner):
- return instance.meter * 3.2808
- def __set__(self, instance, value):
- instance.meter = float(value) / 3.2808
+ def __get__(self, instance, owner):
+ return instance.meter * 3.2808
+ def __set__(self, instance, value):
+ instance.meter = float(value) / 3.2808
-class Distance(object):
- '''Class to represent distance holding two descriptors for feet and
- meters.'''
- meter = Meter()
- foot = Foot()
+class Distance(object):
+ '''Class to represent distance holding two descriptors for feet and
+ meters.'''
+ meter = Meter()
+ foot = Foot()
+
Copying
@@ -551,23 +566,25 @@ Pickling Your Objects
Pickling is so important that it doesn't just have its own module (pickle), but its own protocol and the magic methods to go with it. But first, a brief word on how to pickle existing types(feel free to skip it if you already know).
Pickling: A Quick Soak in the Brine
Let's dive into pickling. Say you have a dictionary that you want to store and retrieve later. You couldwrite it's contents to a file, carefully making sure that you write correct syntax, then retrieve it using either exec() or processing the file input. But this is precarious at best: if you store important data in plain text, it could be corrupted or changed in any number of ways to make your program crash or worse run malicious code on your computer. Instead, we're going to pickle it:
-import pickle
+import pickle
-data = {'foo': [1, 2, 3],
- 'bar': ('Hello', 'world!'),
- 'baz': True}
-jar = open('data.pkl', 'wb')
-pickle.dump(data, jar) # write the pickled data to the file jar
-jar.close()
+data = {'foo': [1, 2, 3],
+ 'bar': ('Hello', 'world!'),
+ 'baz': True}
+jar = open('data.pkl', 'wb')
+pickle.dump(data, jar) # write the pickled data to the file jar
+jar.close()
+
Now, a few hours later, we want it back. All we have to do is unpickle it:
-import pickle
+import pickle
-pkl_file = open('data.pkl', 'rb') # connect to the pickled data
-data = pickle.load(pkl_file) # load it into a variable
-print data
-pkl_file.close()
+pkl_file = open('data.pkl', 'rb') # connect to the pickled data
+data = pickle.load(pkl_file) # load it into a variable
+print data
+pkl_file.close()
+
What happens? Exactly what you expect. It's just like we had data all along.
@@ -590,37 +607,38 @@ Pickling your own Objects
An Example
Our example is a Slate, which remembers what its values have been and when those values were written to it. However, this particular slate goes blank each time it is pickled: the current value will not be saved.
-import time
-
-class Slate:
- '''Class to store a string and a changelog, and forget its value when
- pickled.'''
-
- def __init__(self, value):
- self.value = value
- self.last_change = time.asctime()
- self.history = {}
-
- def change(self, new_value):
- # Change the value. Commit last value to history
- self.history[self.last_change] = self.value
- self.value = new_value
- self.last_change = time.asctime()
-
- def print_changes(self):
- print 'Changelog for Slate object:'
- for k, v in self.history.items():
- print '%s\t %s' % (k, v)
-
- def __getstate__(self):
- # Deliberately do not return self.value or self.last_change.
- # We want to have a "blank slate" when we unpickle.
- return self.history
-
- def __setstate__(self, state):
- # Make self.history = state and last_change and value undefined
- self.history = state
- self.value, self.last_change = None, None
+import time
+
+class Slate:
+ '''Class to store a string and a changelog, and forget its value when
+ pickled.'''
+
+ def __init__(self, value):
+ self.value = value
+ self.last_change = time.asctime()
+ self.history = {}
+
+ def change(self, new_value):
+ # Change the value. Commit last value to history
+ self.history[self.last_change] = self.value
+ self.value = new_value
+ self.last_change = time.asctime()
+
+ def print_changes(self):
+ print 'Changelog for Slate object:'
+ for k, v in self.history.items():
+ print '%s\t %s' % (k, v)
+
+ def __getstate__(self):
+ # Deliberately do not return self.value or self.last_change.
+ # We want to have a "blank slate" when we unpickle.
+ return self.history
+
+ def __setstate__(self, state):
+ # Make self.history = state and last_change and value undefined
+ self.history = state
+ self.value, self.last_change = None, None
+
Conclusion
diff --git a/magicmarkdown.py b/magicmarkdown.py
index 80513de..930eeb3 100755
--- a/magicmarkdown.py
+++ b/magicmarkdown.py
@@ -40,7 +40,7 @@
['def_list', 'codehilite'])
appendix_text = markdown.markdown(appendix, ['tables'])
-with open('magicmethods.html', 'w') as out:
+with open('index.html', 'w') as out:
out.write(HEADER)
out.write(table_text)
out.write(body_text)
diff --git a/table.markdown b/table.markdown
index 7a2b5aa..38e64fb 100755
--- a/table.markdown
+++ b/table.markdown
@@ -6,7 +6,7 @@ Copyright © 2012 Rafe Kettler
Version 1.17
-A PDF version of this guide can be obtained from [my site](http://www.rafekettler.com/magicmethods.pdf) or [Github](https://github.com/RafeKettler/magicmethods/raw/master/magicmethods.pdf). The magic methods guide has [a git repository at http://www.github.com/RafeKettler/magicmethods](http://www.github.com/RafeKettler/magicmethods). Any issues can be reported
+A PDF version of this guide can be obtained from [Github](https://github.com/RafeKettler/magicmethods/raw/master/magicmethods.pdf). The magic methods guide has [a git repository at http://www.github.com/RafeKettler/magicmethods](http://www.github.com/RafeKettler/magicmethods). Any issues can be reported
there, along with comments, (or even contributions!).
**Table of Contents**