Better Partials and Combinators in Python
Using Partial Function Applications With Recursive Wrappers and Functional Combinators
I know that it's a word salad of a title, but if you're reading this, then I'm sure you're A) old enough to read, B) have some idea of the topic, C) in the know that word salad titles are click-bait.
I cut my teeth on functional programming in Elixir. While it is a great language for many things and it is a fun language to use, ... functional it is not. When compared to something like Ocaml, there are many features of a more 'pure' functional language that Elixir lacks. These same features are also either missing in Python, or poorly implemented.
I like Python, I'll admit it. I can do many things quickly without much dev overhead. It's a good thing that I like it because my job is roughly 98% Python development. But I'm always on the search on how do do things a bit faster without making the code hard to read or maintain.
One trick that I have picked up from Ocaml is the use of partial function applications. In Ocaml you can do some weird stuff like so:
let adder x y = x + y;;
let add_five = adder 5;;
add_five 7;;
12
This seems like a strange and an unnecessary thing to do, but partial function applications are basically simple closures on-the-fly which is quite useful; a point that I will flesh out a bit later.
How Ocaml does this is quite unique. In the Ocaml world there are, technically, no multi argument functions, but a function that is a chain of functions. If you take a look at the signature of adder above it will look like the following:
x => y => x+y
To your eyes , this is a bit weird and a little hard to grok unless you understand that there are no variables only functions. The signature basically says “adder is a function that yields x. X is a function that takes an input and returns function y. Y is a function that takes an input and returns x+y.” This also explains why, typically, functional languages don't use parenthesis for functions, because they would get overused quite quickly and would be semantically difficult.
You may be saying “But the partial class exists in Pythons functools package.” That is true, but using it can give you some pretty wild results.
For example let's say you have a function like so:
def foo(a, b, c):
return f"{a=}, {b=}, {c=}"
Now lets also say that you need to make 2 partials from this one function because you're going to get the data you need at different times and you want to pass these partials around to other functions. And, let's also say that you get a, b and c out of order. You might do something like this:
bar = partial(foo, c=5)
# some other code
foobar = partial(bar, a=2)
# some more other code
foobar(1)
And I will tell you that the above won't work the way you think it will. In fact, it won't work at all. When you use a partial of a partial and then use keywords on positional arguments for out of order assignment (which is a totally bad thing to do in Python), all of your arguments must be keyword arguments. In the situation above the partial class will try to apply 2 to a, then 1 to a (not b like you would expect) and then throw an TypeError exception.
But the problem is that I still need that hot, sweet partial fix. A better solution is to just make our own partial wrapper that behaves a bit more sanely. For this let's implement a generic wrapper that implements recursion to do the job:
from functools import wraps, cache
from inspect import signature
@cache
def _get_required_arguments(func):
parameters = inspect.signature(func).parameters
positional_args = 0
keyword_args = 0
for k, v in parameters.items():
match v.kind:
case v.POSITIONAL_OR_KEYWORD:
if v.default == inspect._empty:
positional_args += 1
else:
keyword_args += 1
case v.VAR_POSITIONAL:
# if there are no positional args,
# then we are only expecting variadic arguments
# if there are positional args,
# then a variadic argument is optional
# and doesn't increment the required amount
if positional_args == 0:
positional_args += 1
case v.VAR_KEYWORD:
if keyword_args == 0:
keyword_args += 1
return (positional_args, keyword_args)
def partialize(func, *args, **kwargs):
@wraps(func)
def recurse(*nargs, **nkwargs):
return partialize(func, *args, *nargs, **kwargs, **nkwargs)
p_args, k_args = _get_required_arguments(func)
if len(args) < p_args:
return recurse
return func(*args, **kwargs)
Now we can do funny things like this which works in a similar fashion to what Ocaml offers:
@partialize
def foo(a, b, c):
return f"{a=}, {b=}, {c=}"
foo(1, 2, 3)
foo(1)(2, 3)
foo(1, 2)(3)
foo(1)(2)(3)
However, it does not fix the out of order problem from above. Nor does it fix the double application to a because of mixed keyword/positionals problem. What it does do is allow you to do is to wrap a function to make it partialable (not a word, I know) at function definition, something the partial class cannot offer.
To fix the out of order problem we would need to make a one-off combinator like so:
def foo(a, b, c):
return (a, b, c)
@partialize
def bar_combinator(c, a, b):
return foo(a, b, c)
bar_1 = bar_combinator(5)
bar_2 = bar_1(2)
result1 = bar_1(5, 10)
result2 = bar_2(1)
This of course does mean that you must know the order of application ahead of time. If you're in a situation where you have no idea what order you will be getting your arguments applied, then you should stick with only keyword arguments like so:
@partialize
def bar_combinator(a=None, b=None, c=None):
return foo(a, b, c)
Now, that's a real simple combinator. In fact, it doesn't really combine anything, but just rearranges arguments. So let's look at a real world use case that I had just t'other day.
I needed to build multiple Mac objects that take different argument and treat them in the same way. The two objects were ISO9797-algo3 and ISO9797-algo5 (aka: CMAC). Algo3 requires that the data to be macked is padded first in one of three different methods also defined in ISO9797. CMAC does not need padding, but it does need to know what crypto algorithm to use. The class definitions for each are like so:
class MacAlgo3:
def __init__(self, key: bytes):
self.key = key
def __call__(self, data: bytes) -> bytes:
return how_the_mac_is_made
class Cmac:
def __init__(self, key: bytes, algo: Literal["AES", "TDES"]):
self.key = key
self.algo = get_algo(algo)
def __call__(self, data: bytes) -> bytes:
return how_the_cmac_is_made
I knew that I needed my mac objects first, and my key and data would come later. To make this work:
@partialize
def comb1(cls, padder, key, data):
return cls(key)(padder(data))
@partialize
def comb2(cls, algo, key, data):
return cls(key, algo)(data)
def do_something(macker):
# derive key and get data
macker_loaded = macker(key)
mac = macker_loaded(data)
if option.a:
macker = comb1(Mac3, algo2)
else:
macker = comb2(Cmac, 'AES')
do_something(macker)
In this way I can shoehorn various objects into other objects that share the same signature. If I were to do this just using the partial class without combinators, I'm not sure it could be done. If I were to use combinators but use the partial class, it would probably be a mess. If I had to make a combinator without any partialization and only use hand-rolled closures, then I would probably shoot myself and my foot in the process.
Combinators can make your life EZ, but you have to make partial functions. Make that EZ too.