Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Show HN: Lazy evaluation in Python (github.com/llllllllll)
88 points by joejev on March 2, 2015 | hide | past | favorite | 21 comments


Somewhat related -- A few months ago I wrote a decorator that can allow for function currying [0], like so:

    @curry
    def add3(a,b,c): return a+b+c 

    # normal function application
    >>> add3(1,2,3)
    6

    # add3(1,2) returns a unary function which is then applied to 3
    >>> add3(1,2)(3)
    6 

    # rebinding partially applied functions to another variable, then applying
    >>> add2 = add3(100)
    >>> add2(5,6)
    111 

    >>> add1 = add2(3.14)
    >>> add1(5)
    108.14
    >>> map(add1,range(5))
    [103.14, 104.14, 105.14, 106.14, 107.14] 
[0] https://gist.github.com/grantslatton/9221084


Cool, but you don't need that black magic inspecting the arguments, I think you can refactor to use `functools.partial`:

    >>> from functools import partial
    >>> f = partial(lambda a, b: a + b, 1)
    >>> f(1)
    2


I suspect you will still need to inspect, otherwise you don't know when to actually evaluate. Consider applying all of the arguments at once:

    >>> partial(add3, 1,2,3)
    <functools.partial object at 0x2b335e0453c0>
    >>> partial(add3, 1,2,3)()
    6
Note that you have to stick an extra () at the end.

You could maybe try to evaluate it and catch the error telling you that there weren't enough args. Both are kinda hacky, I think.



That fails for something like this:

    def test(x, y=None):
        return x

    assert curry(test)(1) == 1 # fails


I know, I left kwargs as a later exercise. You can still refactor with `partial` though since it accepts kwargs as the third argument.


This is really cool. The AST transformation stuff here is neat, but relatively well-trodden ground.

The more impressive new science here is the lazy_function decorator, which is implemented as a bytecode transformer on the code object that lives inside the decorated function. The author built his own library for the bytecode stuff, which lives here: https://github.com/llllllllll/codetransformer.


You said in a comment that you're looking for a usecase for this technique, so I'll provide one for something similar, perhaps we'll get ideas.

I've been toying with something similar lately, as a caching framework for scientific computations. I will have something like:

    x = load_big_file(filename)  # takes 2 minutes
    y = sqrt(1 / x ** 2)  # takes 4 seconds
    ...
Then, as my work proceed, I will change and tweak and re-run in the same process...clearly a lot of my time would have been saved by caching (though a different part each time depending on what I tweak).

The way to go currently is use joblib, which provides a decorator to put on a function to do basic caching. However you have to take care to manually clear cache if a function you're dependent on changes, or sometimes it will clear cache itself because you changed something irrelevant to the computation.

So the lazy alternative idea I had is a lazily evaluated tree similar to this, where the purpose is looking up a cache using the AST as key. What I have now looks more like this:

    @pure
    def load_big_file(filename): ...

    >>> x = load_big_file(lazy(fileref(filename)))  # fileref is like a string but hashes by timestamp of file..
    >>> y = sqrt(1 / x ** 2)
    >>> print y
    <lazy 23wfas
      input:
        v1: 32rwaa "/home/dagss/data/myfile.dat" @ 2015-03-03 08:43:23
      program:
        e0: 43wafa v1**2
        e1: 4rfafq 1 / e0
        e2: sqrt(e1)
    >
The point is every node in the syntax tree gives a hash (leafs having their value hashed, inner nodes having a hash based on the operation and the inputs). Then implementing a cache is simply

    NULL = object()
    y = cache.get(y, NULL)
    if y is NULL:
        y = cache[y] = compute(y)
I'm leaning a bit towards explicit being better than implicit though (having to call compute(y) rather than it happening when you need it to be evaluated transparently..)


Currently, there is no way to clear an object's cache in lazy because it doesn't hold onto the function and arguments longer than needed to make sure that the gc has time to step in and clear unneeded objects. The built in memoization could help with this though.

Also, in lazy, you must call `strict` on things to get a result, it is not implicit. What is somewhat implicit is that calling `bool(somethunk)` (or any other converter implemented with a magic) will return a strict value because of the python standard.


Are there any interesting use-cases for this and/or problems it solves? Not that there necessarily have to be, of course, I'm just curious.


The codetransformer that I worked on actually has some real use cases, namely exposing an object to a function at runtime without making it available to the calling code by name.

As far as lazy itself, it was purely developed for fun in my spare time; however, that does not mean it is not useful. I will say that there was no intended use case for this project and it was not designed to solve a particular problem.


Do you actually intend on writing a description later, or is that just a joke?


It will be written when you open an issue about it, forcing it to be evaluated.


Yeah, that is the joke; however, I think the description will stay as it is.


Interesting, very reminiscent of Lisp macros! Glad to see that python code transforming isn't too difficut.


You can do a lot with Python's AST:

http://docs.hylang.org/en/latest/


your github username is frustrating.


It is ten lower case 'L' characters.


At least it is better than http://www.rrrrthats5rs.com !


But why?


Because I think my username is fun, just like I think that programming is fun.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: