Connecting to a Cluster

Once you’ve booted a ufora cluster (remotely or locally), you can connect to it and start executing code:

import pyfora
ufora = pyfora.connect("http://localhost:8000/")

The variable ufora now holds an Executor, which is a live connection to the cluster. There are two ways to use an Executor: using a futures model ( powerful but more cumbersome), or using with blocks, which we’ll detail here.

First, let’s define a function we want to work with:

def isPrime(p):
    x = 2
    while x*x <= p:
        if p%x == 0:
            return 0
        x = x + 1
    return 1

Calculating on the Cluster

Now, we can use the executor to do something interesting with the function.

with ufora.remotely.downloadAll():
    result = sum(isPrime(x) for x in xrange(10 * 1000 * 1000))

print result

The code contained in the with block gets shipped to the Ufora server along with any dependent objects and code (like isPrime) you’re referencing in the block. This code gets translated into Ufora bitcode, and executed by the Ufora VM. The resulting objects are returned over the ufora connection, which downloads them and copies them back into the local environment because we used ufora.remotely.downloadAll().

Working With Proxies

Now, imagine that we want to get a list of primes. We can then write

with ufora.remotely.remoteAll():
    primes = [x for x in xrange(10 * 1000 * 1000) if isPrime(x)]

Now, because we used ufora.remotely.remoteAll(), the variable primes is a proxy to a list of primes (actually, it’s a RemotePythonObject). Remote python objects can be used in two ways: they can be downloaded into the local python scope, or they can be passed to additional computations. To download a proxy, we might write

primes = primes.toLocal().result()

This will cause all the data in the remote primes variable to be downloaded from the Ufora cluster to the client machine. If the list is very large, or the connection to the cluster is slow, this can be a slow operation. Alternatively, we can chose not to download our result, but compute with the result by using the proxy object inside of another with ufora.remotely block. For example:

with ufora.remotely.downloadAll():
    lastFewPrimes = primes[-100:]

Ufora recognizes that primes refers to an object living remotely on the server, and allows us to perform dependent computations on it, which we return as regular python objects.

For convenience, we may also write:

with ufora.remotely.downloadSmall(bytecount=100*1024):
    ...

in which case objects requiring more than bytecount bytes will be left on the server, and smaller objects will be downloaded. This pattern works well as long as your objects are obviously on one side or the other of the threshold. If they’re not, we recommend leaving objects as remotes and downloading them as you need them.