RabbitFarm

Part 1

Recently the question came up of how to call a Python function from Perl. Here is one way to do it.

The method here is to use Expect.pm to create a subprocess containing the Python repl. Python code is then loaded and called interactively. In my experience this is good for calling, say, a BERT model on some text from Perl. This approach is minimalistic as compared to other solutions such as standing up a Fast API instance to serve the model. Furthermore, this same pattern can be used for any arbitrary Python code you may need to call from Perl.

While this works well it does introduce additional complexity to an application. If at all possible it is preferable to re-write the Python functionality in Perl. An ideal use case would be where it would be too laborious to re-implement the Python code in Perl. Imagine, say, we want to use KeyBERT to extract keywords from a given body of text. In this case we may be doing substantial data and text processing in Perl and merely need to call out to Python for this single function. If at some point KeyBERT were to become available natively to Perl, perhaps through the Apache MXNet bindings, then that interface should be preferred. If nothing else, the performance improvement would be dramatic.

Solution

use strict;
use warnings;
##
# A simple example of calling a Python function
# from a Perl script using a Python repl started
# as a subprocess.
##
use Expect;
use boolean;
use constant TIMEOUT => 0.25; 
use constant PYTHON => q[/usr/bin/python];

sub create_python{
    my($io) = @_;
    my $python = do{
        local $/;
        ;
    };
    $$io = new Expect();
    $$io->log_stdout(false);
    $$io->raw_pty(true);
    $$io->spawn(PYTHON);
    $$io->send("$python\n\n");
    $$io->expect(TIMEOUT, q[-re] , q|m/[0-9]*/|);
    $$io->clear_accum();
}

sub call_python_sample{
    my($io, $arg) = @_;
    print $$io->send("sample(" . $arg . ")\n");
    $$io->expect(TIMEOUT, q[-re], qr[\d+]);
    my $r = $$io->exp_match();
    $$io->clear_accum();
    return $r;
}

MAIN:{
    my($io);
    create_python(\$io);
    print call_python_sample(\$io, 1) . "\n";
    print call_python_sample(\$io, 9) . "\n";
}

__DATA__
import os
os.system("stty -echo")
def sample(a):
    print(str(a + 1))

The results

$ perl call_python_.pl
2
10

Notes

The code here is a minimum working example. Well, fairly minimal in that I could have avoided breaking things up into multiple subroutines. In terms of cleanliness and explainability these divisions make sense, with only the added need to pass a reference to an Expect object back and forth as a parameter.

For a self-contained example the Python code we are going to run is contained in the DATA section. For more complex use cases it would make sense to have the Python code in separate files which could be read in and loaded. They could also be specified directly as arguments to the Python interpreter.

• sub create_python instantiates a new Expect object, sets some parameters for the object, and spawns the Python repl. We also clear the Expect buffers so that upon the next invocation we need not worry about the Python header messages.

• sub call_python_sample calls the function of interest. Here it is just sample() which takes a single argument, adds 1 to the argument, and prints out the result.

Effectively what we are doing is interprocess communication using text passed between the two processes. Perl knows nothing of the state of the Python code, and vice versa. If you call a Python function which does not print a value to STDOUT then you will need to add your own print() call. This is not actually so bad a situation since Expect works by pattern matching on the expected (pun intended!) output. To ensure you are collecting the right values some massaging of what the Python code is doing is to be anticipated (pun avoided!). For example, suppose we want to call the KeyBERT function to extract key words from some given text. We might consider writing a wrapper function which takes the output from KeyBERT.extract_keywords (a list of tuples, each tuple a pair: key phrase and a distance) and concatenates and prints each of the pairs to STDOUT on a single line. In this way our Perl regex can most easily pick up the phrase/distance pairs.

Expect is a very mature tool, with a generous set of options and abilities. This sort of use is really just the tip of the iceberg. In terms of Perl being a "Glue Language" consider Expect to be a key ingredient that causes the glue to stick. Peruse the documentation for further inspiration.

References

Expect.pm

KeyBERT