A Journey to Creating a Micropython User Modules

Norman

While working on updating the PicoGame engine, I wanted to implement the core of the game engine in C/C++ instead of Python to try to increase the speed. I succeeded in implementing the engine in C++ (more on this in a future post); however, we still wanted people to be able to use the engine in Python. Therefore, we needed to create a user module for Micropython.

Initially, I planned to create a native mpy module. An mpy module is a compiled python script. You write the code in C and then use Micropython tools to compile and create the mpy module. I thought this would be easier for a user.

Ultimately, this did not work, because you are unable to link to external libraries with mpy. This is a dealbreaker for two reasons. First, you are unable to link to the Pico SDK (which is necessary for the game engine). Second, I wanted to try to keep the game library separate so it was still useful for anyone who wants to create a game for the Raspberry Pico using C++. Therefore, I needed to implement as part of a Micropython build.

In today’s post, we will cover: 

  • Quick overview of the SimpleMath sample code
  • The basics of creating a user module
  • How to compile into mpy
  • How to build into Micropython

In our next post, we will talk about how we transformed our PicoGame engine into a Micropython user module.

Quick Overview of the Sample Code

Before we dive into the Micropython specifics, we need code that we can turn into a Python module. Based on the idea from the Micropython examples, I created a SimpleMath library. The library has three functions:

  • Factorial
  • isEven
  • isOdd

Here is the code for simplemath.c:

#include <stdbool.h>

#include "simplemath.h"

// Returns factorial of x
int factorial(int x){
    if(x == 0){
        return 1;
    }

    return factorial(x-1)*x;
}

// Returns whether a number is odd
bool isOdd(int x){
    if(x % 2 == 1){
        return true;
    }

    return false;
}

bool isEven(int x){
    return !isOdd(x);
}

Here is simplemath.h:

#ifndef SIMPLEMATH_H
#define SIMPLEMATH_H

int factorial(int x);
bool isOdd(int x);
bool isEven(int x);
void fibonacci(int x);

#endif

Basics of Creating a User Module

Now that we have some code to work with, let’s turn it into a Micropython module. There are three parts to the process:

  • Create the class
  • Make a globals table to hold a reference to the class (and any other non-class functions)
  • Register the module

One of the challenges of the Micropython API is that it takes multiple statements to create something in Python. To me, this makes it difficult to read and follow the code. Here is a high level diagram of how the pieces fit together.

Creating the Class

We will start by creating the class. There are three parts to creating the class:

  • Create a structure to hold the information for the class in the background
  • Make the class itself
  • Set up a locals table to hold the instance methods

Create the Structure

The first step is to create an internal structure to hold information about the class. We will call this PicoMath_obj_t.

typedef struct _PicoMath_obj_t {
	mp_obj_base_t base;
	int result;
} PicoMath_obj_t;

Our struct includes two variables:

  • mp_obj_base_t base: required to create a Micropython object
  • int result: we really didn’t need this, but it shows how to use internal variables. In addition, we used it to avoid unused variable warnings.

Note that the variables in this structure will not be accessible through Python.

Create New Function

Next, we will create the function that will be used whenever a new class is created.

static mp_obj_t PicoMath_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
	PicoMath_obj_t *self = mp_obj_malloc(PicoMath_obj_t,type);

	return MP_OBJ_FROM_PTR(self);
}

In our case, this function is really simple. We simply allocate space for our internal structure (PicoMath_obj_t). We then return a pointer to the struct.

Create the Instance Functions

Now we will create our instance functions. These are created in the same way as non-class functions with one exception. We will add a parameter that is a reference to our class object (self_in), and in the first line of the method, we will covert the parameter to a pointer to our internal struct. For example, the function below is a method that adds two integers. We store the result in our internal struct.

static mp_obj_t pico_add(mp_obj_t self_in, mp_obj_t x_obj, mp_obj_t y_obj){
	PicoMath_obj_t *self = MP_OBJ_TO_PTR(self_in);
	int x = mp_obj_get_int(x_obj);
	int y = mp_obj_get_int(y_obj);

	self->result = x+y;
	
	return mp_obj_new_int(self->result);
}
static MP_DEFINE_CONST_FUN_OBJ_3(pico_add_obj, pico_add);

We would not need a class method for this; however, I wanted something simple to show how it works.

Once we have created all of our instance methods, then we will create a locals dictionary table that contains all of our instance methods.

static const mp_rom_map_elem_t PicoMath_locals_dict_table[] = {
	{ MP_ROM_QSTR(MP_QSTR_pico_add), MP_OBJ_FROM_PTR(&pico_add_obj) },
	{ MP_ROM_QSTR(MP_QSTR_pico_factorial), MP_OBJ_FROM_PTR(&pico_factorial_obj) },
	{ MP_ROM_QSTR(MP_QSTR_pico_isEven), MP_OBJ_FROM_PTR(&pico_isEven_obj) },
	{ MP_ROM_QSTR(MP_QSTR_pico_isOdd), MP_OBJ_FROM_PTR(&pico_isOdd_obj) },
	{MP_ROM_QSTR(MP_QSTR_pico_fibonacci), MP_OBJ_FROM_PTR(&pico_fibonacci_obj) },
};
static MP_DEFINE_CONST_DICT(PicoMath_locals_dict, PicoMath_locals_dict_table);

Create the Class Itself

Finally, we will create the externally facing class.

MP_DEFINE_CONST_OBJ_TYPE(
	type_PicoMath,
	MP_QSTR_PicoMath,
	MP_TYPE_FLAG_NONE,
	make_new, PicoMath_make_new,
	locals_dict, &PicoMath_locals_dict);

This code creates a new type (type_PicoMath) and registers the name (MP_QSTR_PicoMath). We are not using any flags (MP_TYPE_FLAG_NONE). To be honest, I have not figured out what flags are available yet. We register our make_new function (PicoMath_make_new) and our locals dictionary (PicoMath_locals_dict).

Create Globals Table

Next, we will create a globals table. This table will include a name for our module (MP_QSTR_picoexample). It will also include our class (PicoMath) and a function that is part of the module but not part of the class.

static const mp_rom_map_elem_t PicoMath_globals_table[] = {
	{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_picoexample) },
	{ MP_ROM_QSTR(MP_QSTR_PicoMath), MP_ROM_PTR(&type_PicoMath) },
	{ MP_ROM_QSTR(MP_QSTR_test_add), MP_ROM_PTR(&test_add_obj)},
};
static MP_DEFINE_CONST_DICT(PicoMath_globals, PicoMath_globals_table);

Register the Module

The final step is to register the picoexample module.

// Define module object.
const mp_obj_module_t picoexample = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t *)&PicoMath_globals,
};

// Register the module to make it available in Python.
MP_REGISTER_MODULE(MP_QSTR_picoexample, picoexample);

We create a module object (mp_obj_module_t) with a base of mp_type_module. We set the globals attribute to our globals table. Finally, we call MP_REGISTER_MODULE that registers the module.

The Full Code

Here is the full code for PicoMath.c.

#include "py/obj.h"
#include "py/runtime.h"

#include "simplemath.h"

#ifdef __cplusplus
extern "C" {
#endif

typedef struct _PicoMath_obj_t {
	mp_obj_base_t base;
	int result;
} PicoMath_obj_t;

const mp_obj_type_t picoclass_type;

static mp_obj_t test_add(mp_obj_t x_obj, mp_obj_t y_obj) {
	int x = mp_obj_get_int(x_obj);
	int y= mp_obj_get_int(y_obj);

	return mp_obj_new_int(x+y);
}
static MP_DEFINE_CONST_FUN_OBJ_2(test_add_obj, test_add);

static mp_obj_t pico_add(mp_obj_t self_in, mp_obj_t x_obj, mp_obj_t y_obj){
	PicoMath_obj_t *self = MP_OBJ_TO_PTR(self_in);
	int x = mp_obj_get_int(x_obj);
	int y = mp_obj_get_int(y_obj);

	self->result = x+y;
	
	return mp_obj_new_int(self->result);
}
static MP_DEFINE_CONST_FUN_OBJ_3(pico_add_obj, pico_add);

static mp_obj_t pico_factorial(mp_obj_t self_in, mp_obj_t x_obj){
    PicoMath_obj_t *self = MP_OBJ_TO_PTR(self_in);
	int x = mp_obj_get_int(x_obj);

	self->result = factorial(x);

    return mp_obj_new_int(self->result);
}
static MP_DEFINE_CONST_FUN_OBJ_2(pico_factorial_obj, pico_factorial);

static mp_obj_t pico_isOdd(mp_obj_t self_in, mp_obj_t x_obj){
	PicoMath_obj_t *self = MP_OBJ_TO_PTR(self_in);
    int x = mp_obj_get_int(x_obj);

	self->result = x;

    return mp_obj_new_bool(isOdd(x));
}
static MP_DEFINE_CONST_FUN_OBJ_2(pico_isOdd_obj, pico_isOdd);

static mp_obj_t pico_isEven(mp_obj_t self_in, mp_obj_t x_obj){
	PicoMath_obj_t *self = MP_OBJ_TO_PTR(self_in);
    int x = mp_obj_get_int(x_obj);

	self->result = x;

    return mp_obj_new_bool(isEven(x));
}
static MP_DEFINE_CONST_FUN_OBJ_2(pico_isEven_obj, pico_isEven);

static mp_obj_t pico_fibonacci(mp_obj_t self_in, mp_obj_t x_obj){
	PicoMath_obj_t *self = MP_OBJ_TO_PTR(self_in);

    int x = mp_obj_get_int(x_obj);
	mp_obj_t list[x];

    //fibonacci(x);
	int first = 0;
    int second = 1;
    int current = 0;

	for(int index=0; index<x; index++){
        if(index == 0){
			list[0] = mp_obj_new_int(0);
        } else if(index == 1){
			list[1] = mp_obj_new_int(1);
        } else {
            current = first + second;
			list[index] = mp_obj_new_int(current);
            first = second;
            second = current;
        }
    }

	self->result = x;

    return mp_obj_new_list(x,list);
}
static MP_DEFINE_CONST_FUN_OBJ_2(pico_fibonacci_obj, pico_fibonacci);

static mp_obj_t PicoMath_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
	PicoMath_obj_t *self = mp_obj_malloc(PicoMath_obj_t,type);

	return MP_OBJ_FROM_PTR(self);
}

static const mp_rom_map_elem_t PicoMath_locals_dict_table[] = {
	{ MP_ROM_QSTR(MP_QSTR_pico_add), MP_OBJ_FROM_PTR(&pico_add_obj) },
	{ MP_ROM_QSTR(MP_QSTR_pico_factorial), MP_OBJ_FROM_PTR(&pico_factorial_obj) },
	{ MP_ROM_QSTR(MP_QSTR_pico_isEven), MP_OBJ_FROM_PTR(&pico_isEven_obj) },
	{ MP_ROM_QSTR(MP_QSTR_pico_isOdd), MP_OBJ_FROM_PTR(&pico_isOdd_obj) },
	{MP_ROM_QSTR(MP_QSTR_pico_fibonacci), MP_OBJ_FROM_PTR(&pico_fibonacci_obj) },
};
static MP_DEFINE_CONST_DICT(PicoMath_locals_dict, PicoMath_locals_dict_table);


MP_DEFINE_CONST_OBJ_TYPE(
	type_PicoMath,
	MP_QSTR_PicoMath,
	MP_TYPE_FLAG_NONE,
	make_new, PicoMath_make_new,
	locals_dict, &PicoMath_locals_dict);

static const mp_rom_map_elem_t PicoMath_globals_table[] = {
	{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_picoexample) },
	{ MP_ROM_QSTR(MP_QSTR_PicoMath), MP_ROM_PTR(&type_PicoMath) },
	{ MP_ROM_QSTR(MP_QSTR_test_add), MP_ROM_PTR(&test_add_obj)},
};
static MP_DEFINE_CONST_DICT(PicoMath_globals, PicoMath_globals_table);


// Define module object.
const mp_obj_module_t picoexample = {
    .base = { &mp_type_module },
    .globals = (mp_obj_dict_t *)&PicoMath_globals,
};

// Register the module to make it available in Python.
MP_REGISTER_MODULE(MP_QSTR_picoexample, picoexample);

#ifdef __cplusplus
}
#endif

Troubleshooting

A quick note on troubleshooting. Documentation on building a C extension is a little sparse. It took me several tries to work through the example. In the process, I came across a troubleshooting page that was a lifeline. The class definition here was different than the Micropython example code. It turned out that Micropython changed the way it stored data about the class. In the previous method, there was a lot of wasted space for unused attributes.

Compiling

Compiling was relatively simple. I cloned the Micropython git.

git clone https://github.com/micropython/micropython.git

Next, in the Micropython directory, I created a new folder called cmod (short for c module). I created a folder called SimpleMath and copied in simplemath.c, simplemath.h, and PicoMath.c. I also created a make file named micropython.cmake with the following:

# Create an INTERFACE library for our C module.
add_library(pico_math INTERFACE)

# Add our source files to the lib
target_sources(pico_math INTERFACE
    ${CMAKE_CURRENT_LIST_DIR}/PicoMath.c ${CMAKE_CURRENT_LIST_DIR}/simplemath.c
)

# Add the current directory as an include directory.
target_include_directories(pico_math INTERFACE
    ${CMAKE_CURRENT_LIST_DIR}
)

# Link our INTERFACE library to the usermod target.
target_link_libraries(usermod INTERFACE pico_math)

Conclusion

This extension may not do very much. However, it helped us to understand how to create an extension in Micropython. Next up, we will be modifying our PicoGame C code to create a Micropython module.

Related Posts