“It’s only a passing thing, this shadow. Even darkness must pass. A new day will come. And when the sun shines it will shine out the clearer.” – Lord of the Rings
Coding is really problem solving. It is extremely frustrating, but once you figure out the problem, the reward for the accomplishment is well worth the frustration. This post represents one of those frustrating but rewarding problems. I went on a journey to implement my PicoGame engine in C/C++ with the SDK available through Python (specifically Micropython on the Raspberry Pico). In this post, we will cover:
- The Background: What I am trying to achieve
- The Challenges:
- External Micropython Module versus Native Machine Code (MPY)
- Understanding the Micropython API
- Building the module
The Background
Over the past few weeks, I have been stuck. My son and I are creating our own game system using the Raspberry Pico. I created the initial version of the game engine in Python. However, mainly due to availability, I upgraded the screen from the Waveshare 1.44, which was 128×128 pixels, to the Adafruit 2.2, which is 240×320 pixels. While the new screen is 2x better and did not feel like would require a significant change in the code, this created challenges that I was not prepared for. My existing Python library was not able to handle the new screen in 2 ways: (1) it was way too slow (for a simple screen clear, I could see the pixels being drawn) and (2) it required significantly more memory (240x320x16bpp requires 150k versus 32k for the smaller screen). Therefore, I decided to transition the SDK to C++.
The biggest challenge in transitioning to C++ was getting the new library to compile correctly (this led me to creating this post on best first languages). However, once I had the PicoGame library working in C++, I had a new challenge. By making the library C++, I had made the game system so my children were not able to actually use it. They are happiest using Scratch (although they can do some JavaScript if necessary). I tried to find something like Blockly for C++, but could not find anything suitable. Blockly only includes interpreted languages like JavaScript/TypeScript or Python. There are workarounds, but those workarounds have limitations. In addition, I wanted to provide a stepping stone for my children to move from something like Scratch to an actual language (such as Python).
With all of these considerations in mind, I started down the path of creating an external module for Micropython. This would allow me to use the C++ SDK (changing to C, which I will explain later) but allow the SDK to be called from Micropython. I get the speed plus the improved control of memory with the ease of Python. While the outcome met my requirements, the path to get to the outcome was very difficult.
However, I ultimately succeeded. One of the greatest feelings for a coder is the moment that compilation first succeeds followed by a successful test run. It is that feeling that can make weeks of frustration worth it.
The Challenges
So what were the challenges? There were three key challenges that I faced:
- External Micropython Module versus Native Machine Code (MPY)
- Understanding the Micropython API
- Building the module
External Micropython Module versus Native Machine Code (MPY)
There are two ways to use C code into Micropython:
- External Module: An external module is integrated into Micropython. To build, you must build the entire Micropython source code with your module. The benefit is it is tightly integrated with Micropython. The downside is that it is not user friendly (i.e., the user has to be able to build micropython).
- Native Machine Code (MPY): This was my initial preference. The idea is that you can compile the code into an MPY file. The MPY file can be loaded on a Pico that is already running Micropython and simply used. The MPY file is then imported the same way that you would import a Python source file.
I ultimately ended up with the external module. However, this was not my intent. I thought the MPY file was the clear winner. It would be much easier to use than building Micropython every time I make a change to the SDK. However, the MPY file did not work for me for two reasons.
First, the MPY file is specific to a version of Micropython. If you attempt to use a different version, you get an error.
Second, I learned that I could not link against the Raspberry Pico SDK using MPY. This made this option not possible. I needed significant functionality from the SDK (i.e., GPIO, SPI).
Understanding the Micropython API
The second challenge was understanding the Micropython API. I covered this in much more detail in this post. TLDR, the Micropython API uses significant boilerplate code. There is not an easy way around it, you just have to learn the code/approach.
Relatedly, I also changed some of my SDK from C++ to C. This is not about speed. I will leave the debate on C versus C++ speed to others. My preference is always C++. I find it much easier to think in terms of classes. For example, in my original PicoGame SDK, I used a class to represent the screen with methods for clearing and drawing on the screen. However, the Micropython API is C. Therefore, in order to use a C++ class, you need an additional file that sits between Micropython and the PicoGame SDK that creates the interface from C++ to C. This felt like a waste. Therefore, I converted parts of the SDK from C++ to C.
The change was easy, it just required a structure to hold the class instance variables and a pointer in each function to refer to the structure.
Building with CMake
Micropython for the Raspberry Pico requires uses of CMake. Prior to this project, I had not worked much with CMake. In college, I always used Makefiles. Although I have seen CMake before (for example compiling open source projects like Blender), CMake was always in the background, and I did not need to touch the CMake files. However, in the case of creating an external module in Micropython, I had to create the CMake file. I found the book Professional CMake: A Practical Guide to be really easy to understand and very comprehensive (I only wish that this book was natively available on Amazon Kindle).
Ultimately, I ended up using a single CMake file. This is definitely not my preference. I would like to have a CMake file (and library) for individual parts of the SDK such as the screen code. I found that I needed to consolidate to one file to actually build (although this may still be my inexperience with CMake). I discovered that I can only use the command target_link_libraries to link my code to the final usermod. Below is my final CMake file (picogame.cmake).
add_library(picogame INTERFACE)
target_sources(picogame INTERFACE
${CMAKE_CURRENT_LIST_DIR}/picogame.c
${CMAKE_CURRENT_LIST_DIR}/picogame_sdk/screen/screen.c
${CMAKE_CURRENT_LIST_DIR}/picogame_sdk/screen/ili9341.c
${CMAKE_CURRENT_LIST_DIR}/picogame_sdk/input/input.cpp)
target_include_directories(picogame INTERFACE
${CMAKE_CURRENT_LIST_DIR}/
${CMAKE_CURRENT_LIST_DIR}/picogame_sdk/
${CMAKE_CURRENT_LIST_DIR}/picogame_sdk/screen/
${CMAKE_CURRENT_LIST_DIR}/picogame_sdk/input)
target_link_libraries(usermod INTERFACE picogame)If I tried to create sub-libraries and to link to those libraries, the build failed. This appears to be due to a line in the Micropython CMake file that attempts to recursively process any library included in target_link_libraries. Below is the relevant section of micropython/py/usermod.cmake:
# Gather library sources
get_target_property(lib_sources ${LIB} INTERFACE_SOURCES)
if (lib_sources)
list(APPEND ${SOURCES_VARNAME} ${lib_sources})
endif()
# Gather library includes
get_target_property(lib_include_directories ${LIB} INTERFACE_INCLUDE_DIRECTORIES)
if (lib_include_directories)
list(APPEND ${INCLUDE_DIRECTORIES_VARNAME} ${lib_include_directories})
endif()
# Recurse linked libraries
get_target_property(trans_depend ${LIB} INTERFACE_LINK_LIBRARIES)
if (trans_depend)
foreach(SUB_LIB ${trans_depend})
usermod_gather_sources(
${SOURCES_VARNAME}
${INCLUDE_DIRECTORIES_VARNAME}
${INCLUDED_VARNAME}
${SUB_LIB})
endforeach()
endif()By consolidating into a single CMake and using target_link_libaries once, I was able to finally build.
The Code
If you just want the code, here is a link to the module code in GitHub. To build this code, you will need the Micropython source code.
Conclusion
In this post, I talked about my journey to implement a PicoGame engine in C/C++ but to make the engine available in Micropython. Through this journey, I faced many challenges including understanding Micropython external modules and learning CMake.
We ultimately succeeded in building the module. While the challenges were frustrating, the success was incredibly rewarding.
