We’re on the cusp of restarting Yukon development and I was reminded of some lessons I’ve learned that I haven’t applied to this project. @lorenz.meier recently remarked on the importance of full and realtime logging for this tool and the performance problems we’ve had trying to read, deserialize, and present logs using Python for very busy systems with lots of traffic. Lorenz has seen this problem with the v0 tools and I’ve seen this problem on my systems with the v0 tools (CAN-FD makes it much worse). One school of thought is that we should expect beefy development boxes are used for such things but in reality it’s crufty, old maintenance systems or specialized embedded devices that will run the Yukon backend. This suggests that using Python as the backend technology for Yukon may be a fundamental mistake? Perhaps we should be writing this in C++ to maximize efficiency and portability? Perhaps we should use Rust because all the cool kids are doing it and it’s way more fun…and we’ll get the performance we need if not the portability to embedded devices (yet)? Perhaps even (dare I say it) go?
The problem with this is we have pyuavcan so we can start on a python backend immediately. To choose C++ we need libuavcan and to choose rust we need uavcan.rs (anyone working on uavcan.go-or-whatever-convention-that-language-uses)? Even so, this may be important enough to sequence things differently.
Additionally, allowing the Yukon backend to run on an embedded target opens up the possibility of creating purpose-built UAVCAN debugging appliances that are portable or even integrated with the vehicle system itself.
Not all UAVCAN implementations are created equal. High-integrity software development guidelines prescribe that there are to be no unused execution paths in an executable image, from which follows that an implementation designed for high-integrity embedded applications will be devoid of features that are needed for in-depth network diagnostics.
We could certainly construct one such implementation designed for advanced network inspection and diagnostics in any language. I expect that the design requirements for such an implementation would be very different from those of a typical embeddable library; the difference is likely to warrant the development of a completely independent implementation from scratch even if we had an available embeddable solution in place. The two key differentiators are:
An inspection/diagnostic implementation does not have to follow high-integrity software development practices. Lack of necessity to follow stringent software development standards reduces the cost.
An inspection/diagnostic implementation should be extensible, support all transports including early-stage experimental ones, and should permit low-cost modification and experimentation (e.g., experimental transports and new protocol features).
One practical consequence is that the above might render C++ a suboptimal choice. I long for the day when I could wipe all C++ development tools from my computers. Rather, it might make sense to pick a garbage-collected language, perhaps something from the newer generation of JVM-based languages, like Kotlin or Scala.
I agree that Python is probably not the best choice here – its dynamic typing keeps getting in the way, in 2020 its GIL is still there, and the performance is poor – but I don’t think it’s terrible, either. Given that our resources are extremely limited, it makes sense to build upon what we already have, with a possibility of future migration to a different backend built from scratch for the task. As long as we are stuck with Python, we could explore its alternative implementations such as PyPy (which is really superior performance-wise compared to CPython).