On standards and regulation

Development & Maintenance
1

Moving this discussion here as it turned out to be a serious sticking point: https://github.com/UAVCAN/specification/pull/32#discussion_r234205382

Before we begin, let me make sure we are all on the same page.

What is currently written in the proposed spec is that DSDL type definitions can have fixed port IDs defined for them. If a fixed port ID is defined, it is guaranteed that the function for which the data type is created will always be available at this port. This is supremely useful for some standard things; like, for example, by looking at uavcan.node.Heartbeat one will know immediately that to receive Heartbeat from all nodes, one should subscribe to the subject 62805; or, say, to get a node info, one should call the service 430. Any variance is excluded, things are nice and uniform for all. Having a fixed port ID does not preclude the possibility of using the type at other port IDs concurrently if so is desired.

Now, there are regulated and unregulated port identifiers. Regulated are those that are reviewed and vetted by the community. Being maintained in a centralized manner, they are always the same for everyone, so the possibility of conflicts or collisions is excluded. Currently, there are two locations where regulated identifiers are used:

Besides fixed identifiers, there are also non-fixed ones (we used to call them “dynamic”, but Scott objected; the reasons were considered valid (at least that was my own understanding) so we are not using that word anymore). They can be assigned per-subject or per-service at the system definition time; for example, by a vehicle designer, or by an integrator, or similar.

Fixed identifiers are dangerous, because if one is used, it can’t be changed anymore until the node is upgraded. There is no possibility for anybody except the node designer to change it, because otherwise it would break the function availability guarantees described above. If anybody would like to challenge this point, please ignore the following text – until this is settled, further discussion would not make sense.

Since fixed identifiers cannot be changed, they can lead to unresolvable conflicts in the real world. As in, a vehicle designer would procure two nodes from two different vendors and then find out that they use the same fixed ID for different purposes. Too bad, they can’t work together; complaints afoot.

To prevent such conflicts, we make the following power move: we demand that if one wanted to use a fixed port ID with their data type, they make the data type definition public. They get in touch with the maintainers and ask for a fixed ID (by sending a pull request). Maintainers grant the ID (which we call regulated ID) and the definition gets accepted into the vendor-specific type repo linked here earlier.

Summary:

  • If you want a fixed ID, you make the data type definition publicly available. Maintainers give you your brand new ID, which is also called a “regulated ID”.
  • If you want your types private, you don’t get any fixed ID. Because if you could have one, you would eventually run into conflicts with other vendors.
  • If you tried to trick the system by using a fixed ID with your private data types, DSDL parsers would catch that and report you to authorities raise an error.

Now, it may seem restrictive, indeed. Consider an entirely hypothetical scenario where a huge online retailer wanted to build a robotic submarine for their own maritime delivery needs. Their submarines would never be available off-the-shelf to any third parties, the ecosystem is entirely closed, so who cares if they re-use port IDs already taken by some Chinese noname? Indeed, nobody; right until the point where their ecosystem ceases to be so closed: either because they decided to provide their submarines or any parts of them to third parties, or if they decided to procure some third-party parts for themselves. We are interested in promoting a non-fragmented shared ecosystem where COTS equipment can easily collaborate; allowing conflicts here would very much go against that goal.

How about we demand that vendors switch from fixed IDs to non-fixed IDs once they make their ecosystems not-so-closed? Well, seeing as it can’t be enforced automatically, good luck with that. We meatbags are notoriously bad at following complex rules, especially if they require planning ahead. Nobody wants to mess with the code of a perfectly working submarine to un-fix some ID just to appease some standards.

To the point raised by Kjetil:

If they were starting to sell the unit (instead of just the integrated application) they would have to move the definition to the vendor specific domain and request a new static ID (or use dynamically configured IDs). They would have to move their definition to the vendor specific namespace either way (to make it available and check out on Port compatibility checks) which would be impossible with an ID from the application specific DTS.

No, it will not work as decribed. Many would opt to not go back and change a working product; they would release it with a footnote in the doc saying that unfortunately, due to historic reasons, this thing is stuck using this fixed ID; if you have problems with that, give our tech support a call. There are few things that can be more detrimental to the ecosystem than that. People don’t care about standards enough.

Paging @scottdixon @kjetilkjeka

2

I agree with the sentiment of avoiding the fragmented ecosystem but I suspect the way you propose to do this is inefficient and damages more then it helps.

A group I’m helping integrate Uavcan has full control of every part on the vehicle. They have their good reasons for having it this way, and they would not have any way to sell the vehicle or any part of it in the future. They can statically allocate these IDs completely safely, as even if they were to buy uavcan compatible hardware in the future they would be able to use it on IDs they haven’t used yet. This suggestion would not stop them in doing anything, they would instead just define the static message ID as an associated constant and use it for ID, they would probably be a bit annoyed over the discrepancy between the standard definitions and their custom definitions. Hence what you’re suggesting is highly inefficient in what it attempts to achieve, but causes annoyance even when used safely.

We should deeply care about volunteer groups, DIYs, research project and similar that wants to use Uavcan. Using statically allocated IDs can be the correct thing to do in some scenarios, as long as it’s done with care. I agree that the commercial robotic submarine guys should use configurable IDs, but this example isn’t general enough to apply for every use case. And they would be able to work around your proposal just as easy as anyone else.

Let me contribute some ideas to efficiently achieve what we want without having destructive side effects:

  • Being explicit in the specification that the lower DST needs to be choosable freely for the system integrator, this is a hard specification for the part being uavcan compatible.
  • Highly discourage the use of statically allocated IDs for custom types.
  • The DSDL compiler will give an error with an explanation why it’s bad to do so by default, can be opted out of using a flag but will still issue one single warning when the flag is present.
  • Having high-quality code provided for making it as easy to use dynamically allocated IDs.
  • Having high-quality mechanisms in DSDL making it trivial to use dynamically allocated IDs.
  • Having it as an official policy to only use dynamically allocated IDs for custom types in the examples.

Being an open source project, there are always going to be people that will pick and choose. Some are going to do things such as using the transport layer with their own datatypes, and some are only going to allow single frame transfers, etc.

I believe making it smooth to use the “official system” will always have a higher chance of working and attracting the “partial users” as constructive contributors. Knowing how they use it (and why) is an important step to converge to a system that is great for everyone.

3

@kjetilkjeka Your group might be a rare exception if their ecosystem ends up entirely closed indeed. This exception might not be important enough to accommodate for their needs in the specification. One could consider adding some implementation-specific options to DSDL parsers/compilers that permit them to accept unregulated fixed port IDs, but this is not a great idea, clearly.

I am curious what would @scottdixon say about the submarine scenario. I mean, Scott, if you were a submarine designer, what would you say?

4

This argument still counts for DIYs and researchers that use one highly custom (maybe ad-hoc) unit together with COTS units. The custom units can be assigned IDs statically and the COTS units dynamically in a way that avoids all conflicts. The only way it’s going to make sense for them to use dynamic IDs is if it’s as easy as using static (and the DSDL compiler warn them against static IDs).

Also note that my critique against the proposal is mainly that it is highly inefficient as it’s trivial to work around. The group I’m working with will not be hindered by the proposal, just annoyed. I suspect other will end up doing the same. I therefore think more constructive alternatives will be more efficient. If a compiler switch is used you will get a reminder every time you compile that you did a potentially bad thing.

5

I’d say I prefer the term “Naval Architect” but seriously, this is a big thread. Let me catch up on this tonight.

6

Ultimately if submarine designers et-al have enough unregulated address space for services and subjects to work with then I don’t think we have much to argue about here. I agree with @kjetilkjeka that forcing DSDL to always error when using regulated IDs from custom types is draconian and not true to the open-source spirit. I think a default failure that can be demoted to a warning with a switch is a good approach (if you mention the switch in the error message, that is). I also agree with @pavel.kirienko that, for example; a certified submarine type would never undergo a complete reallocation of port identifiers just to accommodate COTS hardware but that such systems might incorporate equally certified COTS hardware where it could be purchased as an LRU. This latter point drives the requirement that we error by default and perhaps nag a little when using the override but, coming full circle, the only reason makers and submarine designers wouldn’t just stick to the unregulated port identifier spaces is if these spaces were too small.

1 Like
7

Okay, I can propose a sort of a middle ground that might be not bad enough to be outright rejected. Before I describe it, I want you to know that I recognize the problems with my proposed restriction, but the liberal alternatives discussed above seem unacceptable because they mostly ignore the actual problem.

We could probably spin the problem in a way that would encourage vendors to either use dynamic non-fixed ID or publish their type definitions to obtain proper, community-approved, regulated fixed ID (while we’re at it, why don’t we also grant shiny watermarked certificates printed on stamped paper). We could do this by artificially encouraging competition for fixed IDs. The idea is to prohibit usage of fixed ID in the unregulated (lower) range as initially proposed, but to permit them in the regulated range. So, if you have a message type and you desire its subject ID to be fixed, you pick a value in the range [57344, 59391] (we agreed on that earlier) and the DSDL compiler doesn’t bat an eye. You go on your merry way using your proprietary message definition, but there’s a catch: nobody except you and your partners in crime enlightened colleagues knows about the existing fixed ID. One day some other vendor will submit a request for a fixed ID to the UAVCAN maintainers, and they will receive a regulated fixed ID, which just might be the same as the one you’re using for your needs, except that their ID is approved, regulated, legal and certified, while you’re now an outlaw. You’d say: “Eh, so what? My ecosystem is closed, who cares?”, while withdrawing into an even deeper underground, and you’d be right, but the conflicting ID is now your problem, and the community doesn’t care. If you want any part of your ecosystem to be released in the wild, you will have to do something about the conflict; and you have two options:

  1. Switch to unregulated non-fixed ID.
  2. Come out of the underground and get your data type published, obtaining a new ID.

If you chose to push forward anyway disregarding the conflict, your customers will point their fingers at you, and that vendor whose ID you’re illegally sitting on might just file some lawsuits or whatever. The problem is almost self-resolving, maintainers don’t need to get involved.

If you were lucky and nobody sat on your ID unknowingly, you can always come out of the underground at any moment, send your data types for the approval asking for regulated ID, and explain that unfortunately, you can’t really live with any other ID, lest your precious certified system will require some major revamp. The maintainers will be very understanding and pronounce your ID legalized.

If we chose to go this way, we will need to enlarge the regulated ID range a bit. We could also use a global ID counter to reduce the likelihood of collisions: some kind of an online big red button which anybody can press and obtain an ID back; the returned ID would be from the regulated range but guaranteed to be officially unoccupied at the time of clicking the button. The returned ID would start from the lower boundary going upwards while skipping those values that are already officially taken; once it reached the top, it would just overflow and start over. The probability of collisions would thus be minimized, conditional on the fact that people don’t abuse the button. To prevent that, we could ask for an email, requesting an email verification for every requested ID; although the fact of verification really provides no guarantees whatsoever because eventually the counter will overflow again and your IDs might be granted to someone else, too bad; thus it’s not a substitute to the official approval process. Although the hypothetical team of naval architects might consider this unacceptable because it amounts to an information leak about their system. Scott, if you were a naval architect, how would you react?

8

Then your equipment might conflict with other COTS equipment. This means that correct use of statically allocated IDs for custom messages may be broken any time in the future for reasons out of your control. This is very very very very very bad. I think the suggestion of encouraging the use of the vendor DTS for custom data types is unacceptable and outright dangerous.

9

Exactly! This is no different from the alternative solution discussed above, where closed ecosystems were permitted to use fixed IDs in the lower range. The problem just becomes more visible, but it’s the same problem. The advantage of my latter proposition is that it escalates the problem of conflict where there are two parties, one of which is compliant with the standard and it’s in their interest to force the other party to fix their products.

10

And if you are involved with a closed ecosystem, you can always avoid the risk of conflict by doing either:

  • Using non-fixed ID.
  • Obtaining regulated ID.
11

It’s a huge difference. If you allow a system integrator to use static IDs in the lower range for custom hardware there can never be a conflict with other hardware following the protocol. You can always buy any protocol following COTS hardware and integrate it easily into your system without worry.

I’ve added this discussion as a point to the dev meeting tonight. Let’s discuss it a bit there.

12

Right, but I think this is one of those cases where you can end up with a dysfunctional standard unless you look beyond its letter and see how it can actually be used. We just talked about it, see my first post in this thread. We can’t force vendors to adhere to the requirement of using non-fixed ID in the lower range; hence, non-compliant devices will show up on the market, and it will be nobody’s problem. Hence I don’t think that my latter proposition increases the risk of conflicts; and everyone can easily avoid them entirely by not relying on any unregulated fixed ID in the first place.

13

I still completely agree with the sentiment of avoiding fragmentation, and I’m still absolutely certain that your proposed restrictions will not work and be borderline destructive.

People are going to rip the specification apart and use bits and pieces. If there is no way the dsdl parser can accept default IDs in the low ID range I’m going to be first in line to fork it and modify it to allow that (or just add them as associated consts). And I am one of the individuals that care about the specification enough to contribute towards it, I just disagree with disallowing the use of statically allocating these IDs in a controlled environment. I’m pretty sure the naval architect guys are quite smart and will figure out a way to bypass this little restriction as well. I’m also pretty sure that the naval architect guys are less probable to contribute back upstream if their code either diverge from the tooling or they feel their use case is disallowed for no good reason, it will perhaps also make them more probable to diverge further. This will in the worst case make the protocol miss out on much-needed development contribution, without having gained anything.

Even if people rip the spec to bits and pieces good actors that follow the specification cannot be punished. In my point of view, it’s absolutely unacceptable to create the specification in a way that may create conflicts if used correctly. This will show the good protocol following actors that we don’t take stability seriously and even they will hesitate to use the protocol, even if they wouldn’t be directly impacted.

They will probably read the datasheet beforehand, and find out that this part is not specification adhering. They will then send a letter to the manufacturer, urge them to create firmware that is spec adhering. Or buy an equivalent part from a competing manufacturer that actually follows the specification.

We can host a user/manufacturer edited database (list) of commercially available and open source “uavcan compatible” hardware where we go as far as discouraging the use of these units if they fail to follow the DTS/ID rules.

14

I think the point related to forking is more important than I initially considered, so I will have to agree to that.

I am going to update the current PR accordingly and we’ll get to discuss it afterward. Meanwhile, would be great to have input from Scott.