This is one of the many reasons there should be a universal data standard using a format like JSON. Heavily structured, easy to parse, easy to debug. What you lose in footprint (i.e., more disk space), you gain in system stability.
Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.
Parsing the data formats had zero contribution to the problem. They had a problem running an algorithm on the input data, and error reporting when that algorithm failed. Nothing about JSON would improve the situation.
Yes, but look at the data. The algorithm was buggy because the input data is a nightmare. If the data didn't look like that, it's very unlikely the bug(s) would have ever existed.
ADEXP sounds like the universal data standard you want then. The UK just has an existing NATS that cannot understand it without transformation by this problematic algorithm. So the significant part of your suggestion might be to elide the NATS specific processing and upgrade NATS to use ADEXP directly.
Using a JSON format changes nothing. Just adds a few more characters to the text representation.
No change at all? I find that hard to believe. There's also a data design problem here, but the structure of JSON would aid in, not subtract from, that process.
The question at hand is: "heavily structured data vs. a blob of text as input into a complex algorithm, which one is preferred?"
Unless you're lying, you'd choose the former given the option.
The issue is using both ADEXP and ICAO4444 waypoints, and doing so in a sloppy way. For the waypoint lists, there is no issue with structurelessness -- the fact that they're lists is pretty obvious, even in the existing formats. Adding some ["",] would not have helped the specific problem, as the relevant structure was already perfectly clear to the implementers. I am not lying when I say the bug would have been equally likely in a JSON format in this specific case.
Now I'm wigging out to the idea of how the act of overcoming the inertia of the existing system just to migrate to JSON would spawn thousands of bugs on its own — many life-threatening, surely.
To me and XML-ified this would look more nightmarish than the status quo... it's just brief, space separated and \n terminated ASCII. No need to overcomplicate things this simple.
> The algorithm was buggy because the input data is a nightmare.
No, the algorithm was "buggy" because it didn't account for the entry to and exit points from the UK to have the same designation because they're supposed to be geographically distant (they were 4000Nm apart!) and the UK ain't that big.
There are already standards like XML and RDF Turtle that allow you to clearly communicate vocabulary, such that a property 'iso3779:vin' (shorthand for a made-up URI 'https://ns.iso.org/standard/52200#vin') is interpreted in the same way anywhere in the structures and across API endpoints across companies (unlike JSON, where you need to fight both the existence of multiple labels like 'vin', 'vin_no', 'vinNumber', as well as the fact that the meaning of a property is strongly connected to its place in the JSON tree). The problem is that the added burden is not respected at the small scale and once large scale is reached, the switching costs are too big. And that XML is not cool, naturally.
On top of that, RDF Turtle is the only widely used standard graph data format (as opposed to tree-based formats like JSON and XML). This allows you to reduce the hoop jumping when consuming responses from multiple APIs as graph union is a trivial operation, while n-way tree merging is not.
Finally, RDF Turtle promotes use of URIs as primary identifiers (the ones exposed to the API consumers) instead of primary keys, bespoke tokens, or UUIDs. Followig this rule makes all identifiers globally unique and dereferenceable (ie, the ID contains the necessary information on how to fetch the resource identified by a given ID).
P.S.: The problem at hand was caused by the algorithm that was processing the parsed data, not with the parsing per se. The only improvement a better data format like RDF Turtle would bring is that two different waypoints with the same label would have two different URI identifiers.
Furthermore, there are already XML namespaces for flight plans. These are not, however, used by ATC - only by pilots to load new routes into their aircrafts' navigation computers.
I'm not sure whether there is an existing RDF ontology for flight plans; it would probably be of low to medium complexity considering how powerful RDF is and the kind of global-scale users it already has.
Airport software predates basically every standard on the planet. I would not be surprised to learn that they have their own bizarro world implementation of ASCII, unix epoch time, etc.
(There is a modern replacement for AFTN called AMHS, which replaces analog phone lines with X.400 messages over IP... but the system still needs to be backwards compatible for ATC units still using analog links.)
Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.
But after you did it, you'd still have exactly the same problem. The cause was not related to deserialization. That part worked perfectly. The problem is the business logic that applied to the model after the message was parsed.
I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating.
The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.
Does the system work if it randomly fails and collapses the entire system for days?
People generally prefer to be lazy and to not use their brains, show up, and receive a paycheck for the minimum amount of effort. Not to be rude, but that's where this attitude originates. Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.
> Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.
Yes, but in real life doesn't work.
Processes have corner cases. As you said, people are lazy and will do everything to find the corner case to fit in.
Just an example from the banking sector.
There are processes (and even laws) that force banks to use only certified, supported and regularly patched software: there are still a lot of Windows 2000 servers in their datacenters and will be there for many years.
Broadly speaking I think this is done for new systems. What you need to identify here is how and when you transition legacy systems to this new better standard of practice.
I'd argue in favor of at least an annual review process. Have a dedicated "feature freeze, emergencies only" period where you evaluate your existing data structures and queue up any necessary work. The only real hang up here is one of bad management.
In terms of how, it's really just a question of Schema A to Schema B mapping. Have a small team responsible for collection/organization of all the possible schemas and then another small team responsible for writing the mapping functions to transition existing data.
It would require will/force. Ideally, too, jobs of those responsible would be dependent on completion of the task so you couldn't just kick the can. You either do it and do it correctly or you're shopping your resume around.
Great. It should be fixed by replacing the FORTRAN systems with a modern solution. It's not that it can't be done, it's that the engineers don't bother to start the process (which is a side-effect of bad incentive structure at the employment level).
No migration of this magnitude is blocked because of engineers not "bothering" to start the process. Imagine how many approvals you'd need, plus getting budget from who-knows how many government departments. Someone is paying for your time as an engineer and they decide what you work on. I'm glad we live in a world where engineers can't just decide to rewrite a life or death system because it's written in an old(er) programming language. (Not that there is any evidence that this specific system is written in anything older than C++ or maybe Ada.)
That's... not how that works. I take it you're probably more of a frontend person than a backend person by this comment. In the backend world, you usually can't fully and completely replace old systems, you can only replace parts of systems while maintaining full backwards compatibility. The most critical systems in the world -- healthcare, transportation, military, and banking -- all run on mainframes still, for the most part. This is isn't a coincidence. When these systems get migrated, any issues, including issues of backwards compatibility cause people to /DIE/. This isn't an issue of a button being two pixels to the left after you bump frontend platform revs, these systems are relied on for the lives and livelihood of millions of people, every single day.
I am totally with you wishing these systems were more modern, having worked with them extensively, but I'm also realistic about the prospect. If every major airline regulator in the world worked on upgrading their ATC systems to something modern by 2023 standards, and everything went perfectly, we could expect to no longer need backwards compatibility with the old system sometime in 2050, and that's /very/ optimistic. These systems are basically why IBM is still in business, frankly.
Many of them have been upgraded. In the US, we've replaced HOST (the old ATC backend system) with ERAM (the modern replacement) as of 2015.
However, you have to remember this is a global problem. You need to maintain 100% backwards compatibility with every country on the planet. So even if you upgrade your country's systems to something modern, you still have to support old analog communication links and industry standard data formats.
In some sense, yes. Notice that most of the responses to what I've said are immediately negative or dismissive of the idea. If that's the starting point (bad mindset), of course nothing gets fixed and you land where we are today.
My initial approach would be to weed out anyone with that point of view before any work took place (the "not HR friendly" part being to be purposefully exclusionary). The only way a problem of this scope/scale can be solved is by a team of people with extremely thick skin who are comfortable grabbing a beer and telling jokes after they spent the day telling each other to go f*ck themselves.
Anyone who has worked with me knows that I have no issue coming in like a wrecking ball in order to make things happen, when necessary. I've also been involved in some of these migration projects. I think your take on the complexity of these projects (and I do mean inherent complexity, not incidental complexity) and the responses you've received is exceptionally naive.
The amount of wise-cracks and beers your team can handle after a work day is not the determinate factor in success. /Most/ of these organizations /want/ to migrate these systems to something better. There is political will and budget to do so, these are still inglorious multi-decade slogs which cannot fail, ever, because failure means people die. No amount of attitude will change that.
> The amount of wise-cracks and beers your team can handle after a work day is not the determinate factor in success.
Of course it isn't. But it's a starting point for building a team that can deal with what you describe (a decade-plus long timeline, zero room for failure, etc). If the people responsible are more or less insufferable, progress will be extremely difficult, irrespective of how talented they are.
Airplane logistics feels like one of the most complicated systems running today. A single airline has to track millions of entities: planes, parts, engineers, luggage, cargo, passengers, pilots, gate agents, maintenance schedules, etc. Most of which was created all before best-practices were a thing. Not only is the software complex, but there are probably millions of devices in the world expecting exactly format X and will never be upgraded.
I have no doubt that eventually the software will be Ship of Thesus-ed into something approaching sanity, but there are likely to be glaciers of tech debt which cannot be abstracted away in anything less than decades of work.
It would still be valuable to replace components piece-by-piece, starting with rigorously defining internal data structures and publically providing schemas for existing data structures so that companies can incorporate them.
I would like to point out that the article (and the incident) does not relate to airline systems; it is to do with Eurocontrol and NATS and their respective commercial suppliers of software.
The problem was not in the format, but with the way the semantics of the data is understood by the system. It could be fixed-width, XML, json, whatever, and the problem would still be the same.
Imagine a world where everybody uses JSON and if they offer an API, you can just consume the data without a bunch of hoop jumping. Failures like this would vanish overnight.