Huawei’s machine-learning team thought of themselves as artists more than programmers. That was the first slide in their deck, the one the recruiters showed at the big job-fairs at Stanford and Ben-Gurion and IIT. It was what the ML people said to each other, so repeating it back to them was just good tactics.

When you worked for Huawei, you got access to the firehose: every scrap of telemetry ever gleaned by a Huawei vehicle, plus all the licensed data-sets from the other big automotive and logistics companies, right down to the driver-data collected from people who wore court-ordered monitors: paroled felons, abusive parents under restraining orders, government employees. You got the post-mortem data from the world’s worst crashes and you got all the simulation data from the botcaves: the vast, virtual killing-field where the machine-learning algorithms duked it out to see which one could generate the fewest fatalities per kilometre.

But it took a week for Samuel to get the data from the mass hijackings in Melbourne and Damascus. It was all national-security-ied up the arse of course, but Huawei was a critical infrastructure partner of the Seven Eyes nations, and Samuel kept his clearances up with the four countries where he had direct-line reports working in security.

Without that data, he was left trying to re-create the attack through the Sherlock method: abductive reasoning, where you start with a known outcome and then come up with the simplest possible theory to cover the facts. When you have excluded the impossible, whatever remains, however improbable, must be the truth. If only that was true! The thing that never happened to Sherlock, and always happened to machine learning hackers, was that they excluded the impossible and then simply couldn’t think of the true cause – not until it was too late.

For the people in Damascus, it was too late. For the people in Melbourne, it was too late.

No pressure, Samuel.

Machine learning always started with data. The algorithm ingested the data, crunched it, and spat out a model, which you could test by feeding it some of the data you’d held back from the training set. Feed it 90 percent of the traffic info you had, ask it to model responses to different traffic circumstances, then test the model in the reserved set to see if it could correctly – that is, non-fatally – navigate the remaining traffic.

Data could be wrong in many ways. It was always incomplete, and whatever was left out could bias the model. Samuel always explained this to visiting school groups by inviting them to imagine training a model to predict height from weight by feeding it data from a Year Three class. It didn’t take the kids long to get how that might not produce good estimates for the height of adults, but the kicker was when he revealed that any third years who wasn’t happy about their weight could opt out of getting on the scales. ‘The problem isn’t the algorithm, it’s the data used to make the model.’ Even a school-kid could get that.

But it was more complicated than just biased data. There were also the special cases: what to do if an emergency vehicle’s siren was sensed (because not all emergency vehicles could transmit the lawful interception overrides that would send all traffic to the kerb lanes), what to do if a large ruminant (a deer, a cow, even a zebra, because Huawei sold cars all over the world) stepped into the car’s path, and so on. In theory, there was no reason not to use machine learning to train this too – just tell the algorithm to select for behaviours that resulted in the shortest journeys for simulated emergency vehicles. After all, there would always be circumstances when it was quicker for vehicles to drive a little further before pulling over, to prevent congestion, and the best way to discover those was to mine the data and run the simulations.

Regulators did not approve of this: non-deterministic, ‘artistic’ programming was a cute trick, but it was no substitute for the hard and fast binary logic of law: when this happens, you do that. No exceptions.

So the special cases multiplied, because they were like crisps, impossible to stop at just one. After all, governments already understood how special cases could be policy instruments.

Special cases were how pirate sites and child porn were excluded from search-results, how sensitive military installations were excluded from satellite photos in mapping apps, how software-defined radios stayed clear of emergency bands when they were hunting for interference-free channels. Every one of those special cases was an opportunity for mischief, since so many of them were secret by definition – no one wanted to publish the world’s most comprehensive directory of online child porn, even if it was supposed to serve as a blacklist – so the special case bucket quickly filled up with everything that some influential person, somewhere, wanted.

From gambling and assisted suicide sites being snuck into the child-porn list to anti-Kremlin videos being added to the copyright filters, to all the ‘accident- prevention’ stuff in the cars.

Since 1967, ethicists had been asking hypothetical problems about who should be killed by runaway trolleys: whether it was better to push a fat man onto the tracks (because his mass would stop the trolley) or let it crash into a crowd of bystanders, whether it made a difference if the sacrificial lamb was a good person or a bad one, or whether the alternative fatalities would be kids, or terminally ill people, or…

The advent of autonomous vehicles was a bonanza for people who liked this kind of thought-experiment: if your car sensed that it was about to get into an accident, should it spare you or others? Governments convened secret round-tables to ponder the question and had even come up with ranked lists: saving three children in the car topped saving four children on the street, but three adults would be sacrificed to save two kids. It was a harmless and even cute diversion at first, and it gave people something smart-sounding to say at lectures and cocktail parties.

But outside the actual software design teams, no one asked the important question: if you were going to design a car that specifically tried to kill its owners from time to time, how could you stop those owners from reconfiguring those cars to never kill them?

Samuel had been in those meetings, where half-bright people from the old-line automotive companies reassured quarter-bright bureaucrats from the transport ministries that there’d be no problem designing ‘tamper-proof’ cars that would ‘resist end-user modification.’ Meanwhile, much brighter sorts from the law – enforcement side of the house licked their chops and rubbed their hands together at all the non-trolley problems that could be solved if cars could be designed to do certain things when they got signals from duly authorised parties. Especially if the manufacturers and courts would collaborate to keep the inventory of those special cases as secret as the child-porn blocklists on the national firewalls.

He’d been in the design sessions after, where they debated how they’d hide the threads and files for those programs, how they’d tweak the car’s boot-cycle to detect tampering and alert the authorities, how the diagnostic tools provided to mechanics for routine service-checks could be used to double-check the integrity of all systems.

But then he’d started getting signed, obfuscated blobs from contractors who served governments around the world, developing ’emergency priority’ apps he was just supposed to drop in, without inspecting them. Of course he ran unit-tests before Huawei shipped updates, and when they inevitably broke the build, Samuel would go around and around with the contractors, who’d want access to all his source code without letting him see any of theirs.

It made sense for them to behave that way. If he failed to help them get their code into Huawei’s fleet, he’d have to answer to governments around the world. If they failed to help him, they’d have to answer to precisely no one.

Unit-tests were one thing, real-world performance was something else. Sensors couldn’t tell a car whether it was about to crash into some pedestrians, or a school bus, or an articulated lorry full of dynamite. All sensors could do was sense, and then feed data to machine-learning systems that tried to draw conclusions from those data. Even with all the special cases about what the car must and must not do under which circumstances, machine learning systems were how it knew what the circumstances were.

That’s how Melbourne happened.

It had taken him a long time to figure this out. At first, he assumed that finally, the worst had come to pass: the cryptographic keys that were used to sign police override equipment had leaked, and the wily criminals had used them to hijack 45 percent of the cars on the roads of one of the biggest cities in Australia. But the forensics didn’t show that at all.

Rather, the crooks had figured out how to spoof the models that invoked the special cases. Samuel figured this out by accident, his third day at his desk, running sim after sim on Huawei’s high-confidentiality cloud, which was protocol, even though it was the slowest and least-provisioned cloud he could have used. But it was only available to a handful of senior internal Huawei groups, not even contractors or partners.

He’d been running the raw telemetry from a random sample of the affected cars looking for anomalous behaviour. He’d nearly missed it, even so. In St Kilda, someone – face in shadow beneath a hat, thermal profile obscured – stepped in front of a subject car, which slowed, but did not brake, and emitted two quick horn- taps.

Regression analysis on accident data had shown that hard braking was more likely to result in rear-end collisions and frozen pedestrians who couldn’t get out of the way. The car tasked more compute time to the dorsal perimeter to see if it could shift into an adjacent lane without a collision, and if that wasn’t possible, to estimate the number of affected vehicles and passengers based on different maneuvers.

The pedestrian feinted towards the car, which triggered another model, the ‘suicide by car’ system, which invoked a detailed assessment of the pedestrian, looking for clues about sobriety, mental health and mood, all of which were difficult to ascertain thanks to the facial obfuscation. But there were other signals, a mental health crisis clinic 350 metres away, six establishments licensed for serving or selling alcohol with 100 metres, the number of redundancies in the past quarter, that gave it a high weighted score.

It initiated hard braking, and the pedestrian leapt back with surprising nimbleness. Then, across the road, another pedestrian repeated the dance, with another car, again in a shadowing hat and thermal dazzle makeup.

The car noticed this, and that triggered another model, which some analyst had labeled ‘shenanigans’. Someone was playing silly buggers with the cars, which was not without precedent, and well within the range of contingencies that could be managed. Alertness rippled through the nearby cars, and they began exchanging information on the pedestrians in the area: gait profiles, silhouettes, unique radio identifiers from Bluetooth devices. Police were notified, and the city – wide traffic patterns rippled, too, as emergency vehicles started slicing through the grid while cars pulled over.

All these exceptions to the norm were putting peak load on the car’s internal network and processors, which were not designed to continue operating when crises were underway – freeze-and-wait being the optimal strategy that the models had arrived at.

Eventually, the sheer quantity of compromised vehicles, combined with the minute variations in lane-spacing, small differences in car handling characteristics and, finally, a blown tyre, led to a pile up of ghastly proportions, a crash that they would study for decades to come, that would come to stand in for the very worst that people could do.

Samuel had always said that machine learning was an art, not a science, that the artists who designed the models needed to be able to work without official interference. He’d always said it would come to a bad end. Some of those meetings had ended in shouting matches, Samuel leaning over the table, shouting at bureaucrats, shouting at his bosses, even, in a way that would have horrified his parents in Lagos, where jobs like Samuel’s were like lottery jackpots, and shouting like his was an unthinkable act of economic suicide.

But he’d shouted and raged and told them that the fact that they wished that there was a way to put a back-door in a car that a bad guy couldn’t exploit didn’t mean that there was a way to do it.

He’d lost. If Samuel wanted to argue for a living, he’d have been a lawyer, not an algorithm whisperer.

Now he was vindicated. The bad ideas baked into whole nations’ worth of infrastructure were now ready to eat, and they would be a feast that would never end.

If this is what victory felt like, you could keep it. Elsewhere in the world, there were other Samuels, poring over their own teams’ reports: GM, VW-Newscorp, Toyotaford, Yugo. He’d met some of those people, even tried to recruit a few of them. They were as smart as Samuel or smarter, and they’d certainly shouted as loudly as he had when the time had come.

Enough to satisfy their honor, before capitulating to the unstoppable force of non-technical certitude about deeply technical subjects. The conviction that once the lawyers had come up with the answer, it was the engineers’ job to implement it, not trouble them with tedious technical wheedles about what was and wasn’t possible.