A man in Oakland, California, disrupted web development around the world last week by deleting 11 lines of code.
The story of how 28-year-old Azer Koçulu briefly broke the internet shows how writing software for the web has become dependent on a patchwork of code that itself relies on the benevolence of fellow programmers. When that system breaks down, as it did last week, the consequences can be vast and unpredictable.
A man in Oakland, California, disrupted web development around the world last week by deleting 11 lines of code.
The story of how 28-year-old Azer Koçulu briefly broke the internet shows how writing software for the web has become dependent on a patchwork of code that itself relies on the benevolence of fellow programmers. When that system breaks down, as it did last week, the consequences can be vast and unpredictable.
“I think I have the right of deleting all my stuff,” Koçulu wrote on March 20 in an email that was later made public.
And then he did it.
Koçulu had been publishing code he wrote to npm, a popular service that’s widely used to find and install open-source software written in JavaScript. It has become an essential tool in web development, invoked billions of times a month, thanks to npm’s ease of use and its enormous library of free code packages contributed by the open-source community.
The open-source philosophy is what drove Koçulu to contribute to npm in the first place, and why he ended up abandoning the service. Like many in the broad community of people who write code that anyone can use, he is influenced by the “hacker ethic” of early programmers at MIT and a set of more concrete values that were later put forth by the programmer Richard Stallman.
“The fundamental act of friendship among programmers is the sharing of programs,” Stallman wrote in his 1985 manifesto. He railed against “the commercialization of system software,” and laid out ways to make code more communal and widely useful. Many of Stallman’s ideas continue to influence programmers like Koçulu.
“I’m a self-taught high school graduate who learn everything thanks to open source community,” Koçulu, who was born in Turkey, wrote in an email to Quartz. “I owe everything I have to the people who never gave up with open source philosophy.”
One of the open-source JavaScript packages Koçulu had written was kik, which helped programmers set up templates for their projects. It wasn’t his most popular work. But it shared a name with Kik, the messaging app based in Ontario, Canada. On March 11, Koçulu received an email from Bob Stratton, a patent and trademark agent who does contract work for Kik.
Stratton said Kik was preparing to release its own package and wanted to call it kik. “Can we get you to rename your kik package?” Stratton wrote.
“Sorry, I’m building an open source project with that name,” Koçulu wrote back.
The conversation quickly escalated, with Stratton threatening legal action: “We don’t mean to be a d*** about it, but it’s a registered trademark in most countries around the world and if you actually release an open source project called kik, our trademark lawyers are going to be banging on your door and taking down your accounts and stuff like that — and we’d have no choice but to do all that because you have to enforce trademarks or you lose them.”
“Hahah, you’re actually being a d***,” Koçulu replied. “So, f*** you. Don’t email me back.”
Stratton offered to pay for the name, and Koçulu suggested $30,000 “for the hassle of giving up with my pet project for bunch of corporate d***s.” It was clear the two men weren’t going to reach an agreement.
The company called npm is based, like Koçulu, in Oakland. Though a for-profit enterprise, npm runs its eponymous registry of open-source software for free and has a mission of fostering open-source JavaScript development. The company generates revenue from private services for code that isn’t open-sourced, a business model similar to GitHub.
Stratton brought Kik’s request for the name to npm, again citing the company’s trademark and potential confusion. Isaac Schlueter, the chief executive of npm, agreed to turn the name over to the company.
“In this case, we believe that most users who would come across a kik package, would reasonably expect it to be related to kik.com,” Schlueter wrote to Stratton and Koçulu on March 18. “In this context, transferring ownership of these two package names achieves that goal.”
“I know you for years,” Koçulu replied, “and would never imagine you siding with corporate patent lawyers threatening open source contributors.”
Many programmers, particularly in the open-source community, are critical of intellectual property law in the United States, and the patent- and trademark-holders who seek to enforce it. For years, software companies have been bombarded with lawsuits over patents that cover ubiquitous technologies, like displaying images on web pages or sending news stories over email. Software developers tend to see these lawsuits as a hindrance to innovation and borderline extortion.
To Koçulu, npm’s decision to transfer ownership of the kik package to Kik ran counter to the values of the community it serves. In his reply, Koçulu said he wanted all of the packages he had registered on npm taken down. “I don’t wanna be a part of NPM anymore,” he wrote. “If you don’t do it, let me know how do it quickly.”
Two days after Koçulu’s last email to npm, on March 22, JavaScript programmers around the world started receiving a strange error message when they tried to run their code. The issue was severe enough to keep some developers from updating apps and services that were already running on the web. The error spit out many lines, but one stood out: npm ERR! 404 'left-pad' is not in the npm registry.
It meant that the code they were trying to run required a package called left-pad, but the npm registry didn’t have it.
Most programmers had never heard of left-pad, but now, somehow, their code couldn’t run without it. To understand how this could happen, it’s important to understand that almost all software is built on top of other software, which also depends on other software. Loading your own app might require a certain set of packages from npm, but those packages may load their own packages, and so on. That’s one reason npm has become so popular, helping to manage those dependencies by maintaining all of the packages in one, reliable place.
Reliable, that is, until one of the packages goes missing.
By early evening, developers began congregating at the GitHub repository where left-pad was maintained. Most were confused because packages don’t usually disappear. This one was particularly perplexing because it was just 11 lines of straightforward code. Here is left-padin its entirety:
Screen Shot 2016-03-25 at 7.28.34 PM
That code can be used to add characters to the beginning of a string of text, perhaps a zero to the beginning of a zip code. It’s a single-purpose function, simple enough for most programmers to write themselves. Lots of npm packages, however, relied on left-pad to do it for them, which is how this tiny bit of code became so important.
Some of the largest, most widely used npm packages in the world were suddenly broken. One of the affected packages, React, is used by major websites like Facebook, which created it, and a wide variety of smaller sites like Quartz’s own Atlas. In the past month alone, more than a million people have downloaded React from npm. React didn’t require these 11 lines of code directly, of course. It depended on one set of packages, and each of those depended on another set, et cetera, and one of those branches eventually led to left-pad. And now, left-pad was gone.
It’s absence was felt globally; the commenter’s on left-pad's GitHub page were writing from Australia, Germany, the United States, and the Czech Republic. In Ontario, where the issue had originated in its roundabout way, programmers at Kik were ironically running into left-pad problems, as well. Mike Roberts, who runs the company’s messaging app, said in an interview that the error prevented his colleagues from running software they had been working on. “What the heck,” Roberts recalled thinking, “one of our packages is missing?”
An hour after the issue was first noticed, Koçulu surfaced with a post on Medium titled, “I’ve Just Liberated My Modules.” He briefly explained the dispute with Kik and npm, and said he’d deleted his packages from npm in protest–all 273 of them. One of those—hardly the most popular or even the most important, even to Koçulu—was left-pad.
“This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because, Power To The People,” Koçulu wrote.
Facing a crisis, with so much important software falling apart, npm decided to restore the 11 lines of code. “Un-un-publishing is an unprecedented action that we’re taking given the severity and widespread nature of breakage, and isn’t done lightly,” wrote Laurie Voss, the chief technology officer of npm. He added, “This action puts the wider interests of the community of npm users at odds with the wishes of one author; we picked the needs of the many.”
With that, the issue was fixed, about two hours after it first emerged.
That left-pad was able to wreak such havoc, even for a brief period, speaks to the way that modern software is developed. Web services of outsized importance, like Facebook, can come to be dependent on obscure lines of code written by other programmers. Soon after the ordeal was resolved, an incredulous post rose to the top of Reddit’s section for programmers: “An 11 line npm package called left-pad with only 10 stars on github was unpublished…it broke some of the most important packages on all of npm.”
Some programmers blamed Kik, for threatening legal action over an open-source project, or npm, saying the breakdown was a sign that the service’s infrastructure is too fragile. Many also called into question npm’s choice to accede to Kik’s demand. “Was there really no way this could have gotten resolved,” one commenter wrote, “without npm swiping someone’s module out from under them? Or even any public discussion? Does this mean npm will cave to any legal threat?”
When asked in a phone interview with Quartz what he would do if Twitter of Google asked for the rights to npm packages currently registered under those names, Schlueter said it would depend on the packages themselves. “Generally,” he said, “that’s just sort of a matter of looking at how Twitter would want to use the module called twitter or how the current developer is using it, and how well-established it is, and how many people are depending upon it, and countless other factors.”
Still others lamented the fact that an 11-line npm package existed at all, suggesting that programmers should be able to write those 11 lines of code themselves. Jokes on that topic quickly proliferated across the internet. Someone created leftpad.io, poking fun at the massive dependence on such a simple piece of code. (“In order to prevent such a terrible tragedy from occurring ever again during our lifetimes, ‘left-pad.io’ has been created to provide all the functionality of ‘left-pad’.”)
Mike Roberts, from Kik, said in an interview that he regretted not reaching out to Koçulu himself in the first place. “From my perspective,” he said, “open-source, the community, is about helping each other out.”
Courtesy: Quartz
