Hasty Treat - Hasty Horror Stories

Monday. Monday. Monday.

Open wide dev fans. Get ready to stuff your face with JavaScript, CSS, node modules, barbecue tips, get workflows, breakdancing, soft skill, web development, the hastiest, the craziest, the tastiest TS, web development treats coming in hot. Here is Wes, Barracuda, Boss, and Scott, El Toro Loco, Tolinski.

CSM.

Oh, welcome to Syntax.

In this Monday spooky treat, We're gonna be diving into some spooky stories from Twitter that we heard on WebDev Twitter. We asked you, we need your WebDev horror stories. For every year, we do an annual guess that's what every year means, an annual horror stories where we read off scary dev horror stories, Things that have happened to real people, these are not made up events or anything. These are things that have happened to real life devs, and we read their stories to you So that you can sit in your seat and cringe a little bit and feel all tense all over for other people's mistakes.

And this actually is a little bit of a Hasty Amuz Boosh here. We're we're doing we'll be releasing before the the full course that we're gonna be doing on Wednesday for our Spooky treats. We have a lot a lot of cool stories planned for that one as well. So buckle up. Let's get into some spooky stories. But before we do, I should introduce myself. My name is Scott Tolinski. I'm a developer from Denver, Colorado, and with me as always is the spookiest West boo. Oh, hello.

Hello.

Hello.

Yes. This episode is sponsored by and Century.

Sentry and Linode are 2 amazing sponsors. Wes, I'll talk about Sentry. You could talk about Linode. Sentry is the perfect place to catalog All of your creepy crawly errors and bugs in your application, and we have them all available to you in a nice interface for you to be able to Catalog, log them, and, eventually, squash those bugs. Fix those bugs and push those bugs up to production. But, hopefully, you run your tests and all that stuff first because as As we get into these stories, you will see that there are a lot of people who do not run their tests before pushing to production.

And it well, they're gonna end up with more bugs in their application, and then Sentry is going to have to find them for you. So head on over to century@century.i0. Use the coupon code at tasty treat, all lowercase, all one word. You'll get 2 months for free,

and you can, you know, you can squash those spooky creepy crawlies in your application. We are also sponsored by Linode, and one of our our spooky Key stories of of the next episode that we have is somebody at Facebook accidentally blew out an entire DNS and took Facebook out.

So Or Facebook was trying to run their own DNS. Should they be doing that? Probably not. They should be using Linode. Linode is cloud computing. You probably know them for Hosting Linux servers, but they have all kinds of other products and solutions. Specifically, one is they have an entire product to manage your DNS, To import stuff, they're highly available.

You could automatically manage your DNS.

Also, Facebook should just be using this thing instead of trying to do it themselves because They could've called up Linode and say, hey. We goofed up. Let's roll that sucker back.

So, Linode, check it out. They're gonna give you a $100 towards Hosting or or any of their products, which is pretty sweet. So check it out at linode.comforward/syntax.

Sick. Alright. Let's Get into the spooky episodes.

These ones are just short little ones. In the next episode, we have much longer stories.

These are ones that we've been submitted over Twitter, so this is the first one. I've been laughing about this one all morning. So this developer worked For the largest Bible app available on iPhone and Android, and they were releasing a he says, We were about to release a feature for our Bible app called Stories.

So, like, kinda like Instagram stories, but in the Bible.

And I created a CMS right before we launched public testing. I remember a verse that I chose, and it goes like this. Listen to me, you fat cows living in Samaria. You women who oppress the poor and crush the needy and who are always calling out to your husbands, Bring us another drink. So that's Oh my gosh. I remembered at 3 AM and fixed it immediately.

CSF. You're watching watching watching this big product, and that's that's the way it's shown to everybody.

Oh my gosh.

Yeah. That'd be a big big deep horror story right there. No. Thank you. Oh. Oh, man. Okay. Why did I Issues that I have no clue, but it's pretty funny.

Oh my.

Oh my. Yeah. That's a good one. Starting off with a bang. I'm gonna be starting off with this, moving on to one that is less of a bang and more of like tippy taps here. My cat walked across my keyboard and deleted a client's entire production API FTP.

And let me tell you, this one is so believable because my dog loves to give me her paw.

She just goes, paw. You know, paw? And if I'm trying to develop and I'm sitting there and she's right next to me, she'll just keep giving me her paw. But since my hand's on the keyboard, she's just slapping my keyboard over and over again, and I've had files be, like, just all sorts of extraneous text being added to them. Never deleted anything off FTP, but that feels very, very possible to me.

Oh, that is hilarious.

Next one here is I managed to DDoS our production server 1 night at my last job by pushing an update to 40 clients that made a request To our API server that would endlessly retry on failure. I discovered it the next morning, and I receive a mountain of emails Of apps not loading. So a DDoS is a distributed denial of service. So the way that that works is often a malicious User will have a botnet of computers that request data over and over and over and over and over again From a thing, and if you do that too much, you can overwhelm a server. So this guy

himself. Oh, yeah.

CSF. That's a good one. That sucks.

Next one here, says, I used to develop marketing emails for previous job that sent emails to millions of customers.

There there were some hundreds of thousands of emails that weren't opening.

It turns out that Grammarly Marley was injecting code into the inputs in a service that we use to send the emails.

Yeah. That one sounds awful, especially because a lot of people are Yeah. But, also, it's, like,

out of your control, but still your fault because, ultimately, you use that tool. Yeah. That is that's a rough one. Doing a mistake in an email is the worst because once those emails are sent, there's literally no way other than sending a follow-up email saying, sorry. We did it to stop f. People from replying saying, hey, did you know that this happened? I've done it a couple times myself. Even a little spelling mistakes, You get hundreds of emails from people being like, hey. Did you know you did this? Next one we have here is, I wrote a Lambda. It process is and then saves image to s 3 brook. It's a Lambda's a script, and he saved it to s 3. That's where you save images. It returns the s three URL and saves it to the database. Okay. So if someone uploads an image, uploads it, and and you put the URL in the database, It defaults to staging bucket. There was a mismatch prod versus production that sent all the images to staging, So no one caught it until we purged the staging bucket.

So all the images were being saved, and then they just wiped them all out at once. Hopefully, Were you able to get that back? That sucks. Next one here. Before we started using Git,

I accidentally deleted a client's Custom website that I've been working on for 2 weeks. Yes, people. Back up. Back up. Back up.

Uh-oh.

Uh-oh. Uh-oh. Next one. I sent out 35100 overdue account emails 3 days after they had already been emailed While trying to make a change in our automated system, apparently, the depot got a lot of phone calls from angry customers who are trying to explain that they already paid their Yeah. It's it's so funny when these include numbers, like big numbers, they instantly get that much worse. Yeah.

And, like, also, like, you had to pay people to reply to these people and say, hey. It was it was fine. Like, sometimes you sit on hold for 45 minutes on a thing, and You don't think that, oh, maybe some developer accidentally sent out an email, and all their customers are freaking out right now. Here's another one. Once upon a time, I was giving styles via FTP,

and the file was corrupted, and I have to do the styles All over again. How many of y'all ever been out there and have been cowboy coding on a server with FTP? You're just you're updating a file and then yeah. It never works. And I remember, like, some point in your career, there's, like, oh, yeah. You should be using Git and this and that, whatever, And he's feeling like that was, like, super overkill. This was way back in the day when I first started. I was like, but I'm fine saving the file and pushing to FTP, And it's always so funny because I wish I would have heard stories like this earlier on in my career where I've been like, oh, yeah. I need a better I need to I need to invest that time ASAP to learn this stuff. Because people, backups and having your repo somewhere else, whether that is a public or private service, you know, we have public Or private repos for your private code if you need it. So, make sure you're you're backing up ASAP. Next one, coded functionality to manually approve users After registration,

so I can only let known users into the beta. Okay. So that makes sense. He's coded things up and don't wanna let people in until you mainly approve it. I forgot to remove the code when the app went live and made the guys from tech support tell users that they were just mistyping their password, And that's the reason that they couldn't log in. Normally, this happened on a big day where we spent 1,000 of dollars on marketing campaign to bring customers into our brand new location just to see them leave frustrated and never get back in again.

Oh,

I I just can't stop laughing at some of these. I I'm sorry. Like, obviously, it's in the past, and it's all been resolved, but just oof. I think the laughs are like it's coming from a place of love here. Yeah. The laughs is like the equivalent to seeing somebody like Have you ever watched, like, rope swing fails on YouTube? I love fails. It's that kinda lot. Yeah. It's like a We used to have a neighbor that would like, they didn't have a TV, and And they would just, like, knock on our door, and they'd be like, can I come watch America's Funniest Home Videos? It's on right now. It's just like, okay. When I was in college, They just come down into our apartment and watch America's Funniest Home Videos and just sit on the couch and just be like, oh,

that's that great.

Yeah. That's exactly it. It's America's Funniest Home Videos laugh. I love the,

the, like, the dance compilations where they just show people dancing at weddings and falling over and stuff. Love Me is America's Funniest Home Videos. I gotta give a shout out to America's Sound of Funniest Home Videos up there. Alright. Next one is I was working on a set top box that used JS.

I left Twenty boxes running an infinite download loop test overnight to downloading a Linux distro.

The Office ISP cut us off due to a d DOS attack on the distro.

Did you read this one? No. No. It's it's a similar a different DDOS one. Yeah. So there are 2 self DDoS ones in here. I I was, like, halfway through. I was like, wait. Wait. Wait. Wait. This is the same one. No. The whole office lost connection for a few days. You lost And then you d you DDoS ed your whole office, and and people had to that's some They turned the Internet off. That's some money lost right there. Woof. Woof. Woof. Woof. Not to mention, like, corporate Stuff is often usually not unlimited, you know, like, especially if you're Yeah. Paying for Bandwidth.

Oh, yeah. Paying for Bandwidth. No. Yikes. Big yikes. Next one, I misplaced a closing tag on a comment, and it removed the where clause in an SQL trigger that updated records with username and date time when they were updated. Users complained that changes were taking too long. I spent a Hours debugging the app, and I found it a week later in the database. Oh. So every time you update your username, it updates Hundreds of thousands of other

people as well. It's so slow. Talk about a moose boost. This right here is the little, like, pre course to what CSF. You you heard the main episode because it turns out a lot of people use the where claw or forget the where clause. There are SQL queries.

That turns out to be a a big theme here over the next the next episode.

Should we do a couple more? Yeah. This one right here. I once IP blacklisted my own balancer. It was dumb and hard to figure out as you might expect.

It was dumb and hard to figure out is,

the best way to describe a lot of our errors. Oh, that's so funny. Blacklist the thing that lets you in. That's good.

Let's see. Here we go. Weeks after a database migration, I realized the target at DB was case sensitive, but the source was not. Users ended up with new accounts via the same email address upon next log and lost access to their original data. Merging the data between the 2 accounts was quite a puzzle. Oh, That one, I feel that one deep in my bones because that seems like something that I could do. I did that very early on in my course platform where it was case sensitive,

And I just use the email as the user signed up with, and they signed up, they were logged in, and then they try to log in later, and they type it differently. And they're like, I swear my password is correct or, like, I reset it. And, like, the the password reset was normalizing it, but not the sign up process.

Yeah. So very thankfully, being a developer, you get somebody who emails you being like, I figured out why. It was because That's a classic bug. Yeah. Very easy just to to lowercase everything. Yeah.

Classic bug right there. I was once fixing a bug where some items weren't showing the values of stored in the DB. Turns out someone had hard coded a check directly into the view That tested it if an item was created before the hard coded date and displayed the wrong value if that was the case. I spent a few hours digging through controllers and testing queries. Couldn't believe my eyes when I saw it. It didn't cause any problems. It was just yikes.

Just yikes. Back in the early 2000, one of the team f. Had put in a funny error messages for testing that was supposed to be replaced before released into production. They were, except for 1, which needless To say the client didn't appreciate being told where to stick their input data, which, again, folks, come on. This is this is something that we see over and over.

Oh, man. Here's a good one. I think this one got, I think, more favorites and more retweets than my actual tweet to this is. Yeah, ratio. I had to chase down A bug that caused very broken styles on a banking platform, but only for 1 client and no one else. After a lot of head scratching, we discovered The client had a very aggressive swear filter, and it was blocking our CSS file that had the comment f IE 11 in it. Oh, yeah.

Yep.

Oh. Wait. Wait. Is it actually f IE 11 or, like No. He did this the whole swear. The whole swear. Yeah. The whole swear. It's so funny because we're PG podcast, so, And, like, trying to dance with the fudge, the whole the whole, yeah. That's funny. Oh, there's a lot of really good ones lower down. Yeah. I'm I'm going down. Somebody mentioned the HBO test integration email, which we did talk about. That's very funny. HBO sent out a test email, but luckily, it was not anything other than just, You know test.

I wrote an infinite recursion in our alarm system. Me and the CTO got about 20 1,000 text messages overnight, and both of our carriers blocked the phone number. The Twilio bill was high, and we had to switch everything over to a new phone number because the whole one got blocked.

Oh my god. Oh. Yeah. That's messing around with some stuff because how do you unblock a phone number? You know, you can't do that. You know, that is that is a tough one to come back from. I used an integer field in PostgreSQL when we should have used a big int.

It took 7 years for this bug to manifest itself, but the whole app was instantly down when it finally did.

CSF. That's like a regular y two k kinda situation right there. That was,

syntax when we first started the website. I labeled the episode numbers with leading zeros, 001, 002, and then I would just parse int on those, and it would give me the actual number. And then at a certain point, Because of decimals or something, I forget the actual issue. But at a certain point, I think it was, like, episode 192, the whole thing croaked.

It was because Yeah. I was storing numbers as leading decimal zero. So we had to go through every single episode and Change the episode number to a number and then just format it with leading zeros for display purposes.

That's classic. Classic. Next one here. My 1st web dev job in the early aughts, I was bringing in Google Analytics for a university. I managed To bring down the whole SIS, the student information system, which wasn't my responsibility and didn't get Google Analytics because it was misconfigured.

And when it saw the new cookies, it freaked out and just crashed.

Can you imagine, like, an entire university is down because of something you did? Yeah. That's at a lot of these. Like, the worst ones are, like,

taking down something for a lot like, the the more people that are involved, the worse they are every single time.

Because if it's affecting 10, 12 people, not a big deal. Affecting 30,000 people, boy, hold on to your buds. No. Thank you. Okay. Next one here is I ran a huge forum with several 100 thousands of posts and pictures in the early 2000. Yeah. They were they were everywhere back then. And to save space and money, I changed backup from daily to once a month. Then a week later, a Fourteen year old hacker managed to issue a drop command in the DB, and we lost 28 days worth of posts for when I restored. The hacker managed to get root access Be a poorly written chat program, which we had linked to in the forum, but it he wasn't smart enough to spook it spoof his IP.

They wrote spook, which I like to think is a really nice little typo for this episode. He wasn't smart enough to spoof his IP, and we were able to trace him With FBI help, his dad was angry at him big time. Wow. So FBI getting out here in the, forum space.

Don't take down anybody. Don't take this guy's forum down. The FBI is gonna be knocking at your door.

I changed the DNS and logged out of our hosting service account, and I didn't know the password.

Everyone knew it, but they didn't know the password either. So this can be subtitled as dev and customer immobilization 99%.

Oh, so that sucks. I wonder how you got in the 1st place, but that's what happened at Facebook is they just would literally not able to get into physically into the the building

At first, but also, like, they literally couldn't get into the servers because they were not on the network. You know? Yeah. That's a big one. Well, I would like to say that we tweeted this 10 minutes before Facebook went down. I feel like we had a bit of a 10 minutes before Facebook down. Seemed like there was, like, a ton of outages and little goof ups over the course of that day. So that was, like, super funny. I I was just, like, watching this all this take place. I'm like, oh, yes. I'm gonna make sure really good content on that spooky episode.

Well, cool. This is a nice little introduction with a whole bunch of these really great little dev stories. We have some bigger, More elaborate stories in the next episode, and they're very spooky, and they will make you very they'll make you shake in your boots. They will make your git commits quiver. I don't know what else. I don't have anything else there, but we will see you on Wednesday for the tasty spooky treat. Peace. Peace.

Head on over to syntax.fm for a full archive of all of our shows, and don't forget to subscribe in your podcast player or drop a review if you like this show.