TicketPop Server Crash Raises Questions

One of the most controversial posts I've ever made was when I claimed that Banco Popular was full of bullshit when they claimed they had spent $2 million on a complete redesign of their Internet Banking site. Ahh, good times. Of course part of the flame war that ensued could have been because I implied that one possible explanation was that the programmers at Banco Popular suck. Again, boy people really seem to react strongly to the use of the word "suck."

Buying Metallica tickets on TicketPop

Now I want to go on the record as never being too impressed with TicketPop.com, but my experience last Saturday morning has so unacceptably horrible that I need to share my experience and opinion. At 9:00 a.m. Saturday morning the Metallica tickets went on sale and it's likely that thousands of people began hammering the site in their desperate attempt to get choice seats. Now it's never fair to say that a site is slow because there are so many failure points between your computer and a server, but it seemed that the site was quickly starting to slow down.

Tip: If you're ever going to buy tickets using TicketPop, make sure you create a login first, and be already logged in before the gates are opened and the thousands of requests start hitting the servers.

So following my advice, at 9:00 a.m. I was already logged in and trying to get just the right tickets. From my very first attempt it was clear that, like many other concerts, the primo seats had already been snatched up or reserved. These days the only way to get the best seats is to be a fan club member of whatever band you're looking to buy tickets for. And while some fan clubs are still free, Metallica (and AC/DC) memberships run about $45 a year.

So with the serving slowing down with every click, I finally got a hold of a couple tickets that I liked. I started the checkout process, which with a pre-established account, is just a couple of clicks. Then on the last step where you have to accept the Terms and Conditions (which you should have already signed off on when you created a login id, but whatever), when tragedy struck. The TicketPop server crashed. My connection was reset and I was in a state of limbo. Did my transaction go through before the crash? Well the only way to know for sure was to log back into the system and check my orders. Now it took about 10 or 15 minutes before the system recovered from the failure. When I was finally able to logon I discovered, to my horror, that my transaction had been lost. By the time I got around to securing a couple of more seats, they weren't quite as good, but I guess they were best I could do, so I was finally able to get my tickets.

Analysis

Now I'll be the first to say, that sometimes bad things happen, and there's just nothing you can do. Hell, one day I shocked the senior management of a company I was working with by saying that "Shit happens," when they were pressing me on the deadline for a project. However, I think that what happened last Saturday with TicketPop was unacceptable and reeks of negligent mismanagement.

I used to always piss off marketing executives when they would walk into my office and explain some crazy promotion they wanted to do and then ask me if we could do it. My standard answer was always, "Sure, we can do anything you want as long as you're willing to spend the money and resources necessary." That seemed to baffle them, their eyes would glaze over, they would mumble something, and then wander out of my office.

If you asked me, having a ticket server crash during a peak demand suggests the inadequate allocation of resources. Of course, as I tried to mention above, in any technology project there is a delicate balance between the "cost" of the solution and the "benefit" that technology provides. But failure is failure, there are NO possible explanations, exceptions, or qualifications.

The big question here is: "Was this the case of a good-enough solution created with a limited budget collapsing under excessive demand or is the case of the best solution possible with an adequate budget failing because of human negligence?" We'll probably never know, but as this is not the first time I've heard of TicketPop crashing during peak demand scenarios, I'm thinking that the failure is a little of both.

Is fool-proof software possible?

According to the theory of diminishing returns, the answer is NO. For as the effort ($) increases, the reliability gained becomes gradually less and less. However, just for the record, I DO believe it is entirely possible to build a reliable ticket purchasing system that can withstand 50,000 simultaneous users. With the dirt cheap cost of hardware, automated web software testing tools, and the relatively well understood requirements of an e-commerce site, there is no reason that should prevent Banco Popular from creating a crash-proof TicketPop web-site.

Let's get real here. Last Saturday morning, how many people do you think might have been trying to buy tickets simultaneously. My guess is somewhere between 1,000 and maybe 5,000. Now if you really want to give them the benefit of the doubt, you could say 10,000 tops. Ladies and gentlemen, in the ranks of e-commerce sites, 10,000 simultaneous users is, how do I say this, a valley of demand. It's the equivalent of Mr Universe, Arnold Schwarzenegger, picking up a five pound barbell.

Now I know what you're going to say, that we are not talking Amazon, Buy.com, or Overstock.com. This is a local site dedicated to Puerto Rico, in other words a fairly predictable peak stress load. However, I would remind you that Banco Popular is entirely capable of building a 99.9999% reliable system. That's how these types of problems are managed. An organization determines to what place (how many 9's after the decimal) do they want a system to be available? Please remember that Banco Popular, oops I mean Evertec, does know what it takes to build a very reliable system capable of handling tens of thousands of simultaneous transactions in the case of their cash cow "A Todos Horas" (ATH) network.

What is good enough software

So if you are Evertec, let's ignore for the moment the fact that they know exactly what the maximum peak demand for TicketPop could be. Let's say instead, they put little thought into peak demand, and therefore do not plan to handle the demand, nor do they ever test the web site simulating peak demand.

Now accepting the case that peak demand is not a requirement, then Evertec probably followed their ordinary web application software development process. Now one of the things that burned my britches way back in August 2008 in the Internet banking article, was Banco Popular's implication that their $2 million dollar investment would advance the state of Internet applications in Puerto Rico. Because in this case, it is clear that TicketPop didn't benefit much from their $2 million Internet banking system.

No, a "good enough" system starts with as small of a budget as possible to produce a workable automated ticketing system. If I had to hazard a guess, I bet that Evertec spent quite a lot less than $2 million to develop TicketPop. In the "good enough" scenario, management accepts from the very beginning the possibility that the system might have limitations. They might even be thinking, if the server crashes, where else are they going to get tickets?

Conclusion

As I mentioned before, if I had to make a guess, TicketPop crashing during a peak demand scenario is probably a mismanagement of resources. Very quickly, most system crashes are the result of: infrastructure failure, hardware failure, external software call failure, or plain and simple software failure. I've been in the Evertec server facility, so I'm going to scratch infrastructure failure off of the list. I'm also going to say, I hope they are not stupid enough to pinch pennies on hardware. You could probably pay off double redundancy of every piece of hardware with the fees TicketPop earns off of one sell-out show like Metallica. So that only leaves us with software problems.

I'd put my money on some kind of crash caused by the external calls out of the web application to the ticket database or the payment purchasing process. It would be that or some kind of memory leak, but in reality, I don't know what happened. All I know is that, one moment I was confirming my transaction, and the next moment the connection to the server was reset.

To conclude, I bet that for Banco Popular, the TicketPop system is considered an important, maybe even a critical application, but not a mission critical application. I bet that the senior IT management in Evertec knows TicketPop's limitations, and have drawn up short on testing the system, especially stress testing. So, in my opinion, they've mis-managed their resources by not investing more time and attention to TicketPop so that embarrassing outages like last Saturday's don't occur.

Unfortunately for us, we don't live in an environment that values trust or authenticity. If we did, Evertec would have already published an apology for the server crash having inconvenienced their customers. I'm sure some of you might be laughing at that notion, however, when you "get" that everything you do on the Internet is under a microscope, then nothing is more important than maintaining your customer's trust. Then again, when you're a monopoly, you really don't care what people think about your business. The sad thing is that if you want to go to the Metallica show, then you HAVE to use TicketPop, so why should they care?

7 comments:

Anónimo

11 de diciembre de 2009, 15:03
Permalink this comment

1

dijo...

Tu vas pa METALLICA con tu mama? en caso de que algo vaya mal, para que te abraze y te diga que todo va estar bien mientras lloras y preparas tu proximo rant. Como que no pega Metallica y to este rant lloroncito sobre comprar tickets.

MC Don Dees

11 de diciembre de 2009, 15:53
Permalink this comment

1

dijo...

Actually it would be be pretty cool to go see Metallica with my mother, especially since she's been dead for 15 years. I'm sure I'd be the talk of the concert. Mano, did you she that guy dragging that corpse around at the Metallica show?

But you're right about one thing, my post is a rant, a rant about how pathetic it is that the largest company in Puerto Rico and supposed technological leader produces such crappy software.

Anyway, thanks for reading and sharing your thoughts, might I suggest, however, that you stay on topic and discuss the merits of my opinion, or by actually adding something to the conversation other than immature taunts.

Anónimo

14 de diciembre de 2009, 11:51
Permalink this comment

1

dijo...

I just love how someone (ie. YOU) explains an error of mismanaged resources being outrages due to the availability of "dirt-cheap hardware" etc. when you have to rely on someone else's blogging platform. Why not then try setting up your own blog on your own hardware and do your own hosting if everything is so cheap?

You're just like every other company out there that outsources your resources to supposedly save a buck.

Since you've toured the Evertec facilities, please enlighten us to the hardware being used and their related costs including infrastructure, electricity, maintenance etc.

You seem to know quite a bit, but for someone as smart as yourself, I would have expected you to have done some research on ticket pop and figured out that the ticketpop platform along with it's managers, designers and programmers came from Mediawire. And as you PROBABLY and OBVIOUSLY know, when a big company acquires a smaller company for their technology, part of the deal is to retain most of the team and promote it's executive management who in turn maintain control of their product. Therefore if you want blame anyone, then blame the original designers and programmers from Mediawire. They're the ones that probably "suck".

Most projects, when done in-house, would work way much better. But unfortunately most parent companies outsource and acquire technologies due to time constraints and the state of the competition and they forgo their own resources. This is nothing new. We people on the low end of the food chain gripe about this all the time.

Anyway, bashing people in general for something you experienced, especially your technological peers, is unprofessional. Things crash. Deal with it. When a plane crashes, their engineers don't suck. When a patient dies on the operating table, those surgeons don't suck. Sometimes shit happens. But blatantly ranting and going out of your way to critisize a group of hard working individuals and saying they suck for something they mostly had nothing to do with because of your lack of due diligence and neglectful researching -- And all because of insignificant thing as Metallica tickets makes YOU suck really bad.

Anónimo

14 de diciembre de 2009, 13:42
Permalink this comment

1

dijo...

Dude. All this whining makes me doubt your cred as a Metallica fan. You sound more like a girly Poison fan in disguise. If you were a true Metallica fan you wouldn't have relied on the wimpy internet and would have gone and bought the tickets personally.

MC Don Dees

14 de diciembre de 2009, 13:54
Permalink this comment

1

dijo...

First, I'm glad you love my work, I aim to please...

Uhh, just for the record, when a plane crashes, companies get sued for negligence. When a patient dies on the operating table, sometimes the doctor gets sued for negligence. When you cause a traffic accident and someone gets hurt, you can be sued. See a pattern? Really? You really wanted to go there? I think you may be a little overly sensitive.

Listen, the beauty of the Internet is that I can say what ever I believe to be true. I'm a customer and I think I received really crappy service, so I'm going to tell everyone what I think.

And by the way, I didn't say anyone sucked in the post. I said that they probably mismanaged resources leading to the failure. In fact, I didn't implicate anyone as you put it, "at the low end of the food chain." I think I said that it was probably a management problem.

As for TicketPop being a platform that EverTec or BancoPopular purchased. I'm not finding that link. The only connection I can find is that MediaWire.net registered the domain TicketPop.com. Do you mean EverTec hosts TicketPop on MediaWire.net servers? Or is it that GM Group outsourced the creation of the TicketPop software to MediaWire, which would explain A LOT!

Finally, I think you must be confused with the whole concept of outsourcing (especially hardware). We're just a couple of guys operating out of our houses, so we outsource our hosting, and anything we can, especially if it is for free. Of course, it's just because of the low cost of hardware that Google can offer so many services for "free." However, I wouldn't use Blogger if it crashed frequently. Oh yeah, you should also know that it is Google standard procedure to redo any software they buy. So if EverTec inherited a bad system from MediaWire, then perhaps it is time for them to look for a new solution.

MC Don Dees

14 de diciembre de 2009, 14:04
Permalink this comment

1

dijo...

Wimpy Internet? LMAO! Come on, you have got to better than that. All hail Satan! LMFAO!

Anónimo

12 de marzo de 2010, 11:21
Permalink this comment

1

dijo...

Evertec tiene excelente planta física y sistemas de computadoras. El problema es que los que trabajan alli no les importa lo que hacen, no aman su trabajo. Por que han sido victimas de años de abusos psicologicos, donde el mas chismoso es el que logra escalar!!! No solo Metallica...cuantas veces uno trata de hacer un pago en mi banco y no funciona......