The last two weeks have been insane here. Between severe performance issues at a new member going live Monday--with five times more customers than the previous record holder, and twenty times more records in the contact tracking system--and a 3:00pm today deadline for an enhancement project, I haven't had a moment to breathe at the office. Nor at home, some of the time: I had to get up at 5:30 yesterday morning to send an update to the aforementioned new member, and then I had to send them yet another one when I got home yesterday evening. Total of eleven hours' work, Thursday.
I've spent long tedious hours testing SQL performance, tweaking it little by little and trying again and again, to save a few milliseconds here and there in the hopes they'll add up. I've found that "not in" and "not exists" clauses are bad for performance; that "or" clauses are often better replaced with unions; that the smaller you can make the initial resultset before joining it to anything else, the better, to the point that an embedded select joined to another table might be better than a straight join with conditions...etc.
Their system had been found to be grinding to a halt several times, and we could never discover why, but yesterday, I got a call from Jim. The member rep had had an epiphany. Each of these occasions had been preceded by an attempt to create a contact record from the IVR/Caller Id system--for a phone number that did not match an existing customer in our database.
Sure enough, in that particular situation, the search criteria resulted in an attempt to load every single contact record into the screen. This totals nearly two million records. Eep, to say the least.
But that one was an easy fix. "So you aren't going to give me a customer number? Here, have an empty resultset." We'd not managed to catch it previously because a) our test system for the IVR interface always returns valid customers (I think I'll be having that changed ;-) and b) our test database has next to no records in CT. Wonder why no other members have had it...or maybe they have, but their databases were small enough that no one noticed.
Then there was the enhancement project. Six or seven CR's, all of them simple little tweaks to a screen, but one of them dependent on coding from another programmer--who finished his part today at 1:15. Remember the part where the deadline was 3:00pm today?
But I got my bits done and checked in the whole lot about 2:20. Then I merged several other little fixes back to the previous release, and since then, I've been relaxing. Had a nice conversation with Tammie, who had had the misfortune of calling me at about 2:00 and I had to tell her I couldn't talk. Surfed a bit. Read the new
Left Behind Friday post. Pulled together the weekly bill list for discussion with C & T. Wrote this.