In February 2008 I was working as an IT consultant and my client was the Portuguese tax services area.
When I started there, about 15 months earlier, I was given a desktop with a dual core processor, 1 GB RAM, 70 GB HD and a lousy CRT.
The computer was not state of the art but it was OK for the job. Back then I had the usual software kit (MS Office, McAfee, Email, etc.) and developed J2EE applications, using Spring, Eclipse, BEA Weblogic and Oracle 9i running on Windows XP SP2 English version.
I had power user privileges on my computer and executed everything locally, except for the database, witch I accessed remotely.
I then needed to work with other stuff, like BEA NetUI, BEA Workshop Integrator and other stuff, so I got a 1GB memory upgrade. This allowed me to continue to be as productive as I was when I first started there.
But suddenly, in December, a "brilliant" mind from the system administration department executed of the most stupidest ideas I have seen in IT: change all the IT department desktops, which were somewhat like mine, by virtual machines.
So, I started January with a desktop, which is similar to my first computer, 2 GB RAM, dual core processor, 60GB HD, the same lousy CRT running Windows XP SP2, Portuguese version.
I login into the system, using domain, let's call it X, and I access a "fantastic"desktop, with a picture of the building where I work, plus:
- Access to domain X but with no privileges what so ever on the system, I can't even save a bookmark on IE;
- MS Office is installed, but I cannot keep files locally on the computer;
- Access to a shared network drive;
- MacAffee anti-virus running, with "paranoid" configuration which cannot be changed;
- MS Outlook for email, which works fine but I must use the web interface due to a stupid policy;
- MS Virtual PC with a Windows XP SP2, English version.
Since I cannot work in this system, I must start the Virtual PC with the XP image provided by the system administration.
After the virtual system boots up, I must login into the system using domain X (yes, the same domain) but now I have administrator privileges. Here's what I get from the virtual system:
- Access to domain X with power user privileges on the system, I can install, uninstall and kill applications;
- MS Office is not installed and cannot be installed, due to a stupid policy, so I must copy-paste office files between the real and the virtual systems when I need to read a Word documento or to check an Excel file.
- MacAffee antivirus running, with "paranoid" configuration which cannot be changed;
- 1GB to 1.5GB of memory available, the computer has 2GB but it is not possible to use much more than 1500KB.
Here's my first conclusions:
- Yes, I do have completely different privileges on the same domain, depending if I'm working on the real or on the virtual system.
- Yes, I do have MS Office on the "wrong" system, I am forced to checkout files from the version control repository in the virtual system, copy the files into the real system, edit the files in the real system, copy the files back to the virtual system and the check in the files, also in the virtual system.
- Yes, I do have two anti-virus, that I cannot configure to be less intrusive, which sometimes scan the same files at the same time.
- Yes, I do have two operating systems running, but one is actually resting or doing nothing at all.
- Yes, I do have 2GB of memory but cannot use more than 1.5GB beacause the operating system that is doing nothing needs 0.5GB.
- Yes, I do have a standard hard disk but cannot use it since I cannot save files into it.
I though that was stupid enough and it couldn't possible get worse, I was completely wrong, the nightmare had not even started...
I had to reinstall everything on the virtual system, and that's when I really understood how bad this architecture is.
When I installed the first application, it took ages... It's not difficult to understand why, a simple explanation is: any disk or memory access must go through the virtual anti-virus, the virtual operating system, the virtual disk and then it must go through the real anti-virus, the virtual operating system and the virtual disk. I do know that not all access are this bad, but the overall outcome is bad enough.
I made some measurements and here's the conclusions: this new system architecture is 4 times slower! Here's an example, a tipical BEA Weblogic 8.1.5 installation takes about 5 minutes on a regular computer, it took 15 minutes. But the record was from BEA Weblogic Integration 9.2, it takes 15 minutes on a regular computer but it took 60 minutes to install on the virtual system.
For the skeptics, yes, I did used a chronometer for time measurement.
An installation procedure is done once in a while, so it was annoying but I could live with that.
But on a working daily bases, I cannot live with such a slow system.
I continued my measurements and concluded that each consultant doing the same job as I do has seen a productivity decrease of 90 minutes per day. Yes, each consultant is one hour and thirty minutes slower than before, per day...
This makes 33 hours slower per month, on 40 hour labor week, it's almost one week of vacation per month... The client to which I'm working has over 40 people doing the same job and I do...
And we're not cheap...
When I showed these number to my superior I asked him if I could use my laptop and go home 1h30m early each day... He and I laughed. :)
When I asked him the killer question of "why" were our desktops migrated into such a stupid architecture, he could not tell and has also not received any answers from the client.
I can point a lot more reasons not to adopt such a stupid architecture, but have not found a single reason for doing so...
The client has shot its own foot: it should give us the best environment possible so that we can be high performers, but actually it is slowing us down and paying us for it...
So, if you want to do nothing and get payed for doing it, just find a client which is at least as stupid as this one. Find a killer combination like the one I have found: stupid system administrators and dumb organization that cannot even fight back, even when people from the organization complain...
This is definitely one of the stupidest ideas I've ever seen from system administration (and believe me I've seen a few)... But then again, it may be just me, after all, I do have bad temper...
But the story of the virtual machines continues...
I have complained about the 90 minutes performance decrease per day per developer to my manager and I also have reported it to the client in the monthly report.
One Thursday we had a meeting with the client and one of the topics was the virtual machines performance.
My client had been warned by the 90 minutes performance decrease, my manager has done his job right, and wanted to know what was going on.
She started by performing a simple calculation, multiplying the 90 minutes for all the resources she has that develop using the virtual machines times 22 working days and came up with 1600 hours.
1600 hours per month that she's paying us for doing nothing! She freaked out.
I've explained her that the system is running two operating systems, two anti-virus, we can only use 75% of the available RAM, the processor is never 100% available and any I/O disk operation is extremely slow.
The guy responsible for that 1600 hour loss had no words to justify why she was paying 1600 hours per month of inactivity.
After some discussion, the guy came up with the following solution: double the RAM of all the machines. This solution is totally lined up with the "brilliant" idea of the virtual machines implementation: it is totally stupid!
Instead of admitting that the virtual machines did not work and revert the process, he decided to buy more RAM for each machine. This will increase the client costs and even with 4GB RAM, he admitted that:
1. The developers will not be able to use the full capabilities of the machine.
2. The anti-virus will still be intrusive and will still decrease the developers performance, since nothing will change.
Lesson learned: if you made a mistake, do not fix it, just make slight variations of the mistake and eventually everybody will drop the subject...
./M6
Posts
