I have been reading articles on the state of the art in computer gaming technologies from the August issue of the Communications of the ACM. One in particular, Scaling in Games and Virtual Worlds by Jim Waldo, describes the (up to now) unique characteristics of computer gaming from a scalability perspective, which to my surprise began to resonate with my vision of the future of enterprise IT, particularly healthcare enterprise IT and health informatics.
At the risk of plagiarizing, I'm going to reproduce the entire introduction, which is a very small part of the whole article, and is aimed at enticing you to subscribe to the magazine (by joining ACM) or pay for the article. Here goes:
I used to be like you.
I used to be a systems programmer, working on infrastructure used by banks, telecom companies, and other engineers. I worked on operating systems. I worked on distributed middleware. I worked on programming languages. I wrote tools. I did all of the things that hardcore systems programmers do.
And I knew the rules. I knew that throughput was the real test of scaling. I knew that data had to be kept consistent and durable, and that relational databases are the way to ensure atomicity, and that loss of information is never an option. I knew that clients were getting thinner as the layers of servers increased, and that the best client would be one that contained the least amount of state and allowed the important computations to go on inside the computing cloud. I knew that support for legacy code is vital to the adoption of any new technology, and that most legacy code has yet to be written.
But two years ago my world changed. I was asked to take on the technical architect position on Project Darkstar, a distributed infrastructure targeted to the massive-multiplayer online game and virtual-world market. At first, it seemed like a familiar system. The goal was to scale flexibly by enabling the dynamic addition (or subtraction) of machines to match load. There was a persistence layer and a communication layer. We also wanted to make the programming model as simple as possible, while enabling the system to use all the power of the new generations of multicore chips that Sun (and others) were producing. These were all problems that I had encountered before, so how hard could these particular versions of the problems for this particular market be? I agreed to spend a couple of months on the project, cleaning up the architecture and making sure it was on the right track while I thought about new research topics that I might want to tackle.
The three months have turned into two years (and counting). I've found lots of new research challenges, but they all have to do with finding ways to make the environment for online games and virtual worlds scale. In the process, I have been introduced to a different world of computing, with different problems, different assumptions, and a different environment. At times I feel like an anthropologist who has discovered a new civilization. I'm still learning about the culture and practice of games, and it is a different world.
The next section is entitled "Everything You Know Is Wrong" - and from a computer science perspective it is true, with respect to gaming now and healthcare enterprise IT as it will soon be. Let's explore why.
Waldo writes,
The computational environment for online games or virtual worlds is close to the exact inverse of that found in most markets serviced by the high-tech industry. The clients are anything but thin; game players will be using the highest-end computing platform they can get, or a game console that has been specially designed for the computational rigors of these games.
These client machines will have as much memory as can be jammed into the box, the latest and fastest CPU, and a graphics subsystem that has super-computing abilities on its own. These clients will also have considerable capacity for persistent storage, since one of the basic approaches to these games is to put as much information as possible on the client.
The reasons for this are related to one of Waldo's epigrams: "Latency is the enemy of fun." Latency refers to the lag between an event and the expectations of perception the event generates in the mind of the player.
Games simulate a virtual world, in most cases purely imaginary, in which participants are role players, and the success of the game depends in large part on maintaining the characteristic "willing suspension of disbelief" on the part of the participant that computer games hold in common with theater and film.
In virtual-world games, three factors predominate in maintaining the willing suspension of disbelief: the compelling nature of the world's story and metaphor; the credibility of game artifacts and game-generated dialog; and the social and physical laws of the virtual world. Latency comes into play in the virtual world's physics, and also in the timing of dialog interchange. If one is engaged in a sword-fight with a generated character (or as it is commonly called, avatar), computation must keep up with expected perception of object movement vectors and the results of "collisions", and also make sure the sounds match reasonable expectations (e.g., the thud sound of an ogre hitting the ground must coincide reasonably well with the visual depiction of the impact).
In a healthcare enterprise, as it becomes increasingly dependent on its computing systems (the "wired hospital"), latency is the enemy of health, in some cases even of life itself. On the surface the reasons are somewhat different in this case than in the gaming world, but not so different as one might expect.
One example of latency from the human-computer interface is the delay that occurs between the click of a button and the changes to the computer display that result from the action the button click is intended to initiate. Another more fine-grained example of latency is a delay in the movement of the pointer cursor on the screen after the movement of the real-world pointing device.
A risk scenario that ties both these examples together can be seen in the possibility of over- or undershoot, where the pointer in the second example will end up pointing to a different screen region than the human user intended. If this inaccurate movement is followed by a click, the click may trigger an incorrect underlying action, such as setting an incorrect alarm threshold on the read-out of a vital-signs sensor attached to a remote patient. If the display updates too slowly, the care provider may be in too much of a hurry to wait for the visual confirmation that the level is set correctly.
Another latency-related justification for the fat client in gaming has to do with much larger-grained latency issues, such as network downtime. In a multi-player game, especially one that is massively multi-player, every effort is made to minimize dependence on the network by moving as much computation and state maintenance as possible onto the client, but networks will go down, and for non-deterministic periods of time. The success of the game in the marketplace depends to some degree on maintaining the virtual world of the game as seamlessly as possible through such outages.
In gaming, latency is dealt with by taken into account a factor I have long referred to as intermittent connectivity, which is the assumption that network downtime is a natural and inevitable occurrence; and another factor that I would call the assumption of continuity. This is the assumption that simulation data points vary over time in accordance with reasonable laws and heuristics. The assumption becomes increasingly untenable over time, in the game as in the real world, because random or externally-originated stimuli interfere with the algorithmic execution of these reasonable laws and heuristics. Nonetheless, to the degree possible the game system should maintain the illusion of continuity by extrapolating from the last known state of the larger world according to the applicable laws and heuristics.
Healthcare enterprise IT in a wired hospital faces a similar requirement. Client machines on which lives depend need to be able to degrade gracefully and minimally, continuing to function usefully in the absence of direct connection to central data processing and storage systems.
The care providers are the ultimate "fat clients" of course (independent of their individual body-mass indices), exercising judgment based on knowledge and intuition, but given the volume of information involved, it is critical that the computational and communications affordances proximate to the provider not simply freeze up when the network goes on the fritz. In a cardio ward, for example, when all the vital-signs sensors go out, some patients will be in more stable condition than others, and the nurse or doctor must mentally triage the ward population to determine which patients are most likely to require attention; these patients should be visited first and most often until the sensor network is restored to operational status.
There are more parallels between gaming and enterprise IT, but I think the situation can be summed up as follows: both the gaming community and the healthcare enterprise IT community face the challenge of maintaining a large-scale simulation of some world - virtual or real - and the critical factor in both disciplines is scalability. The gaming world may have things to learn from the enterprise, but the converse is more likely to be true: computer gaming is much closer to the state of the art in exploiting information and communications technologies. As Waldo puts it,
With the possible exception of the highest end of scientific computing, no other kind of software has ridden the advances of Moore's Law as aggressively as game or virtual-world programs. As chips have gotten faster, games and virtual worlds have become more realistic, more complex, and more immersive.
As they become increasingly dependent on information and communications technologies, healthcare enterprises must learn from whatever other disciplines have the most to teach them. The gaming world is a cardinal exemplar. will the healthcare IT and informatics gurus perceive this and exploit the opportunity, or will they adopt a "not-invented-here" attitude or suffer from narcissistic blindness? only time will tell.
Comments