The larger square shows the total area of the small virtual world. Each smaller square represents a microcell, an area of the world to be assigned to and updated by a single worker node. The worker nodes are shown by the three colours in the images below, which are each viewed from the central server. The simulation was run for a number of minutes, with cells being redistributed dynamically between nodes. Numbers at the bottom right indicate the number of minutes the system had been active for when the screenshot was captured.

This image shows the view from a worker node, each blue dot represents a single entity following a path. This node is completely under-loaded just so that the entities can be seen clearly, when thousands are processed it can be hard to pick out individuals.

This shows the use of varying microcell sizes. The use of different sizes didn’t actually make a difference to average load, maximum load or load disparity between worker nodes except when particularly small cells were used. This caused a great deal of load for the central server, a huge increase in the number of transfers required and as the images show prevent the world from being re-partitioned in a reasonable period of time.

Finally, this image shows the cell distribution between worker nodes when the number used was scaled to match the problem size. As the number of entities was increased and then decreased, the number of computers used was scaled similarly.

That’s all for now, I’ll do my best to go over some details when my laptop has been repaired.

Additionally, the framework now supports both hexagons and quads as cells. Hexagons have the advantage that only 3 cells meet at any corner, limiting the maximum number of nodes which might be involved should an event take place there. Here’s some images of the view from the server with the most basic (block) distribution of cells:

A major part of the project involves distributing microcells (fixed areas of the world) between a series of worker nodes both statically and dynamically. In order to test methods of achieving this, I created a small testbed application which simulates the task without networking. It is capable of using various methods to distribute cells, both at the start of the application and as it runs, both controlled by a central server and initiated by worker nodes and gives graphical and numerical feedback.

The images below were taken with a greatly reduced number of cells, and the work involved in processing each cell is artificial: there are no pathfinding characters yet. However hopefully it demonstrates to some extent how division of the world takes place, and allows me to easily test distribution methods before using them in the proper simulation.

Coloured by number of characters per cell.

Initial block distribution.

Random distribution for comparison.

Block distribution with dynamic passing after time.

As the actual application I am working on is written in C#, I used XNA to create the images here. As it is possible to render XNA frames to a windows form application with little difficulty, this is also the approach I will employ for visualising the complete project.

Today we were tasked with presenting our research topic to our lecturers and classmates. As you can see from the slides above, we were given only a short period of time, 3 minutes at most, to cover background information, our own solutions, and the significance of our work.

Since 3 of my slides comprised only images, here are some additional notes:

Simulate characters in a virtual environment
The simulation I will be considering and using to access my project is that of multiple autonomous characters navigating a virtual environment. They might perform multiple trivial tasks or be processed in different ways so as to create a situation involving task parallelism rather than data parallelism.

Distribute processing work
A central sever and database will be created, passing necessary data and information to other nodes. Each node will be assigned a part of the virtual environment and the characters with to process.

Balance workload
Rather than splitting virtual world space evenly between nodes, giving consideration to the spatial distribution of characters is an obvious improvement. Although using a quadtree (or other appropriate data structure) would facilitate partitioning the space based on population, it would be necessary to consider and perhaps attempt to reduce the overhead involved in moving characters between servers.

Following the presentation, I was asked two questions:

Obviously at some stage the overhead involved in adding additional nodes will outweigh the benefits of the additional processing power gained, how will this be taken into account?
Amdahl’s law can be used to determine the improvement expected from adding additional processors to a system and as such the theoretical maximum gain in performance, given the percentage of the program which must be processed in serial. While work can be done to minimise the amount of work which cannot be parallellised, it certainly can’t be removed completely and given the networking component of the system it will always be a major consideration.
Measured and theoretical results will be drawn upon in my dissertation to help reach conclusions about the success and future potential of the project.

Extensive processing of autonomous characters is a very different situation to the work required by a server for MMO games, how can this project relate to that?
There are large differences between the situation I hope to simulate and a game server, not having to deal with hundreds of client connections for example, however I believe they can easily be related and intend to demonstrate this in my dissertation. Most importantly the simulation involves processing a large volume of spatially distributed data, using multiple computers, subject to strict temporal constraints – very much in common with game related situations.

Hey, sounds like a cool idea. Sounds like it might be quite a difficult project to undertake considering the need for multiple computers to be running the same simulation.

Although multiple computers will be used for the simulation, bear in mind that I can run both a server and several client nodes on one machine during implementation and to perform limited tests. I will need to make use of several computers to properly test the application, especially when it comes to networking and synchronisation issues, however hopefully this can be done in the evening, at home, or when the labs are less busy.

Will you start from one computer and then work you up by having more join the simulation?

Partially. Obviously my aim would be to start with the simplest working solution possible, and build from there in several steps:

1. Experiment with multi-threading techniques within a single application.
2. Experiment with networking between several machines.
3. Combine these approaches to create a very simple client-server framework for distributing tasks.
4. Design and build a simple basis for the simulation on top of this – get characters being processed on multiple machines.
5. Expand on the simulation and improve the underlying architecture.

Or will there be a minimum to start with and also maybe a maximum?

The number of machines which will be useful for the simulation will depend on how complicated the processing required is – for example how many entities are created. If the project is successful the simulation should run, albeit slowly, on one machine and faster as the work is distributed over multiple machines, limited by Amdahl’s law. I doubt I will achieve this ideal and the research question can still be answered even if the simulation runs slower in a distributed situation.

I take it the actions performed will just be general tasks such as moving place to place, interacting with other characters or objects, etc?

I am still deciding, to some extent, what tasks would be performed. It is important that more than one task or is handled as this creates a different type of simulation from performing an identical process on each piece of data (character).
My initial plan is to have the character choose a point in a simple 2D city to aim to go to, and then have them navigate there. Next, collisions with other data (characters) could be considered, greatly increasing the complexity of the situation. Finally, the characters could perform simple interactions with items in the world or other characters; However, I think it is unlikely I will have time to implement this option.

Will these action just be randomly assigned to any computer as long as they are free to do so?

This is a major part of what I will be investigating. Tasks will most likely be assigned to nodes spatially – in terms of an area of a 2D map, although other methods will be tested. I would especially like to try partitioning the world based on the density of characters in an area in order to try and create even workloads for different machines.

Or will certain task that may need more processing power be sent to the computer with the best processing power?

This will really depend on the hardware I have available. Ideally, I would like to measure each node’s networking and processing performance and base the amount of work distributed on this. If it is possible to use my laptop on the uni network then there will be more opportunities here than if I am working with identical lab machines. It might also be possible to simulate different hardware using virtual machines.

Thanks again for the questions!

Although I haven’t had as much time as I would have hoped for to construct another research question and I realise my ideas still require a lot of refinement, I have a (very) rough first draft of my proposal below. Hopefully I can re-write and revise this based on any feedback I receive. It’s all text from here on in because this is serious business – photo thanks to Daniel Conway..

Introduction

The aim of the investigation is to demonstrate how a series of computers, working in parallel, can be used to process and manage entities in a simple virtual world. Techniques to manage distribution and handle a range of tasks concurrently, as well as for dealing with scalability and fault tolerance will be considered.

Motivation

Concurrency has become more and more of a prominent issue within software development, even a necessity to make the most of modern advances in hardware (Davies and Shrout, 2009). With the majority current gaming systems sporting multiple-core CPUs (Valve Corporation, 2009) and consoles tailored towards parallel computation (Stokes, 2008), the importance and currency of this issue is particularly applicable to games programming.

However, nowhere is this issue more apparent than concerning the development of servers for Massively Multi-player Online Games. These systems must be capable of managing a range of tasks concurrently, dealing with variations in load while still maintaining a stable throughpu (Waldo, 2008). Even if it were possible to use a single machine for this endeavour, running costs alone(Atwood, 2007) make employing a variable number of distributed nodes a worthwhile consideration.

As such this specialised area of games development combines the problems of developing parallel applications with the issues of distributed computing, but unlike render farms which can take hours to produce movie frames (Christensen, P. H. et al, 2006), or scientific investigations which might extend for years at a time (SETI@home, 2009), results must be calculated in fractions of a second to allow the game to progress.

Improvements to the organisation and throughput of clusters of locally distributed machines can offer clear benefits to the scope and hardware requirements for running massively multi-player online games.

Research Question

How can the processing of a real-time virtual environment benefit from parallelisation over multiple distributed computer systems?

Addressing the Question

Initially, a simplest-solution framework will be created to allow for parallel tasks to be distributed over multiple networked systems, overseen by a central server. The desired simulation will be built on this and will involve the creation of a simple 2D environment, and characters which will be made to navigate and perform actions within it. From this point the simulation will be optimised and improved, in particular investigating whether and how load balancing can increase performance and minimise hardware requirements.

Although performance of the simulation will be measured in terms of throughput, a range of factors will be considered when evaluating the solution and simply allowing processing to occur over multiple systems should answer the question to a large extent.

Resources and Requirements

Although ideally a Beowulf cluster (Beowulf, 2009) could be set up to allow a range of existing software to be utilised, this would involve additional requirements – most notably the installation of the Linux (or another *NIX) operating system to multiple machines. There are few reasons why a similar setup could not be achieved using the Windows operating system currently installed on available lab computers. Multi-platform support could possibly be investigated through my own hardware, if suitable libraries and approaches were chosen for the project.

Although the project will focus on distributed computation, systems with multiple CPU cores should prove beneficial to the simulation. These should be available through the dual-core processors in the lab computers.

In terms of software, again the laboratory machines should provide the necessary packages, which might be supplemented with open source tools and libraries.

References

• Davies, L and Shrout, R. 2009. Who Moved the Goal Posts? The Rapidly Changing World of CPUs [online] Available from: http://www.gamasutra.com/view/feature/4168/sponsored_feature_who_moved_the_.php [Accessed October 2009]
   
• Valve Corporation. 2009. Steam Hardware Survey [online] Available from: http://store.steampowered.com/hwsurvey/ [Accessed October 2009]
   
• Stokes, J. 2008. Game Console Architecture In-Depth [online] Available from: http://arstechnica.com/hardware/news/2008/09/game-console-architecture-in-depth.ars [Accessed 2009]
   
• Waldo, J. 2008. Scaling in Games & Virtual Worlds. ACM Queue. 6(7), pp.10-16.
   
• Atwood, J. 2007. When Hardware is Free, Power is Expensive [online] http://www.codinghorror.com/blog/archives/000868.html [Accessed May 2007]
   
• Christensen, P. H. et al, 2006. Ray Tracing for the Movie ‘Cars’. In: Proceedings of the IEEE Symposium on Interactive Ray Tracing 2006, Salt Lake City USA September 18-20. IEEE. pp.1-6. [online] Available from: http://graphics.pixar.com/library/RayTracingCars/paper.pdf. [Accessed October 2009]
   
• SETI@home, 2009. [online] http://setiathome.berkeley.edu/sah_about.php[Accessed October 2009]
   
• Beowulf. 2009 [online] http://www.beowulf.org/overview/index.html [Accessed 2008]
   

Bibliography

As this is a rough draft, please see my relevant delicious links. These are missing references to books, ACM publications, journals I can/could only read within the university and older bookmarks which I have found useful, so I will attempt to update in the near future with a refined, current listing.

Having completed further research, I strongly feel that one of these alternative topics in particular would be better suited both as an honours project and as something I personally would like to investigate. Although this decision is not yet final and requires much further refinement, I hope that completing the original worksheets and (potentially) receiving some feedback will aid me in reaching a hasty yet prudent decision.

1. State the topic of interest that you may wish to take forward into a project.

Distributed Parallel Computing.
Splitting an appropriate problem across multiple processing units with independent memory, for example using separate machines on a local area network. The aim would be to establish how much of a performance gain (if any) is given above processing on a single machine, or is given by more intelligent methods of splitting workloads. Theoretical gains could be compared to those achieved in practice.

2. Indicate the issues surrounding this topic that you may wish investigate.

• Techniques and patterns for concurrency and parallelisation.
  • How can systems be kept in sync?
  • How can errors and faults be detected and compensated for?
  • Can redundancy be built in to accommodate the loss or addition of processing units?
  • But does it scale?
  • How does the solution allow for fair comparisons between approaches?
  • How can workload be balanced between different systems?
• Adapting parallel solutions to use across a network
  • How much data needs to be transferred?
  • How are varying transfer speed and reliabilities dealt with?
  • How is data loss compensated for?
• Available hardware and software
  • Must all hardware involved be identical?
  • Can the solution be ported to alternative operating environments?
  • What supporting libraries are suitable for this task?
  • Could graphics or console hardware be used additionally or alternatively?
• Uses or problem areas which could benefit
  • Rendering such as ray-tracing or ambient occlusion.
  • Grid based or particle based simulations.
  • Spatial, multi-character and world based simulations.
  • High dimensional noise generation.

3. Outline your initial ideas about how you would carry out the practical aspects of the project. i.e. what do you envisage yourself actually doing for the project.

As I mentioned above, I’m still unsure of the type of application I’d like to focus on for this project. Distributed rendering appears to be a very suitable process which I believe would be achievable in the limited time available.
However, I would be particularly interested in the idea of taking a huge number of characters within a map or area and performing operations on each of these: collision detection, pathfinding, flocking behaviours or primitive AI behaviours. In this way workload could be split by population densities at a given time or by given fixed areas, and comparisons performed. I also think this would result in an exciting, scalable demonstration if some simple visualisations were shown.

In either case, the practical aspects of the project would involve creating a flexible simplest-case framework to allow trivial computations to be performed remotely. This would then be expanded to deal with the problem chosen, again in its simplest form. The application would then be iteratively improved in terms of redundancy and fault recovery, workload balancing, optimisation, with depth and complexity increased, measuring performance at each stage.

Any feedback on the idea would be hugely appreciated, as ever.

Images thanks to nurpax