Why I’m Excited About the Rewrite

August 23rd, 2007

I have mentioned that I am really excited about our rewrite several times, but I haven’t ever gone into much detail about why that is. I hope that we have learned from what worked and what didn’t work in the old Crown and Cutlass code, and applied those lessons to the new Protocce code. I’m sure there are still things that we will realize are not well done, but this is a much more stable base for future work. Here are several specific things I’m excited about:

Exception – We implemented the pcce::Exception class which we now use consistently for error reporting. Coupled with our greater focus on RAII (see below), I think we have made some serious progress. We used to use error codes sometimes and throw other times. Unfortunately, sometimes we threw strings, sometimes we threw char *’s, sometimes we threw ints, etc. This is much better. It does not descend from std::exception at the moment, but we are thinking about doing that.

Greater focus on RAIIRAII is really a great idiom for resource management in C++, especially when coupled with smart pointers like boost::shared_ptr. We have design for RAII from the start, so that should make things much more manageable. Again, I’m sure we have more to do in this area, but we have a good start.

Variant – At work, I have gotten used to using the Win32 Variant class in Delphi. While it certainly has its drawbacks, it sure can be helpful for doing things like reading text from a config file. Our variant class is cross platform, and based on boost::variant. I think it will change a little as we use it, but I’m pretty happy with it now. You wouldn’t want to use it when a statically typed variable will do, but for example the config system was really easy to do using Variant.

PropertyBag – Again, at work I got used to using “property bags” which are basically just a map from a string to a variant value. You wouldn’t want to replace all of your normal classes with a property bag (static typing can be nice!), but it can be really useful. For example, our configuration system is an INI file. That gets loaded into a property bag. When you initialize a subsystem, you pass its settings as a property bag. That way, you can change the config options around without changing the config object or the method signature. We have also used property bags to create “game objects” which are just property bags with a couple of predetermined fields for model name, texture, etc. The game programmer can put whatever else he wants in there.

Consistent use of shared_ptr – I really don’t mind managing my own memory, but once you start talking about exceptions and RAII, using some kind of smart pointers really makes sense. We are making pretty serious use of boost::shared_ptr, which has made a lot of things easy. No need to worry about who owns an object, as long as you have a shared_ptr to it, it will stay valid. Of course, we have to think about circular references, but overall I think using smart pointers is a big win.

OGRE – I think it’s pretty clear why I am excited about OGRE. Originally, the project was just an excuse to do OpenGL programming. I enjoyed that, but now I am more interested in other things. I’m a little sad not to get to try that from the ground up, but eventually we can revamp Protocce to use a custom engine if we want. Until then, using OGRE will simplify our code, standardize our models (our old model loading code was really picky), and open up the doors to easily implementing graphical improvements. It will also allow us to focus on the game itself, rather than the details of the graphics.

EventSystem – Our old code used to be very tightly coupled (see the SailingState for an example), so minor changes could not only cause a large number of units to need to be rebuilt, but also end up causing a large number of code changes. The event system is based on what is mentioned in Game Coding Complete and allows us to significantly decouple our code. Once an event is defined, you can register a boost::function to receive that event. Events can be dispatched immediately, or queued to be dispatched once per frame. I expect the way that will work is that game events will be queued, while internal “plumbing” type stuff will be sent immediately. As an example, the input subsystem needs a window handle to receive input. Rather than connecting it directly to the render subsystem (which would force you to initialize the entire render subsystem if you wanted to use the input subsystem), the input subsystem just sends an event that the render system receives. If you don’t want to use the render subsystem for some reason, you could just register a different handler for that event. Collin actually does that in the input test. As I mentioned in my previous post, the sound system is also event driven. It is running in its own thread, and all the main thread does is send an event that says “play sound X” or “stop sound Y”. That data is easily sent out to the other thread, where the actual sound code handles it. We can totally revamp the sound implementation and not touch client code at all. I think the event system is a much more flexible system and I think it will open up a lot of doors for us.

Improved resource management – I mentioned this a little in my last post too. Between the release of Alpha 1.4 and the decision to start the rewrite, I wrote a resource management system. It was pretty good. It would make sure that only one copy of a texture, sound file, or model was loaded at a time and used reference counting to automatically unload the model when it was finished. I have further refined it (partially based on the OGRE resource system) and posted a reply on gamedev explaining the Protocce resource system.

Anyway, there are other things but that is a decent explanation of the things that I am excited about.

Sound System Progress

August 22nd, 2007

I have been working on the sound system lately, which has been pretty enjoyable. I based the design on the old sound system. I’m not sure if many people saw that, because I implemented it (along with resource management) between the release of Alpha 1.4 and our decision to rework the code from the ground up. It was pretty slick from the end user’s perspective. The user could create a new SoundEffect object and that would acquire a pointer to the shared AL buffer automatically. That buffer was shared, so there was never more than 1 copy of “cannon.ogg” or whatever in memory at one time. It would also automatically unload the buffer when the last SoundEffect using it was deleted. The sound system also managed your OpenAL sources so that only a given number of sources would be created. If you tried to play more sounds than that, it would revoke the source from the oldest sound to play the most recent. That was pretty nice, but it was a little ugly behind the scenes because when I originally wrote it I was not yet aware of boost’s smart pointers. I had a feature where you could “lock” a source so it would not be revoked, but I never used that because I never updated the code to stream ogg files from the disk to use the sound system. As a result, the source management was much more complicated than it needed to be and it was still possible to run out OpenAL sources if you played too many sounds at one time.

I have gotten a good chunk of the way through moving the sound system into our new codebase. I think in general the sound system is much better now. It is a little ugly because it is running on its own thread, but I think that will be worth it. I have renamed the SoundEffect class to SoundBuffer (since it is a sound that is played from a single buffer in memory as opposed to one that is streamed from disk, which is done with the SoundStream class). It has basically the same interface as it used to, but now all the SoundBuffer does is fire off events to the SoundSubSystem (as you can see in the SoundBuffer implementation). Sources are still revoked just like they were, but now I am positioned to combine the source allocation for the SoundStream class too. I wrote a really simple class for reading ogg/vorbis files in a more C++ friendly manner. It allows us to use RAII and hide some of the oddities of the vorbis lib. I reworked our LoadOgg function that loads the entire contents of an ogg/vorbis file into a single OpenAL buffer so that it used the new OggFile wrapper, and it resulted in this diff. That is way more readable, so I’m pretty pleased with that. Now, I’m working on using that ogg/vorbis wrapper to implement streaming music off the disk. It’s nice to have all of my ogg/vorbis code in a single place instead of duplicating it like we used to. Maybe I should do something similar for WAV files. I also probably should rename all of our *Ogg* classes, functions and methods to *Vorbis*…

Anyway, it has been fun to rework the sound stuff using our improved resource management code and just the general better code base we have now.

New Computer

August 15th, 2007

*Warning* Gloating post ahead!

Over the past 4 years or so, I’ve had some computer drama. I bought a Dell laptop 4 years ago. It was about as good as you could get at the time. Pentium 4, GeForce 4200, 15.4 inch wide screen. It was pretty slick. Unfortunately, almost immediately I started experiencing problems with it. It would be running along happily and suddenly shut down for no apparent reason. No error message, no nothing. I shipped the computer back to Dell and a few weeks later I got it back.

It worked happily for 2 more years, when the video card went bad. It was still under warranty (and I thought it was a waste of money to get the extended warranty) so Dell replaced the video card. Apparently they didn’t have a 4200 any more so it was replaced with a GeForce 5600. Pretty sweet! Except that it ran too hot and my computer would again just shut down when it got too hot. After sending my computer back to Dell 4 or 5 times over 6 months (sometimes they wouldn’t actually change anything), they agreed to replace the laptop with a refurbished machine. Sweet! So again I had a really nice laptop, Core Duo, 17″ screen, GeForce 6800, life was good.

Then, 3 months ago, while playing Winning 11, the video card went bad (weird that its been the video card more than once). At this point it wasn’t under warranty any longer so I was stuck. A new video card from Dell was $300 (not including installation) and if I installed it myself, there would be no returns for a defective product. It wasn’t worth the risk or price.

Which leads us to the new machine. I decided I didn’t want to totally break the bank, but I wanted something I could upgrade later. So I got a pretty sweet processor, a Core 2 Quad. I decided most of the other pieces of a computer are easy to upgrade, but I wanted my processor to last. I don’t think I realized what a difference it made until I compiled Ogre. On David’s machine (which is the exact same laptop I had originally) Ogre takes 25-30 minutes. On my core 2 laptop, it took in the 15 minute range. On my new machine, Ogre compiled in 3 minutes!

All that to say I’m really excited about my new machine.

Simplified Protocce (no more “triscene”)

June 29th, 2007

We had this idea in the rewrite to split the scene into (at least) 3 parts: a physics scene, a render scene, and a game logic scene. The different parts would all run on different threads and run at independent speeds (e.g. the scene is rendered as fast as it can, but the physics is only updated 30 times a second). The threads would stay in sync with each other by passing events like “move node X to …” and “set material of node Y to …”. Our plan was to treat the render system as a sink, so it only ever received events. To avoid the physics and game logic scenes getting in each other’s ways, we were going to have objects be moved either by the game logic (for scripted camera movement, etc.) or by the physics system (moving around boats, etc). That way a node would only be modified by a single thread. I think that might work (maybe not), but we have decided to abandon all that and go with a more or less straight OGRE scene instead.

As I was implementing it, it seemed like there was just going to be too much overhead for it to really be worthwhile. All of the scene nodes were identified by name, so in my unoptimized first-shot implementation there was a lot of lookups in maps of strings. I think we could have improved that to the point where it might still be workable, but the real kicker was 3rd party libraries… We decided to do this “per-subsystem” sort of threading instead of something more like a work queue with worker threads because we wanted to leverage mature, 3rd party libraries. We are excited to get all of the power of OGRE without having to implement all of that ourselves. However, most of the 3rd party libs are not thread-safe or multi-threaded. We thought if we controlled the updates to events that got passed to the thread, that would be ok. CEGUI really killed that idea, though. It is not thread-safe, and in our system it would need to be accessed by both the render thread (to display the buttons) and also the input thread (to handle input events). From what I read on the CEGUI forums and saw in their code, that just isn’t safe.

We had to decide basically whether to continue with our “it might work” idea for multi-threading and possibly implement our own GUI system, or just cut out all of that extra work and use the libs more directly. We decided to abandon our triscene idea. The sound system will probably still be threaded, but the input, physics, rendering, and game logic will all happen on the main thread. That clearly isn’t the best way to go with multi-core processors becoming with more and more common, but I think trying to use non-thread-safe 3rd party libs in a multi-threaded way is just a loosing battle. The only real alternative is to write all of our own subsystems (i.e. no CEGUI, no OGRE, no ODE), but at some point we have to just focus on getting a simple, well structured, singled-threaded (and actually running!) version of Crown and Cutlass instead of diving off into all that.

Actually, we decided to abandon the “game view” idea for now as well. We couldn’t really map that onto the rest of our design. Rather than continuing to bat ideas around that I think sound good but can’t quite get pinned down, I am going to focus more on getting what we have running. I guess that’s 2 fewer cool things I am excited about that, but at least we are getting close to working in Crown and Cutlass again for the first time in almost a year and a half!

PS We tagged the SVN repo before we deleted all this code. You can see the tag here.

Quick Update

June 14th, 2007

It’s been a little while since I posted anything about what we are doing, so I thought I’d give a quick update. First off, I applied to some Computer Science grad programs for this fall. I applied to three PhD programs and one MS program, and only got into the MS program. While that was pretty disappointing, the good news is that it made the decision of where to go really easy! I was a little concerned about not having enough time to continue working on Crown and Cutlass in a PhD program, but I think I’ll still be able to contribute and take my classes (and continue working part-time). I’m really excited about this program and going back to school. It even looks like there may be some classes that may help me figure out how in the world you actually write a game!

In other news, Collin and I talked today about what we still need to do before we can start rewriting Crown and Cutlass game code using our new Protocce engine. That’s a significant milestone! I mean, it’s not as big a deal as actually using Protocce in Crown and Cutlass, but at least we are talking realistically about it. We are getting close to having something workable, and we are both really excited about the foundation Protocce will provide. If you are interested, you can take a look for yourself at our (slightly outdated at the moment) Protocce doxygen docs or at the Protocce SVN repository directly. If you really want to keep an eye on what we are doing, you could even subscribe to our SVN commit mailing list. I’ll try to boil some of that down into some specific things we are excited about in a future post when I have a little more time…

More on MySQL AB and the GPL

May 30th, 2007

The other day I linked to this post about MySQL AB and the GPL in a Slashdot comment. As I note in that comment, it appears that the link to the MySQL protocol description has changed.

Anyway, based on some of the responses in that Slashdot thread and in our blog here, I thought I should clarify what I’m trying to say. I have no problem with MySQL using the GPL to license their software. That’s their choice and if I choose to use their product I can’t really complain much about it. I may still prefer to use a database with a less restrictive license since several exist, but I have no issues with the idea of MySQL using the GPL for their database product.

I do have issues with how MySQL AB uses the GPL though. Here is my understanding of what they are doing: They have a document that describes the protocol necessary for a database access library to communicate with the MySQL database. Because the MySQL system is released under the GPL, they claim that terms of the GPL also apply to the protocol itself. As a result, they claim that if you use the document which describes the protocol to implement a library or system which communicates with a MySQL server (or emulates one, etc.), you must abide by the GPL. As far as I can tell, they are attempting to apply copyright (the GPL) to an idea (the description of the MySQL protocol in that document). Again, I’m not a lawyer, but it is my understanding that copyright applies only to manifestations of ideas, not the ideas themselves. That is, they can copyright the code which implements a protocol (which they have, it’s the MySQL database) and they can copyright the text of the document that describes the protocol, but they can’t copyright the protocol itself. If they want to protect the idea itself, they need to patent it. I believe that people who have read this generally agree with me that this is not a valid use of copyright. Am I wrong? If so, how is this valid?

I think we all can agree that this is at the very least questionable use of copyright. I am not sure why they would try to play these games other than to attempt to eliminate less restrictive 3rd party client libraries. I would think that as the company that produces MySQL itself, they should be able to legitimately keep their official client libraries ahead of any 3rd party competitors in terms of performance, reliability, and/or just their ability to offer support. Instead, they have resorted to what looks like misusing copyright. I’ll say this again: why bother with these stupid games when there are better database available like Firebird and PostgreSQL?

My other issue with MySQL AB is just that it appears they do a poor job of educating their users about what their license really means. I guess when they don’t seem to understand what their own license covers, I shouldn’t expect them to be able to explain it to their users. It seems like they are just being intentionally unclear and misleading, rather than being straightforward and honest about the licensing requirements. I admit that this is very subjective. Perhaps the people who commented here defending them disagree. I mean, MySQL AB is free to offer poor and confusing support if they want to. Thankfully, not only am I free to not use their product, but I can also attempt to explain my reasons to others (by whining about it on my blog)!

Event System Changes

May 23rd, 2007

I guess it’s a bad sign when you start rewriting the rewrite, but I’m working on some event system changes now that should make the system much more flexible. In our current system, you register a shared_ptr to an “EventListener” to receive events of a particular type. That is ok, but it is a little ugly for a few reasons. Often a class wants to register itself, and getting a shared_ptr from *this is not pleasant. I ended up just having classes that want to register themselves have a separate listener class that they create so that I can easily get a shared_ptr to use (e.g. the IThreadedSubSystem and ThreadedSubSystemListener). That is pretty ugly. It can also be constraining because you either have to code up separate listeners for every event types you want, or do some kind of “if .. else if .. else if ..” to figure out what to do with the event that the listener receives. That is pretty ugly too!

I decided that it would make a lot more sense to just register a boost::function that takes a shared_ptr to the event. That way, you could bind a free standing function, or a member function, or whatever to receive events. Objects could register themselves to receive events without jumping through hoops. However, boost::function objects are not comparable to other boost::function objects, which makes it hard to store them in map.

I have seen boost::signals before. We actually talked about just pitching our EventSystem altogether and just giving each event type a static signal. Then if you care about that event type, you just connect to that signal. That could be nice, but I think having the event system take care of that is a little more clear and makes event queuing a little easier.

Anyway, what I am working on now is actually using boost::signals for the actual event dispatch, but still within the framework of our event system. That allows us to still use boost::functions (which is what drives the signal library), but it has a workaround for the lack of comparisons. When you register an event handler function with the EventSystem, it looks up and if necessary creates the signal. The event handler is connected to that signal and the EventSystem returns a signal “connection”. The user can then disconnect their handler by calling disconnect() on that “connection” object. That could be pretty unwieldy if you wanted to register for 10 event types and keep track of 10 different connections. As a result, I am also creating another class to manage those connections automatically. You just create an instance of this class and then call the “AddHandler” method with the event type and handler object. This manager class stores the event connection and takes care of disconnecting it when it is destroyed. Alternatively, you can call “RemoveHandler” and it will disconnect the connection. I’m not sure what I’m going to call it though. “EventHandlerManager”, “EventConnectionManager”? I know “Manager” isn’t very descriptive, but I can’t come up with anything better. It automates the management of event connections.

That allows us to still have the flexibility of using boost::functions, the actual event dispatch is handled by boost::signals, and as long as you use one of these connection manager objects (and dispose of it properly) the interface should be pretty clean for the user.

Let me know if you have comments, questions, or suggestions for that class name.

Edit 4/29/2009: We finished these changes a long time ago, so I’m removing old dead links.  We went with the name “EventConnectionManager” for the object that automatically hold event handler connections.  Since this post we have also abstracted the event hander registration and dispatching into a new class “EventDispatcher“.  The existing event system inherits from that, and the threaded subsystem class has a dispatcher as well.  The reason for that is so we can ensure thread-safe messages going to the new thread.  A threaded subsystem (e.g. “SoundSubSystem“) has all of it’s events from the main event system come through a single handler which takes care of the thread safety.  Then the message is placed in a queue for the thread to pick up and handle using it’s thead-specific dispatcher.  It’s a little complicated but made the threading for the sound system pretty easy.

AMD open-sourcing graphics drivers?

May 15th, 2007

I saw this story on /. (which links to this blog post) the other day. I’m sure nothing I say will add anything that hasn’t already been said, but I am pretty excited about the possibility of real open-source graphics drivers (for cards that do a little more than the onboard video). Of course, statements by a VP of marketing that are quoted on a clearly non-official blog post are a long way from actually delivering anything at all, so I won’t be holding my breath. I am happy they are talking about it though.

In the past, I have been a pretty dedicated NVIDIA user, I don’t think I have bought a non-NVIDIA card since the GeForce series was introduced in 1999. Their linux drivers are closed-source binary blobs and make life harder than it should be for linux distros. Once you go through the pain to get them set up, they work well and give good performance. Ultimately, as a Linux game developer, that’s what I am looking for. I’m also excited about Noveau and I’m looking forward to trying their driver when it is farther along. However, if AMD provided decent open-source drivers for the ATI cards (or if Intel gets a decent graphics chipset going), I would happily drop NVIDIA without a second thought.

Excited about CDT

May 14th, 2007

So I read this blog post by Doug Schaefer, the CDT project lead. I’ve been reading his blog lately. It’s interesting. Based on what I’ve found in the milestone builds of the CDT (I’m currently using CDT 4.0 M7/RC1) and what I’ve read on his blog, I’m really excited about what they are doing. Performance is way ahead of the released version, and code navigation is getting much easier. For example, F3 to “Open Declaration” works pretty well, although it seems to only work in headers… I haven’t actually tried debugging yet. I’ve been scarred by my previous experiences, but maybe I’ll try that out now. Anyway, if you are a C++ developer in Linux, I’d recommend you check out a milestone build of Eclipse and the CDT.

PS Is it just “CDT” or is it “the CDT”? Anyone know?

Games as art?

April 12th, 2007

Collin and I have been reading and talking about the several stories lately discussing whether games are art. As a game player and someone who spends a good chunk of my free time making a game, I find that discussion very interesting.

I think I can sum up my thoughts pretty quickly. I think Collin agrees with this, but I’ll let him chime in if he wants. I see two issues: 1) Can games be art? and 2) Are the games we see on the market now art?

#1 seems easy to answer. Sure, games can be art. I guess Ebert may disagree with me, but whatever. I’m not sure how to really define “art” but Wikipedia says art is creating something “with the intention to create an experience for others.” I’d say that is very much what creating a game is all about.

#2 is trickier. I would say that most games we have now share about the same artistic value as the movie Independence Day. Maybe it was fun and maybe it was worth seeing, but I’m certainly not going to watch it again. It didn’t change my world or shed any new perspective on life or whatever. Playing the latest FPS might be fun and I might recommend the game to a friend, but I usually play those once. They don’t make me think, they don’t change my view of the world. However, as many people in the articles linked to above point out, it’s silly to judge the potential of the medium based on a few examples. One of the links says we shouldn’t judge movies as art based on “Wild Hogs”…

I have seen little glimmers of “art” in games though. For example, I remember just being speechless the first time I saw the D-Day level of the first Medal of Honor game. While now the graphics look a little dated, it still has the same feel as the D-Day scene in Saving Private Ryan but it was interactive. Perhaps reading about it in books and seeing a very realistic depiction in Saving Private Ryan helped make it what it was, but that was definitely an impressive level.

I am also almost ashamed to say that I found the ending to Unreal 2 to be very impressive. I’m not sure if I should just say what happens or not. It was really just another shooter. The gameplay wasn’t great. I seem to remember the graphics were impressive at times, but at least on my machine it didn’t run well at all. However, of all the games I’ve ever played, that might be the only one that I actually remember the ending. As I recall, it actually made me feel something for the characters. Of course, that was like 4 years ago and I haven’t played it since.

Several of the stories above talk about games as a medium being in their infancy, and they compare games now to early movies. I find that really exciting. Imagine a game with the power and capability of a good movie. Of course, just there is still a place for a good action movie or funny comedy, I don’t expect “just for fun” games will ever go away. However, I am excited to see how game evolve. Even though we are by no means doing anything groundbreaking in Crown and Cutlass, I’m excited to be involved in this process in some way.

On a side note, the Gamasutra article linked to from that /. story includes a quote by someone from Silicon Knights about what they call “Engagement Theory”. I was interested in hearing more about that idea, so I tried to google it. I finally found them talking about it a little more in this blog post. First off, I’m not too impressed by the Silicon Knights “guiding principle”. I mean, is that really a breakthrough? It seems obvious that the better all the parts of a game are, the better overall experience the player will have. That is to say, even though a player can have a good experience with poor sound, all else being equal if the game had better sound it would be a better game. Brilliant! Beyond just the cheesiness of expressing that in a formula, it seems like that does nothing to address things like what “better” gameplay or graphics means. Does Wind Waker or Gears of War have “better” graphics? I’m not sure, they are going for different ideas.

Anyway, beyond frustration with their “theory”, I found about a million more links to the already established (but totally unrelated) “Engagement Theory”. I find it frustrating when people hijack phrases that are already in use. I try hard not to do that, so if you notice we are using a phrase that is already in use somewhere else please let me know…