On plGLPipeline and Plasma for Linux…

A new beta driver from nVidia on Thursday night spurred me to revisit the plGLPipeline stuff I’ve been poking at over the past few years. I had an unexpectedly productive few days and got a lot of texturing-related stuff done. I posted the following image on Twitter yesterday to show-off how things were looking:

Relto, rendered on Linux with plGLPipeline

It’s pretty cool, but a single screenshot easily gives a false impression of progress. Some people were asking if I needed anyone to help test it. The reality is that this screenshot is all that plGLClient is capable of right now. The OpenGL work is solely the rendering pipeline and while (very slow) progress is happening there, the rest of the pieces needed for an actual functioning client are still quite far from being usable on Mac or Linux.

plGLClient currently has a hardcoded camera position. If I wanted to take a screenshot of a different part of Relto (or a different Age), I need to edit the code and build a new client. The reason for this is that the camera system depends on a working keyboard/mouse input system, and the current plInputCore system is Windows-only code.

Likewise, the Age name is hardcoded. I’d been testing with the Guild of Writers Pub for quite a while, and after it seemed to be working I decided on a whim to try Relto. Using the real plAgeLoader requires plNetClient and pfPatcher code, and those require the actual networking system code which is a tangled mess of several layers that involve of Windows-specific code.

To be entirely honest, to even get plGLClient to work at all involved commenting out sizable portions of other code. Did you know that texture layers depend on the network system (via plLayerSDLAnimation, which requires plSDL, which requires plNetClient & plNetMessage, which require plNetClientComm, which require …)?

One of the next things on the to-do list for plGLPipeline is getting lighting hooked up. You can see in the screenshot that some objects are rendered black because of the lack of run-time lighting. Unfortunately plGLight also has dependencies on other parts of the system that don’t compile on Linux. For now I’ll probably end up commenting those out for the sake of progress, but I can’t keep avoiding these issues forever.

 

I’m going to be honest, I don’t know if this project will ever get to a point where the game is playable on Linux natively. It’s taken me over 2 years to get to what you see above, which probably totals less than 80 hours of actual work. There’s so much to do in terms of replacing non-portable code and cleaning up the tangled subsystems, and it’s very overwhelming and makes it hard to work on it. It’s depressing to think that your next step might involve replacing 1/4 of the entire codebase. It feels a bit like punching your way through a rock wall, you make minimal progress at personal cost, but given infinite time you would eventually accomplish it. But we don’t have infinite time, and we don’t have resources.

There is nobody left for whom Plasma is their main focus. There are 3 or 4 of us who make casual contributions, but general interest in Uru has dropped off so there are no new contributors coming along. Between full-time jobs and other commitments and burnout, the reality is that most of us have very limited energy to devote to a project that feels like a black hole sometimes.

Advertisements

CyanChat

(originally posted at http://mystonline.com/forums/viewtopic.php?p=402386#p402386)

Oh CyanChat… where do I even begin?

I’d been vaguely aware of CyanChat since first finding Myst/Riven stuff online, back in 2000 or so (anyone else remember Leeloo’s Riven Hints site?), but the first time I actually joined CC was in 2006.
August 2nd, 2006 exactly. The day the first beta invites for MOUL went out. Gadren had posted on UruObsession that there was some sort of party going on in CyanChat, and Cyan’s intern RivenChan was giving everyone gold (guest) status.

So I went to see what was going on, and just kinda never left. There were a bunch of prominent community members in there; Tweek, Alahmnat, RIUM+, chucker, Aloys, and the occasional visit from RivenChan or greydragon.

I should mention that I was 15 or so at this point. I would go to the library at school during lunch and log in to the Java chat, reading chatlogs when I got home, and staying up far too late chatting about pretty much any topic imaginable with the people in CC.

As a programmer, one of the first things I did when I saw the CyanChat protocol spec was to start writing my own CC client. In C# first, then a server in C#, then a rewrite of the C# client, then some hacking on Kirsle’s Perl client, then one in Python, then one in C# to try out the new WPF technology, then a Pidgin plugin in C, then a few more C attempts to make a CC API library, and some C++ stuff, and bugfixes to a Ruby client… I wrote a LOT of CC clients, and it was my defacto “Learn this new language” project :P

By mid-2007, I’d become friends with some of the CC regulars. TW, vaaht, Nisan, M’buhir/Ku’mah, wandering_nomad, Edrick, theclam, Ki’Sak, Novah, Zadok, amonre… It was great with the ongoing Uru Live storylines too, because people would come in to CC from work/school to hear about what was going on in the cavern.

We started doing late night Skype calls at some point over the summer. Looking back, despite all the high-school angst and teenage insecurities and mild depression, that summer is one of my happiest memories. It always feels like the “golden days” of growing up.
It was some sort of crazy and wonderful happenstance that we all ended up in CC that summer and throughout much of the next year.

At some point, I decided to write a bot to transmit between CC and an IRC channel. Originally intended to temporarily work around an IP ban for mistyping something and triggering the language filter, the idea proved popular enough that several people switched to IRC full-time. I feel like that contributed significantly to the slow decline of CC, and I continually blame myself for that :(

As interest in Uru waned after the shutdown, and people got busy with jobs and life after school, the CC population dwindled. Ultimately it came down to only 4 of us and the IRC bot, and even on the IRC side nobody was really talking. After a year of that, I closed my CC client.
CyanChat and the people there were an incredible part of my life for those few years. It really felt like a family moreso than a chat room, and it was really hard to finally accept that most of you were gone and leaving a chat window open wasn’t going to make you come back.
I owe so much of my musical tastes to wandering_nomad and his CyanChatRadio shoutcast streaming.
I never would have completed NaNoWriMo in 2008 if vaaht hadn’t been reading my work and making suggestions. vaaht was a huge part of helping me to accept certain aspects of myself during those years.
I learned so much from Nadnerb, and especially liked our technical discussions and almost-telepathy.
Now everyone is off doing their own stuff, living their lives. I wish I’d made more of an effort to stay in contact outside of CyanChat, but that regret applies to almost everyone I’ve ever known.
There’s talk of shutting down CC, after almost 17 years of being the “official” chat room of the Myst community. I hate to see it go, knowing how much it meant to people and the friendships and relationships that grew as a result. But if nobody is connected, and nobody is talking, is it really worth keeping running?

Empty chairs at empty tables, where my friends will sit no more.

From My Hands

(Cross-posted from GoW Forums)

I… haven’t touched Blender in 2 months.
I haven’t done any meaningful work in Blender is far longer.
It’s sad for me to say that, because I have things that I want to do and ideas that I want to develop.

We’re losing people. I’ve seen more people lose interest over the past 6 months than I have in any other period of Uru. We’ve been given hopes and promises, and now we’re all waiting for something that might never come. Times change and people get busier, and I feel that need to seriously ask myself why I’m still here.

It’s not because I enjoy it: Uru is depressing these days, and I don’t want to build Ages that nobody will ever explore;
it’s not for the community: I’ve seen almost all the good people lose interest, leaving only the idiots on MOUL worshiping their Holy Blue Lords of Cyan;
it’s not for the potential: time killed whatever potential was left in Uru.
It’s not even for Plasma anymore, the more I see of it, the more I realise how much it could be improved and yet there’s no way to do that.

I thought things would be more open when MOULa started, but instead they’re more closed than before. Any incentive to work on stuff is crushed by Cyan’s continued silence, and the overwhelmingly vocal disapproval of many community members. We all have bad days where we ask “Why do I put up with this?”; but when you’re seriously asking yourself that question every time you think of Uru, maybe it’s time to move on. I don’t want to be part of this anymore.

Uru’s dead, more dead now than before MOULa started. I keep wanting things to change, but I can’t keep pretending that they ever will. Cyan can’t make Uru work. They’ve lost direction and fragmented so much that the only thing keeping them in business is rehashing existing games on new platforms. The “Golden era” ended with Prologue, and came to a crashing halt after Myst V. MOUL never managed to bring any of that back.

The future is uncertain. I can say for sure that prpl-uruki will probably never be finished, and libHSPlasma development has mostly stopped as all the developers lost interest. PyPRP2 is up in the air, very dependent on what happens with Blender and whether there’s still enough interest to keep developing it. Without maintenance and support for the current tools, Age builders will lose interest and drift away, and with them goes any lingering hope for Uru’s potential.

For now, we keep waiting… but maybe someday you have to accept that the story’s over.

Pidgin 2.7.0 with prpl-uruki

Apologies, I’m not going to do much more than list a series of command here… These are intended for Ubuntu (Lucid Lynx), and will compile prpl-uruki so that you can view who is online. No chat at this point.

$ sudo apt-get install libssl-dev libjpeg62-dev zlib1g-dev
$ hg clone http://dev.zrax.net/hg/libhsplasma
$ cd ./libhsplasma
$ ./configure --prefix=/usr --no-python --with-pkgconfig
$ make
$ sudo make install-dev
$ cd ..
$ hg clone http://dev.zrax.net/hg/prpl-uruki
$ cd ./prpl-uruki
$ ./autogen.sh
$ ./configure --prefix=/usr
$ make
$ sudo make install

If you reload Pidgin, you should have the option to add an “Uru KI Network” account. When(/if) it connects, your buddy list will be displayed.

Known Issues:

  • Chatting is not implemented. Chatting does not work. There is no chat. There is no chat.
  • If the connection fails, Pidgin will hang forever waiting on a thread condition. If I see a lot of complaints, this might be rewritten.
  • You can add buddies, but cannot remove them (through Pidgin)
  • Did I mention that chatting does not work? Do not give me bug reports about it :P

Bugs can be submitted at prpl-uruki on Google Code.

prpl-uruki

The Pidgin KI Chat plugin is beginning to take shape. With Zrax’s libHSPlasma(Net) library as a backend, I’ve started committing code to Google Code.

At the moment it does nothing useful (I haven’t even tried compiling), but I’m hoping to have buddies showing up in the buddylist before the end of the month. Based on school exams and my upcoming Co-Op job, I can’t make any guarantees though.

For those who like to poke around and follow development:
prpl-uruki on Google Code

MO:ULagain on wine

Okay, short note before I start this. It has occurred to me that the only time I blog now is when I’m frustrated and can’t fit it into 140 characters, or when people ask me the same question repeatedly to the point that it’s easier to blog about it. This post is of the second kind. :P

So Uru is back (again), and it’s Windows only. Mac people can possibly use the old GameTap Cider package and replace the application contents, but Linux people need to use wine. I’ve had good luck with wine and Uru, and after a plethora of IMs, PMs, Tweets, and forum posts, I’m writing a tutorial for getting Uru working with wine.
Update: This is compatible with wine 1.1.40!
Update: This is compatible with wine 1.1.41, but requires some minor changes to the patches.
Update: This is (mostly) compatible with wine 1.1.42. Download the latest patches from wine bug 421. After ./configure, you only need to run make (not make depend).
Update: This is compatible with wine 1.1.44! Download the latest patches from wine bug 421.
Update: This should work with wine 1.2! I’ve also been informed that it works with wine 1.3.1 (and probably newer ones like 1.3.4).

Update: With wine 1.3.16 you NO LONGER NEED THE DIB ENGINE PATCHES!

Are you looking to test a native Linux/Mac client for MOUL?
Check out branan’s awesome PlasmaClient project. Download the alpha release and watch for progress!

1. Set Up & Basics

At the time of writing, the latest version of wine is 1.1.39. I will most likely be compiling each new version of wine when it is released, and will post updates if anything changes or breaks.
I am running on Ubuntu Karmic, so instructions may be different for different distros.
wine has a long outstanding bug involving its (lack of) DIB engine. For Uru, this is important for the cursor. If you use wine without patches, your cursor will be a white block. The game is still entirely playable (as far as I can tell), but if you want the cursor you will need to compile wine yourself.

1.1 Getting wine (no patches)

On Ubuntu Karmic, open the Software Sources menu by going to System->Administration->Software Sources.
Then select the Third Party Software tab and click Add.
In the APT Line box, enter the following:
ppa:ubuntu-wine/ppa

Open Synaptic Package Manager by going to System->Administration->Synaptic Package Manager.
Click the Reload button.
Find the wine package in the package list and mark it for installation.
Click the Apply button to install the packages.

Continue to step 2.
Source – http://www.winehq.org/download/deb

1.2 Patch and compile wine

THIS IS NO LONGER NECESSARY IF YOU ARE RUNNING WINE 1.3.16 OR HIGHER

On Ubuntu Karmic, open the Software Sources menu by going to System->Administration->Software Sources.
Then select the Third Party Software tab and click Add.
In the APT Line box, enter the following:
ppa:ubuntu-wine/ppa

Update your packages:
sudo apt-get update
Install the necessary library and development packages:
sudo apt-get build-dep wine1.2

Download the wine source code from SourceForge.
Extract the wine source code to a directory (I’ll assume ~/wine-1.1.39).
Download the DIB engine patches from wine bug #421.
Extract the DIB patches to a subdirectory (such as ~/wine-1.1.39/dibeng-max).

Go through the series file, and apply each patch to wine as follows:
~/wine-1.1.39$ patch -p1 < ./dibeng-max/patch_file.patch
Rebuild the configure script by running:
~/wine-1.1.39$ autoconf
Configure wine:
~/wine-1.1.39$ ./configure
Compile wine: (You only need to use make depend if you are compiling an old version of wine < 1.1.42)
~/wine-1.1.39$ make depend && make
Install wine:
~/wine-1.1.39$ sudo make install

2. Make OpenAL work with pulseaudio

In theory, wine is capable of using the system’s OpenAL library to provide native sound output. In practice, the sound cuts out after the intro video for me.
This is also complicated by the fact that pulseaudio and wine don’t always play nicely either. If this is an issue, I wish you luck. :P

Update: To make the sounds work fully in wine, please see http://forum.guildofwriters.com/viewtopic.php?f=86&t=4203.

Open the wine configuration tool by going to Application->Wine->Configure Wine.
Go to the Libraries tab.
In the New override for library type the following:
openal32
Click the Add button.

You should see in the Existing Overrides box an entry that says
openal32 (native, builtin)

Click Apply to save your changes.

3. Set the wine version

Uru uses PhysX as a physics backend. PhysX requires wine to be in Windows XP mode.

Open the wine configuration tool (it should still be open).
Go to the Applications tab.
Set the Windows Version dropdown to Windows XP.
Click OK to save your changes.

4. Get MO:ULagain

Ideally get a full version from one of the mirrors listed at MystOnline.com.

5. Run Uru

There are two executable files in the main Uru directory: UruLauncher.exe and UruExplorer.exe.
UruLauncher is the patcher program, UruExplorer is the actual game.

Rather than running the patcher directly, I’ve had better luck if I run the game (which will detect and then launch the patcher itself).

If you’re using an unpatched wine, you should be able to simply double click UruExplorer.exe to launch the game.

If you patched wine, I suggest creating a quicklaunch file in your Uru directory to set up the DIB engine and run the game.
Open a terminal and navigate to your Uru directory.
$ echo "WINEDIB=ON wine UruExplorer.exe" > runuru
$ sudo chmod a+x ./runuru
$ ./runuru

What we do…

Brought up by Tai’lahr on the MOUL forums, who had a very good point that not many people actually know what we mean when we say that we “hack” Uru/Plasma. Hopefully an explanation will help to dispel some of the fear, doubt, and uncertainty that is being spread.

Prologue ended in February 2004. With the closing of areas such as the hoods and Ae’gura (which weren’t available in the offline game), some people began to look for ways to access those areas in the offline Ages Beyond Myst. All of Uru’s configuration and game files were encrypted, and started jokingly with the phrase “whatdoyousee”.
The night that Prologue shut down, a user by the name of Anonymous54321 posted a function on the Clockwork Orange BBS (COBBS) board that decrypted these files. The decryption was actually quite easy to figure out: we had the files, and we had UruExplorer.exe. UruExplorer has to decrypt those files when it reads them, so by reading the machine code inside UruExplorer.exe, we could figure out what it did to decrypt the files.

Once the xTEA (whatdoyousee) encryption was broken (which means, we were able to decrypt and re-encrypt the files), we had access to the SDL (State Description Language) system. SDL files store the default state of objects in an Age, such as open doors, barriers, and Journey Cloth visibility. Anything that could be turned on and off in the online Vault could be turned on and off using the SDL files for offline Uru.

After a bit of playing with SDL, some people wanted to look more closely at the game logic, the Python scripts that powered interaction. The process was an interesting one. Again using UruExplorer.exe, people traced to where it loaded the Python.pak file. Then, these people wrote documentation for it, byte by byte, as to what the game loaded.
Despite the fall of COBBS in April 2005, I was able to dig up a copy of the original Python hacking thread. This thread is where Python.pak was first looked at, decrypted, unpacked, and edited; and can be read here.

By the time To D’ni was released for free, COBBS members had already written tools to decrypt and edit the SDL and Python files (UruFun and UruPython were both written by Anonymous54321; UruTweak was written by Marrack; and Kirk had a Java program called WYDS’d).
I didn’t join the scene until shortly before Path of the Shell was released, but around the time of To D’ni came the invention of flymode. Flymode brought a lot more attention to COBBS, some of it positive, and some of it negative. It was really just some additions to Uru’s xKI Python file, which was then recompyled and packed back into Python.pak.

At this time, some people were also wondering if it would be possible to add our own meshes and textures to Uru. They began investigating the PRP files, which held the actual game data. This was a very slow process: the PRP files had a packing format, and contained objects. There were over 300 types of objects, and each object had its own format.

Path of the Shell complicated things a bit, since it added even more objects to the PRP files, and also changed the KI system (which was used as the basis for flymode). But the actual encryption didn’t change, so after examining the changes, we were able to continue making progress.

Aside project that was going on at this time was Alcugs, an open-source server for Uru that was written by Almlys,starting the day that Prologue shut down. When Path of the Shell was released, Alcugs made efforts to support it as well as the original online content. Untìl Uru was released in August 2004, around the same time that Alcugs was made publicly available. Most people initially ran Untìl Uru servers, but those wanting Path of the Shell content ran Alcugs.

During the online stuff, work continued on the PRP files, with IngisKahn writing and releasing PRPExplorer. The first view versions just listed the objects, but support quickly grew to viewing, exporting, and replacing textures. A new era of Uru hacking had been born: Texture hacks.
Unfortunately, the online aspect meant that PRP hacks had to be done offline, so that there were no Vault issues or file version mismatches. But as support for PRP hacks grew (and editors and knowledge about more PRP objects), more people started adding things to their offline games, and some servers even offered extra clothing options to players.

In April 2005, ycikamookow and IngisKahn released a video of an Age that they had built using ycikamookow’s custom 3DS Max exporter. Shortly after, the COBBS server was cracked, and all posts were lost. Luckily most of the documentation from the H’uru wiki was retrieved and reposted.

The Myst V Mac Demo came out in August 2005, and within a few days we had used the MachO executable to break the new AES encryption and begin to figure out the changes to the PRP file format. During this time, we had a brief discussion with Colin Bonstead at Cyan, who wished us luck with the Uru hacking, and said he wished that Cyan could give us some help in our quest to build custom content.
Almlys had also been quietly working on a new project too. A Blender exporter plugin for Uru, which was released in late October 2005. At first, it could create basic geometry. Within weeks support had been added for textures and lights. All of the Blender plugin features depended on the documentation of the PRP object formats.

Over time, tools changed… The PRP Blender plugin became known as PyPRP, which forked in April 2008 to become GoW PyPRP.
PRPExplorer fell mostly to the wayside as Zrax’s libPlasma work allowed us to create tools like PageEditor and PRPShop.
During D’mala, Cyan accepted some small patches from the community and released them as Untìl Uru updates; but many people saw it as “too little, too late”, as some servers had been running with modified content and bugfixes for years.
Untìl Uru‘s global authentication server was shut down, leaving shard owners to either implement their own, or shut down their shards.
Myst Online: Uru Live was released, giving us yet another slightly different PRP packing format, more PRP objects, new encryption methods, but also a better understanding of some PRP objects.
Myst Online: Uru Live spawned the Guilds, and then shut down. A year and a half later we have no further progress on an online Uru, but the tools and documentation for creating custom content have improved dramatically.

So when we say “hacking Uru”, what does it mean?

It means that we are reading the assembly machine code from UruExplorer.exe to find out how it works, and how it reads the PRP objects. Then we are documenting what it reads, and finding out what those values represent.

It means that we are taking existing content and changing it to add new features (such as flymode).

It means that we are using the knowledge we have extracted as a way to write tools that allow us to edit and create content for Plasma.

For some of us, this quest stopped being about Uru and its community a long time ago, the quest now is to understand Plasma and to create new places to explore, ideally in an online environment. To us, an open-source Plasma would allow us to take what we know about the engine and use to it add new functionality and enhance old functionality (when you read the machine code, you tend to see a lot of places where things could be done better, or where bugs could be fixed).
However, without the source code we cannot add new functionality to the actual engine, we can only document its current functionality and use that to create tools.