DKC Atlas Forum

[Super Luigi!]

You are a true pro at what you do, Simion32. Such dedication to this project, it's so...beautiful, so incredible. I would bow down before you, but that would mean hitting my keyboard with my face, and grant me a couple of looks my way.

[Simion32]

Thank you.

Now there are only 3 bugs left to find, but I feel that these last 3 will take effort to squash. I've been gradually whittling away at them over the last week.

[Markster]

I wonder when DELTA will have Donkey Kong's sprite with the physics.

[Simion32]

The very next thing after the physics bugs are gone is to introduce more content into the engine, which likely means that you will need a ROM to extract from by the time that demo gets here.

I'm not sure about what will happen with the objects in levels, just yet. Every single one of them needs coded, and DK isn't really complete either.

The DLL system needs to be "completed" before I can do all of the objects, though.

[Qyzbud]

Haha, yes. What Super Luigi said.

Spoiler!

You are a true pro at what you do, Simion32. Such dedication to this project, it's so...beautiful, so incredible. I would bow down before you, but that would mean hitting my keyboard with my face, and grant me a couple of looks my way.

Simion, you have come leaps and bounds with this project - the physics engine is spot on, and all-round the quality of your work is impeccable. I'm honoured that you've picked our humble community to be host to such a ground-breaking project.

Best of luck with the bug-squashing, and hopefully our favourite little red-orange dot with a yellow trail (or even the likeness of DK himself!) will be roll-jumping before too long. I can't wait to see what's next.

[Simion32]

Thank you Qyzbud. I know everyone is looking forward to this being completed.

This was a very big step (nigh, the most difficult section of the program) and we are extremely close to actually having something for-real-playable (maybe even editable later).

Did any of you bother to try this in something other than 60Hz? You would probably have noticed it's still smooth.

Oh, but the frame-by-frame mode has buggy display (stuttering during pause) on non-60Hz because of a small oversight. Sorry about that...

[HellFire]

I just tested the last build, impressive stuff Simion! The physics are like in the original, thats great! You're for sure a top notch programmer! I'm looking forward for the day when everyone will create their own levels and share with everyone! I'm not the most inspired person but I will for sure have an small project.

[Morzeka]

Whoops. Posted in the wrong thread. I'm still recovering from my blindness.

Nonetheless, my hive in its entirety is buzzing over your release of Delta. I cannot wait. I would agree with SuperLuigi, but bowing just does not seem to suit a king, do you not agree? Buzz, buzz.

Sincerely,
Morzeka.

[Simion32]

Progress Update: CGfxManager Overhaul + Layering Code Underway

I had enough of the physics a day or two after that release, and I have been working on the layering and graphics code.

Image De-Duplication
I just now finished a cool new sorting method that allows me to prevent exactly-the-same images from being loaded twice into RAM. This treats the entire block of bitmap data as an extremely large number, and it inserts the image into an array based on the image data (the actual underlying mechanism is much more complex than a normal binary search, though!). The result is that all the images come into the CGfxManager sorted by their bytes, and I can identify during the loading process that an image is an exact duplicate, in which case it's discarded and the already loaded one is used instead.

I used to have it using a checksum-based method, which is not 100% accurate and could have caused rare (but possible) image replacement glitches wherein 2 images get the same checksum and one of them would be ignored. However, I have now 100% prevented the possibility of any bugs with this sneaky RAM saving mechanism. To top all of that off, this new method is faster than the checksum method, because a lot of the work is combined into one operation and there is no "sort upon every insert" code anymore.

Layering Engine
The beginnings of the layering engine have taken root in DELTA's code. This is the monster that controls ALL of the rendering process, and it's going to be behind making sure that your DLL calls are processed and sprites get drawn to the screen. It's going to take a bit of time to develop this, since I must take into account all possible DLL functionality and what the users will be able to do graphics-wise.

For the possible layers, you can take a look at the DKCLB Layering Diagram (Development Topic) I posted in this subforum.

[Phyreburnz]

Okay, I'm really feeling like an idiot, here

Is there somewhere on the forum that explains how to use DELTA and what it actually does? I really am hoping that I'm not the only person that's extremely confused... or else I'll feel even more like an idiot...

I've been looking for a while through the forums (and the regular DKC-Atlas site), and I didn't know if I was overlooking something that explained exactly what DELTA is supposed to do and how to do it.

God... you don't know how stupid I feel asking this question... I sincerely hope that I'm just overlooking something.

I've opened it up and I just don't get it... I set controls and such... but I'm still so confused. Is the information just kind of scattered in this topic?

[Simion32]

DELTA is the game playing part of DKCLB. I have to make it first before I can start doing the LB editor!

DELTA is attempting to replicate the DKC engine, but for Intel PCs (DELTA is coded in C++).

The engine is in-progress, and you cannot edit levels yet because the editor itself doesn't exist.

Everything else should be in the readme.html that you can do in the engine so far. Remember, DELTA is still very early WIP material.

[Simion32]

Progress Update: CLbFileProvider and the LBIDs System

Yesterday I just finished a class that is one of the major workhorses of the engine - the file loader. It scans every single file under the "Resources\" folder, and grabs a record for each containing the File type, version, LBID, and file path.
Note For Techies: Reparse-point (hard link) folders will be fully ignored if found inside the "Resources\" folder.

LBID, you ask? Every file type in DKCLB has an LBID attached to it (even the DLL plugins must have this). The LBID is supposed to nearly guarantee that you have a unique identifier for your file, and this will allow DELTA to index and access thousands of resources of the same type at the same time, without much chance of an LBID collision.

LBIDs are 64bit and look something like $01234567FB3D8A26 where the first half is a time stamp and the second half is a randomly generated number. For original trilogy files, however, the time stamp is fully zero and the random number area instead becomes a non-random file index. LBIDs are only supposed to be unique within each type of file, not across all files.

An LBID does not change once your file is created on-disk, and remains the same from thereon. This allows you to edit the file and replace its contents without doing damage to the links between various types of files (levels use objects, world maps reference level files, etc...).

The LBID system along with the new CLbFileProvider make for some great working rules:
(1) Files must be placed within the "Resources\" directory for DELTA or LB to find files automatically.
(2) Files can be named anything (even misnamed files with no extension)!
DELTA and LB will still pick them up (since the LBIDs have to be scanned for each file).
(3) The names of sub-folders under "Resources\" do not matter and they can be anything you desire!

Also, as a semi-related note the animation and animation frame (KNI, KNF) formats are now one format; files of this kind may have either file extension based on the number of animation frames.
-----------------------

EDIT: After some perusal of GUID documentation, the current LBIDs may not be sufficient enough to be reasonably unique. However, I want to avoid using personal information such as the network card or computer ID, since that can be traced back to a particular user. I care about people's privacy.

[Simion32]

Progress Update (updated!): LBIDs Revision 3

The format has reached Revision #3 and is now 192 bits, to allow increased security with that last section.

192-Bit LBID Format
42 bits:Time Stamp Data: 128f * 60s * 60m * 24h * 31d * 12m * 1069y [from Jan 1st 1970]
8 bits:The number of times this filetype created an LBID on this PC. Wraps around to zero at 256.
142 bits:(PC_IDENTITY[48]+random[80]) -----AES-----> (scramble[128]) -----SHA256-----> Final 142 bits

The hash used is to be SHA-256, with enough bits cut off to make it 142 bits. This is essentially just doing a checksum, like CRC32 but bigger and badder.

Generating the PC_IDENTITY data:
If a MAC address is not available LB will go do a CRC64 Hash on some windows-related registry data in an attempt to maintain uniqueness, and use 46 bits of that in place of the last 46 of the MAC, and set the first two bits. If THAT isn't available then LB will set the first bit, clear the second, and resort to generating random garbage for the remaining 46.

The PC_IDENTITY data will be combined with some random data before being scambled, to prevent table-lookup-type data mining attacks.

It is then run through an AES cipher just to numerically scramble the data (this alone doesn't really secure anything), and then finally it is hashed which destroys the chances of anything being retrieved from the already scrambled garbage.

This 100% cannot be reversed into a MAC address and cannot be used to uniquely identify any computer. Once the final hashing takes place, the address data and its paired random garbage bytes are completely scrambled and destroyed.

Regarding original trilogy files:
The 8bit counter is followed by a reserved name string, and a 32bit resource ID.

[Simion32]

Progress Update: LBID Generator COMPLETED!

I have successfully coded and implemented the LBID generator function. Now, they can be used for all the LB file types!

Here's what an LBID looks like as raw binary:

A generated LBID: 0A3054692140B24106F355AC9CF0511BACE33A5247994ED1
Here's Another: 0A3055187580C73444D9F7A78F560E1FF7990D1FD0A7E6C1

The green is the time stamp, blue is the creation counter, and red is the PC-specific pseudo-randomly-generated garbage hash.
Where the colors mix means there's a few bits for each kind of data in that hexadecimal digit.

There's two LBIDs here, just to show that the red section always looks wildly different (yes, these are real IDs generated from my computer). As you can see, the number is almost completely made up of scrambled garbage.

There's no way you can pinpoint any specific computer with this stuff - that is, unless you're a psychic.

I hope to be able to print the LBIDs out like this:

[Dec 21st 2012, 11:11:11PM, $7F #AaBbCcDdEeFfGgHhIiJjKkLl-255]

In this way I could try to provide some sort of resource listing page that tells you all the files DELTA or DKCLB has recognized.

[Markster]

I got a question about DELTA, in 2014, Windows XP will be unsupported, in 2014, will DELTA have support for Windows XP?

[Simion32]

Yes, of course. It was designed for Windows XP from the ground up, and all the behaviors work correctly thru Windows 7, so there's no need to worry about some OS stuff borking the compatibility.

The only thing of note for newer windows OS's, is that you NEED the legacy DirectDraw driver. ZSNES needs it, too.

[Markster]

I am sticking with Windows 7 and Windows XP only, Windows 8 is a big piece of crap.

[Katastrophe Kong]

I am sticking with Windows 7 and Windows XP only, Windows 8 is a big piece of crap.

Windows 8? Watch'u talkin' 'bout Mawkstah?

[Simion32]

Obviously the new DRM-ified Window-Lockdown version of Windows mainly targeted at mobile devices. Windows 8 requires a secure boot that could provoke hardware vendors into preventing you from running any other OS on such a device. Desktops apparently (at least now) require a way to get rid of the secure boot, and thusly windows. But I say that any form of hardware DRM lock-in should never have been devised in the first place.

Spoiler!

And no, you cannot just overwrite the lockout code. These things work much like the Wii does - with Wii, the first section of memory is hard coded ROM, which loads the second section of memory, and validates that, and then this 2nd section loads the 3rd which finally goes on with the boot process. Any modification to the 2nd part of the boot-up procedure (or the 1st if you managed to replace the ROM chip) will make the cryptographic hash and/or decryption fail, thus blocking the Wii from booting and bricking it with a black screen.

All DRM and copyright is completely nonsensical because it tries to make information be treated like physical objects, which information definitely is not.

[Markster]

Sadly I have to use a old dinosaur computer with only 512MB of RAM and a low processor, but talk about a life saver, I ran DELTA in 640x480, and DELTA flew on a old low spec computer, it was average speed, thanks for having this feature Simion32.

Here is a screenshot.

0000000000.PNG (10.47 KiB) Viewed 214783 times

by the way, I can't wait for the next big update in DELTA Game Engine.

[Simion32]

That's why I have been optimizing DELTA to be able to run in small resolutions. For those old dinosaur computers you speak of...

It looks like the graphics bus on that computer is slow as hell. Seriously, it's taking like 80% of all the usage (this is the same DELTA main bottleneck, but amplified because of crappy integrated graphics (that or it's just plain sluggish)).

Yeah, at the very minimum you want a graphics card, even if the computer is still quite old. The MMX optimized drawing routines are helping it too.

[Markster]

So far, your engine is 100% like Donkey Kong Country, all though the old dino PC can't run it that good, it's still good.
----------
Also, when do you plan to add rolling physics, DK's sprite, etc.?

[Simion32]

You know, if the PC is VERY old you may even be able to coax DELTA into running in 320x240, that would be a real treat for an old clunker machine like that. I'll bet it would even break below 95% (usually 95% is where frame-by-frame stutter starts happening).

Also, DELTA doesn't yet have the ability to control the frame rate cap based on the CPU usage, which is why you were getting something between 30 and 60. If DELTA had this capability you would get a flat 30FPS and it would all be smooth (at 30FPS, of course - not all that bad considering the SNES shows graphics at a TV-interlaced 60 which translates to effectively 30FPS).
----------------------------

I've been trying to focus around the editing part of things now, because if I don't soon, it will become a roadblock. The file types are the key to making it all work, which is why I had done the recent LBID development (and now the NGUI upgrades).

[Markster]

I got some questions about the Nitro GUI update.
----------
1. How much faster, lighter, etc., will it be compared to the original?
2. Will the interface graphics improve?
3. How long will this take?

[Simion32]

The NitroGUI upgrades only apply in the realm of DKCRE and the DKCLB editor - DELTA will not see any improvements from this particular code.

The upgrades focus around making the editor programs fast and cannot be applied to DELTA at all (with DELTA, the whole screen is controlled by CVideoBox, and the entire game screen changes the whole time, which essentially makes all dirty rectangle methods completely useless).

I'd estimate the DKCRE will probably be on the order of about 3 times faster (give or take some %) once ALL of the adjustments are completed.

This could take any amount of time to do because I'm starting to be busy with university again (classes to start in around 2 weeks). However, it really isn't that complex or difficult to code. It's more of a technical improvement.

[Markster]

So far, DELTA Game Engine is epic, like I said, it is just like DKC's physics, the GUI is epic, everything about it is epic.
----------
And some questions about DELTA & DKCLB.
1. Will there be a test version of the level builder, one that edits tiles and uses similar code to DKCRE.
2. Say I got a 101% game on the SNES, can I import the save file to DELTA and continue my game on the DELTA version of DKC1?
3. How much percent finished in DELTA & DKCLB as of now. (Last time I remember, you said 2%, but that was a year or two ago.)

[Simion32]

Well, when I finally finish the NGUI/RE updates, I'll have an editor "shell" program that I can take the guts out of and reprogram it for LB functionality. The tile editing mode will need a few adjustments regarding variable tile size, but other than that it's pretty much ready to go already - so we will have tile editing right off the bat, as well as any other basic functionality.

Camera stuff will have to wait, as will the objects editing.

Camera is vastly different in LB compared to the SNES DKC way of doing things (it's even different than DKC3's updated camera system).

The objects editing requires that the full object DLL system be implemented because the DLLs handle the custom objects' edit dialogs and right click menus (etc...).

Doing that is going to take a lot of crazy C++ hacking, because by that time we are going to (it's required) need an NGUI dialog builder so that people can make the dialogs that edit the object properties.

Dialogs MUST be easy to design, but that means LB is going to need ubergeek-level code.
----------------------------------------------
No save file importing, sorry. It doesn't work like that. But you can cheat your save file to force it to be the way your SNES save was (manually).

As for percentage completed, I'm willing to bump that up to at 5% or more, since I've been making quite a bit of headway. But I'm unsure how to measure progress related to the actual game objects.

EDIT: It would be nice to compile some sort of task list that would provide some rough estimation of the completion status. But it'll still be hard to tell even if I do that.

[Markster]

Awesome, 5%+ done, it's about time it's changed from 2%, anyways, how much percent is the new Nitro GUI upgrade done so far?

[Simion32]

Probably 5%...

You see, I've been working on the RE side of things first because that provides the most significant speed gain. Then I have to round it out by making NGUI itself optimized. The only part of NGUI I've touched so far is the CPictureBox (dirty-rectangles drawing).

[Simion32]

NOTICE: NOPs

University Classes Have Begun *Today*
Expect Major Delays (again)

[Simion32]

Progress Update: So, I haven't posted much here in a long time. Well, let's just say I think I've found a way to make DELTA burn some serious rubber.

I'm going to attempt using Self-Modifying Code to create an image compositing algorithm that is able to take in several layers at the same time and spit out the result, which can then be used in another round (repeat as many times as needed).

My idea is that it can dynamically copy/arrange blending code snippets within a loop, and bulk-process the layers.

There are very few caveats -- the only times I need to have code copied around are (1) level boot-up, (2) a sprite changes layer priority, and (3) something changes its blending mode -- and all three of those cases shouldn't be happening all that often, even in a very hectic level.
-------------------------------------------------

The speedups: Well, there's an implied memory bandwidth savings, that works out to roughly 3x less pixel copying. Not only that, but since a single several-image pass is only one screen-sized loop, there's a ton of savings from not having to do a loop for each image involved. Overall, this could literally become a 6x (or more) speedup because of the heavy decrease in redundant looping/reading.
-------------------------------------------------

I still haven't completely worked out how sprites with flipping and rotation are going to be worked into this, but other than that this seems to be a solid way to speed up the graphics even more than I originally thought was possible. By running everything through the compositing engine, every image processed should only take a fraction of the regular time, due to the bulk-processing approach.

[Simion32]

Progress Update: Working on the music engine in DELTA!

I got tired of doing NGUI stuff so as a breakaway task I'm working on the music engine.

Here's what I'm planning on doing, and boy is it a doozy to get feature #2 right:

Music Features
--Play any format you want provided user has an appropriate compatible WinAMP plugin for the format.
--Ability to play several tracks simultaneously and modify each of their volumes individually, in-game, in real-time.

The current problem I'm having at the moment is trying to synchronize playing tracks so that I can accomplish seamless "themed tracks crossfading" like the kind of themed music you hear in Banjo-Kazooie. This is essentially the only major issue left to conquer.

I also want to mess with memory mapped files and see if I can hide the extra copies of DLLs that have to be thrown around, into RAM (you MUST copy the dlls in order to have multiple active inputs, and you have to do it with the outputs as well, or it crashes very badly).

Specifically the second feature may involve lots of synchronization voodoo that I have no idea about yet. It might only be feasible to load two or at most three tracks at a time, so I'll have to load files up as they are needed and synchronize the tracks somehow before a theme crossfade happens. Which obviously means this synchronization may have to happen pre-emptively in the background (outside of the level designer's control).


So far: I'm able to sync tracks but the algorithm variations I've tried have a very tough time getting a bead on the sweet spot, and it clips like hell until it does hit the sweet spot (both tracks being reasonably near the same millisecond in play time). Haven't had anything sync faster than a few seconds. I'd like it to be able to do this in a few frames time, or maybe even instantly - but since there's a certain factor of unpredictability, I'm not sure how fast i can make the algorithm converge on the sweet spot.

[Qyzbud]

These are both excellent ideas, and I think you're going about the implementation in a logical and inspired way. I hope this pans out well, even if it's just an 'option' for those with particularly fast PCs — or sound cards, if that has much to do with it...

[Simion32]

I think the only reason for a good sound card is that you need 2 simultaneously playing channels - sound cards should at the very least have 8.

As for the plugins, that's on the CPU. Which will depend on how fast the plugin is (I've read that 64th Note is very CPU intensive - but I can manage running 2 of them without much trouble, so I have no idea how far you can go).

The clipping won't be heard if the syncs are only allowed to happen while a track is 255 volume and the other is 0, and I modify the unheard track's position before Donkey Kong passes a crossfade trigger object. But I still need to do this very quickly, because I don't want there to be much of a "must play in full volume for X amount" limit.

[Qyzbud]

Ah, good to hear. I've been meaning to upgrade my PC's sound hardware, but I've yet to have a solid reason to do so. The slight boost I'd receive in audio quality would be nice, and having extra input/output options is always a plus, but overall it's just an unnecessary expense. Good to know my current hardware will be fine for DELTA's most demanding requirements. (yes, my CPU and GPU are of beastly power)

I'm more than a little excited by the prospect of having OC ReMixes, studio-quality official DKC/DKCR music, and our own compositions as background music. Daaaaamn.

[EvangeliKong]

That is very interesting!

[Simion32]

NOTICE: NOPs

University Classes Begin *Tomorrow*
Expect delays, as usual.

[Markster]

It will probably take several years for the DELTA project to be finished.
Well, gotta be patient.

[Simion32]

Well, hopefully this semester won't have such an insane amount of homework. If I can have less homework I might be able to slip in development time at the end of each school day and on my off days (I DO actually have off days during the week this time!). Not to mention the weekend.

[Qyzbud]

It will probably take several years for the DELTA project to be finished.

Well, weren't we expecting that, anyway? Good things take time.


A lighter uni workload sounds promising though, Simion — hopefully you'll retain enough energy and motivation without burning out... gotta remember to look after your health (physical and mental) as well, y'know!

[Markster]

Good luck Simion32, and I'm glad you got some weekdays and weekends off this time.

Also, DKCGM and Kremling Invasion Confirmed is working out good, I will probably release my DKCGM on my 16th birthday or later. (June 10th +)

[Morzeka]

Greetings. I haven't been around recently. You could call it a distant vacation from the land of the Kongs (or perhaps kremlins - it's debatable).
It's good to see that you've made some progress, Simion32, and I wish you great luck on your classes.

[Simion32]

MemoryDLL Loader Test Program for DELTA

>>> DOWNLOAD TEST HERE <<<

The details go something like this: I want to load DLL files straight from RAM. This isn't just a fluff idea, though.

It has several good backing reasons:
(1) Ability to have a unique file extension other than DLL — to make DELTA plugins recognizable.
(2) Dynamically store (if desired) both 32bit and 64bit DLLs in the same plugin file.
(3) Allows easier detection of LBIDs and makes the fact that these are LB files much more obvious.
(4) Allow the "Pre-Compiled Delta Game" format *.CDL to have DLLs that can be loaded without copying to a disk cache.
(5) Ability to cache the DLL files in RAM, if desired, so that loading plugins a second time becomes much faster.

Please test the program and post:

• your Operating System?
• does the program run all the way through?

[Kingizor]

Windows XP SP3, Windows 2000 & Windows 98.
Finished. No crash!

[Qyzbud]

Windows 7 Home Premium (x64)
Finished. No crash!

Would other PC info be handy, or just the basic OS details for now?
Word has it, basic OS info is fine, people.

[Gaz]

Windows 7 x64 Laptop
Finished. No crash!

[VideoViking]

Windows XP SP3
"Finished. No crash!"

[Markster]

Old computer with 512MB of RAM running Windows XP SP3.

Finished, no crash!

I'm getting my Windows 7 x64 computer with 3GB of ram repair next month!

[Blaziken257]

I have Windows 7 64-bit, and it finished without crashing.

[Mattrizzle]

Windows Vista Home Basic SP2 (x86)
"Finished. No crash!"

(A ~$700 computer with Home Basic? What a rip-off...)