Sunday, June 4, 2017
A few months ago 8Bitdo released an Apple II coloured version of their PRO GamePad Bluetooth controller along with a receiver that can replace the need for an actual Apple II joystick https://www.kickstarter.com/projects/8bitdo/ap40-a-bluetooth-controller-for-ios-android-mac-pc. I love their controller but the receiver was initially implemented without the full analog range which is required by many Apple II games. I believe they have now released a firmware patch which addresses this issue but it makes me sceptical. Is the receiver's hardware design as polished as the product presentation and packaging? I started building my device without knowing if 8Bitdo were going to fix their receiver. This project was never an attempt to make a competing product but to use this as a stepping stone to build bigger and better things. That's the beauty of being able to make something yourself, you can customise it to do whatever you like.
There is no mystery as to how to hook up a wireless controller to the Apple II. It's just a simple extension of RetroConnector's "USB JOYSTICK INTERFACE FOR APPLE II" product (http://retroconnector.com/products/apple-ii/usb-joystick-interface-for-apple-ii/) which has been out for years. Add a USB Bluetooth Host dongle, a HID game controller and a bit of extra code and you're done. If you wanted to stick with the AVR/Arduino theme then the Bluetooth code can be located at https://github.com/felis/USB_Host_Shield_2.0 and the write ups can be found on https://www.circuitsathome.com/mcu/ps3-and-wiimote-game-controllers-on-the-arduino-host-shield-part-3/ and http://blog.tkjelectronics.dk/2013/12/bluetooth-hid-devices-now-supported-by-the-usb-host-library/. Alternately there are several other embedded USB hosts around which could be used. For example http://www.hobbytronics.co.uk/ps4-controller-bluetooth do a pic based one.
I wanted to build a Bluetooth receiver myself because the task did not look all that difficult, I would have full control of how it would work and because I was looking at playing around with a USB host for another project anyway. Initially I looked at using a single Bluetooth module to do the Bluetooth part and the Apple II game port signal processing all in the one chip but it looked like quite a bit of work since most chip manufactures lock down the Bluetooth part due to proprietary and regulatory radio interference reasons. I ended up deciding that the FTDI's Vinculum-II (VNC2) would be a good compromise. The package version of the VNC2 that I obtained (64pin module) is over kill for this project since I'm only using six Input/Output pins but its size will come in handy on my next project. Using FTDI's examples, I was able to get the game controller working using the USB cable method within an hour or two of trying. The remaining part is to write the Bluetooth driver for the VNC2 or port the one from the Arduino platform. This is still a work in progress.
In terms of my hardware design there were two parts to get working, the digital part and the analog part. The trigger buttons are the digital part which is just the result of digital outputs from the VNC2 controller. However because the VNC2 is a 3.3V chip a level converter was required. At first I just tried using a low forward voltage drop diode and a 330ohm pull down resistor but this wasn't enough to register a trigger on the Apple II side. I had a 4066 chip laying around and I used that to make the triggers work. I'm not confident that this is the best way to do a level shifter for the Apple II Game Port. The analog section (X and Y joystick axes) could have been implemented using pulse width modulation (PWM) feeding a low pass filter however I chose to use the proven method instead. Like the RetroConnector option I went with using the Analog Devices' AD5206 chip which contains six virtual/digital potentiometers in a 256 position ladder divider type implementation. I may still implement the PWM option to see how it compares. I thought that the Serial Peripheral Interface (SPI) communications between the AD5206 and the VNC2 would be straight forward since the VNC2 comes with an SPI code library and examples however this is only valid if using 8bit data packets. The AD5206 chip requires an 11bit data packet so I had to bit bang this information out of the VNC2 instead. On the Apple II a single joystick axis requires a 150k potentiometer but the AD5206 chip only comes in 10k, 50k and 100k configurations so on the RetroConnector product two 100k virtual potentiometers are used and then software scales this back. To do the same job I take one 100k virtual potentiometer and effectively scale that to 150k by using extra capacitors (RC network on the 555 timer chip). Since this is already documented I'm not going to reproduce that here. A good explanation can be found on Quinn's blog http://quinndunki.com/blondihacks/?p=2225. After doing the calculations I found that they were slightly out for real world conditions but it did give me a good starting point. I added a diagnostic mode to my Bluetooth receiver which helped in getting the required results. I can fine tune it pretty well however each of my Apple II computers needs slightly different values. Even the x-axis and y-axis setups differ.
I'll add the schematics and code here when I'm done.
Saturday, March 25, 2017
These title pictures are photos taken of a composite monitor attached to a standard Apple IIe with an 80 column card.
Double High Resolution Graphics (DHRG) is the highest resolution of graphics that is natively supported by the 8bit Apple II line of computers. DHGR mode is not available on the earlier models but it is available on the IIe (with a RevB motherboard and an 80column card), the IIc and the IIGS. The monochrome resolution is 560x192 but when using colour this resolution falls back to roughly 140x192 since a block of 4 bits is used to represent 16 different colours (15 colours on the IIe because the two greys produce the same colour). Even after 1986 when the the IIGS was released, my school, myself and many people that I know were still using the IIe with a monochrome monitor. By the time computers with colour monitors were affordable, colour graphics had surpassed DHGR type graphics. Most people will attribute the Apple II with High Resolution Graphics (HGR) and when you compare it to its competitors at the time I believe colour DHGR did not get the opportunity to be represented as much as it could have been. This blog entry is about trying to build a colour relationship model, to show how that can be used to improve the areas where DHGR is used in design and to show off what could have been.
There are references on the web that describe how DHGR came about, how its colours are made up, how DHGR can be programmed and and how the data is stored. Each of these could be discussed at length in their own right but I will not be covering them here today. It was when I was building colour into the A2VideoStreamer that I wanted to know how the colours related to one another but I wasn't able to locate the information easily. I ended up generating a lookup table that gave me the result I was after but I was not content with just using this table. I wanted to delve deeper and understand how the underlying relationships worked. Rearranging the data I was able to put together an Excel spreadsheet showing this relationship. I took this even further and constructed the list of actual block colours and the block relationship table. The source of this data is not based off analog video signals but instead on AppleWin's pixelated estimates of these signals so it's not perfect but it is very good and it helps me to visualise what's going on. I'll be using these simplistic digital models until I can work out the analog ones.
DHGR (in terms of the display) is cut up into 4 bit blocks. You can think of these 4 bit blocks as items in a pull down box. The resolution of 560x192 relates to the monochrome image. For a computer to produce a coloured image at 560x192 (let's call this the imaginary resolution because just like imaginary numbers in mathematics they can't be realized but they are still useful in calculations) it would need to store 4 bits per pixel. The Apple II only has 1 bit per pixel so it only has the capacity for a storage / effective resolution of 140x192 where each 4 bit block corresponds to one of the sixteen colours. The actual resolution that is displayed on the screen is better than the effective resolution. It's somewhere in between the effective and the imaginary resolutions. What makes this feat possible is that each block's pixel colours are affected by its neighbouring block (on both sides of the block).
I don't know if Dazzle Draw was the first program I ever saw containing DHGR but I do remember being amazed by its sample images, especially the monarch butterfly and the room images. I was impressed by how the images made very good use of dithering and made the most of composite colour blending. Sadly, as much as I tried, I was never able to produce images to the same caliber as these examples. Now was my chance to have a good crack. Here is an example I prepared earlier that shows how changing a block of colour changes the actual display.
If we have a line of magenta coloured blocks and we place a single green block in the middle of that line we end up with merging of colours. Some of the green block's pixels change to yellow and black. What also happens is that a pixel in the left magenta block is altered to grey and three pixels in the right magenta block are changed to black. What we see here is that going from the effective resolution (how data is stored) to the actual resolution does not always produce a nice even merge of colour. This results in unwanted artifacts when viewing an image. The colours Magenta (represented in bits as 0001) and Dark Blue (represented in bits as 1000) are probably the biggest contributors in creating this effect and that is because merging with other colours can easily produce these long streams of zero bits resulting in a transformation to the colour black. Other groups of colours can also produce unwanted artifacts. This is not all bad news. Effects like these can be used to our advantage if that is what we want the final outcome to look like. The merging of blocks can be visualized by looking at the block relationship table above.
So why does this information matter and what can it be used for? Well for one, in its simplest form, it was great help when reconstructing a colour image from just a single video bit stream coming from the IIc and IIe. It would greatly help in developing new fonts or graphics editors on modern machines. It can help in improving graphic converters and assist in game development. I believe that we could have had more DHGR content produced if the complexities of DHGR data storage were encapsulated away from the programmer and the graphical artistic part of the design was better documented and concentrated on. How many of today's Apple II software developers would be keener on developing in DHGR over High Resolution, Low Resolution or Double Low Resolution? Let's use this information by putting some theory into practice.
There has been some fantastic work done over the past few years in getting modern images converted over for Apple II display and even more recently having wrappers built for these tools making this process available to the masses (as much as masses can get in the Apple II community). I looked at a few graphic converters (Sheldon's tohgr and Bill's bmp2hgr) and I noticed a few things that made me question what was going on. They made me wonder if we had reached the pinnacle of the DHGR display or if there was still more juice left in the tank, so to speak. I went about trying to test out my suspicions.
In terms of DHGR, these converters were taking an already pre processed image in the effective resolution (140x192 24bit colour) or converting an image first into the effective resolution before doing the processing on the image to turn it into something that was compatible with the Apple II. So instead of going from the effective resolution and working up to the actual resolution I wondered if going from the imaginary resolution (560x192 24bit colour) and working down towards the actual resolution could produce better results. Knowing the DHGR block colour relationship would be critical in testing out this theory.
I started by ripping apart tohgr. This sounds brutal but it really wasn't. Since it has a good structure and a simple approach to the conversion solution it provided a good base for me to add my own extensions. The first thing to get working was colour quantisation. Colour quantisation is the process of reducing the number of colours in an image. In this case we are reducing a 24bit colour into a 4bit colour image.
Here we have the original and two converted images using colour quantisation. The one on the left contains many artifacts which the eye picks up as anomalies. There are few main reasons for this.
- Processing in the effective resolution and allowing the Apple II to convert to the actual resolution as shown by my first example is not great. By knowing what the pixels are actually going to be displayed you can minimise this effect. Using the actual colour block list (list of 159 blocks vs treating the block as one colour ie list of 16 blocks) results in a better outcome.
- Inherent difficulties in trying to represent what our eyes pick up. I don't know of any algorithm that will produce a perfect result. Just because mathematically a picture should look good does not necessarily mean that it will. Mathematically (on average) you should be comfortable if you have one leg stuck in a bucket of hot ashes and the other leg in a bucket of ice however our bodies don't necessarily work that way, especially our eyes. Our eyes perceive different colour frequencies in different ways. Extra processing is needed to limit psycho visual effects.
The image on the right has been processed twice to remove some of the anomalies.
Apart from being able to remove more of the visual faults when using the imaginary resolution instead of the effective resolution we are able to preserve more of the detail of the image. Does this make it a better picture? It all depends on what you want to see. Most of the time is does produce a better image. On occasion there is a compromise between detail and colour quality. You can see an example of this in the thin straight lines on the pilot of the train. Only a few colours are able to give this single pixel thin line over a black background. Therefore the choice is to have thin lines in a limited colour set or thicker lines that match the colour better of the source image.
The next step was to move onto producing dithered images. Dithering introduces noise to an image in a way that makes our eyes average out the colours there by tricking the brain into believing that there are more colours and smoother contours. Because of the introduced noise some of the anomilies that were clearly a problem with colour quantisation are now disguised within the dithering pattern. Knowing the colour relationship helps to iron out these unwanted artifacts. For comparison I have included here the dithered version of the train. Note that the converted images are just previews from the program A2BestPix. The actual output on a composite monitor will have many different factors which will determine how it looks.
There are lots of different dithering methods. Error diffusion wasn't the first dithering method I tried but so far it has produced the best results. With dithering it is inevitable that some detail from the source image will be lost compared to colour quantisation, especially when the image contains small objects. In some cases one may be able to use the best of both worlds and be able to manually cut and paste between the dithered and colour quantised results.
The first thing I tried in terms of dithering was not to implement any dithering code at all. I wanted to test the generation of dithering in an external graphics program called GIMP2 and then just perform colour quantisation on the externally dithered image. The result wasn't all that great. Here we see four pictures. From top to bottom we have one pixel block, two pixel block, three pixel block and four pixel block dithering. So why does the result look so bad? The reason is in the limitation of the Apple II video mode. Dithering models have been developed to alternate two separate single pixels. The four bit blocks of the Apple II do not fit in very well into these models. Looking at the relationship table and the list of actual colour blocks we see that we can't produce a picture with two alternating single pixels. Even alternating two pixels of the same colour at a time is difficult. For example, alternating a block colour 5 with block colour A (in terms of pixels) results in 2 light blue, 2 pink, 1 grey, 2 green, 1 grey and alternating a block colour 5 with block colour 2 gives 3 grey, 2 orange, 1 brown and 2 green. The majority of colours can only be alternated when we use three pixels of the same colour. This means we either treat the whole block as a single colour like existing graphic converters do or we modify / develop new dithering models. I chose to try out the second option.
A good number of revisions later (dozens and dozens of revisions actually) and the results look like this.
A2BestPix Preview. AppleWin NTSC. Photo of the Sony CRT monitor display, using a standard Apple IIe (including an 80 column card).
I can't actually show you how this looks on a real composite monitor because each of these methods is only an estimate. Even the photo of the composite display will have losses / colour adjustments due to the physical camera limitations and file compression routines. Every different model of monitor is going to yield different results. Then there is the PAL/NTSC difference. I chose to display the AppleWin NTSC in "Television" mode instead of "Monitor" mode as that is how it looks on my Sony monitor. The pictures on this blog are quite small so when you view these on a standard composite monitor you will see pixelation. I guarantee it. Unless you're using a one inch screen. It's great considering the technology but don't go expecting HDMI quality stuff. To get the full effect you can download the dsk images attached and view them on real hardware.
Here are a few more examples of what is possible.
If one was really obsessive then these pictures could still be manually touched up in something like Dazzle Draw.
Modeling the way dithering works on multiple pixels was required. We can't just process an individual pixel because the Apple II works in blocks of four but we can process individual pixels and spread the error over the neighbouring blocks. There are so many different ways an error can be spread out. I've implemented just a fraction of what is possible.
There are just as many options when processing the lines as there are processing the blocks. A left to right processing can look very different to a serpentine or even a right to left processing. Not just in terms of the dither pattern but also in terms of the colour propagation. Other factors such as using "<=" instead of just "<" in colour comparisons results in large changes. Shifting the entire image by one, two or three pixels can have a huge difference on the final result.
Dealing with four pixels at a time results in such varying outcomes. Each image source is different as each will have a different way that it lines up with the Apple II blocks. There were so many different combinations that testing which ones were going to produce respectable results was a problem. To produce every single outcome and compare that on the Apple IIe would have been tremendously time consuming. A good preview of what was going to be produced was needed. I had a look at AppleWin NTSC and this produces quite good results but still converting every processed image into a dsk file and using AppleWin to compare would also take up a considerable amount of time. I had to develop a preview using the conversion code. To get a good preview two things were needed. One was knowing how the video is translated from the effective resolution to the actual resolution and having a second independent palette from the processing palette. Both of these I had so I was able to generate a bulk number of preview files and flick through them to pick the best ones.
Working with four bit blocks also causes other issues. Some times the preview can look great but when looking at it on a composite monitor the eye picks up tiny anomalies. You can end up with results where the image will look 99.9% correct but a straight line or colour tinge in the wrong place can make a section of the image look unnatural. You can get instances where the foreground looks great but the background looks average and then another image from the same source results in a great background but an average foreground. That is the reason why I generate several previews using the same source image.
Why DHGR and not the other video modes? Once you get DHGR right then all the other video modes on the 8bit Apples are just restricted subsets. This may not necessarily be the case with digital models but I suspect it will be when using analog ones. I looked into using different dithering techniques, Yliluoma3 and Riemersma, but so far the results have not been as nice as I would like them to be. Error diffusion is not all that great for animation purposes so it would be nice to get these other methods perfected so that they can be used with video and game development. This will most likely benefit the other video modes.
DHGR comparison between the IIe and the IIGS using the same demo disk.
Test pattern: AppleWin, AppleWin NTSC, Composite and RGB.
Displaying these images on a IIGS does not produce the same result. This is because the IIGS does a conversion to RGB (and then again to composite if using the composite port). This results in a different colour palette. The RGB image is sharper than composite so the blending effect of dithering is not as pronounced. Also the conversion process changes the colour relationship. Check out the difference between a test pattern for a composite machine compared to the IIGS. There are instances here that when certain blocks are brought together you get this long red run. ie 0001 0000 1000 turns the middle block red. When I first saw this I thought my IIGS was faulty but I have since tested several IIGSs and a Video Overlay Card and they all produce this same result. I have not seen this behaviour reproduced on any of the IIGS emulators that I have tried. To generate nice looking DHGR on the IIGS requires a different colour palette and a different colour relationship model. Since the IIGS has different video modes that can impress I did not spend any time trying to optimise any DHGR content for it.
Box art in DHGR.
Imagine having title screens like these for our favourite games. Box art could have been brought into the user computer experience and not just be limited to the packaging. If this is the result that can be produced by a layman like me then imagine the stuff that could have been produced by a professional graphic artist. I'm not talking about the actual content here. That was sampled from packaging but instead displaying art on the screen and using the resolution to its fullest.
Here are some results of anaglyph 3D. I pulled out my 3D glasses and had some fun with these 3D conversions. Not bad for 70s technology and a for computer that doesn't even have a video chip. There is so much more that could have been done with this video mode.
Here is my graphic conversion process:-
- Screen capture an image or just load a graphic into Microsoft Paint.
- Cut and paste the section that is needed into a new window. Add black lines to the sides if needed (or the top and bottom) or crop the image and adjust until the graphic ratio is close to 4:3. Save it as a 24bit bitmap image.
- Load the image into Gimp2 and resize it to 512x192. Adjust the colour, contract or brighness if needed. Save the image.
- Rename the file to source.bmp and place it in the same directory as the A2BestPix application.
- Run the application. This will produce several files in the same directory as the application. I have yet to implement external control of the processing options.
- Using the preview files I choose the preview that I like the best and note down the equivalent *.dhgr file.
- O.bmp is the original image file (source.bmp) but stretched vertically so that it looks nicely ratiod.
- Q.bmp is the colour quantised preview.
- Q.dhgr is the colour quantised Apple II graphic file.
- D_xxx.bmp are the dithered previews.
- D_xxx.dhgr are the dithered Apple II graphic files.
Attached are a few demos of my conversions. A2BestPix is also included. The program is still a work in progress but the source is included as reference. https://docs.google.com/open?id=0B5PVarmqxaOnSWRJVm94al9pc2c
Until next time, happy dithering.
Thursday, December 8, 2016
Every now and again I get asked to do archiving of Apple II disks for people who don't have their physical computers anymore. I've archived personal files, commercial software and even disks for court cases. In all that time I've never come across disk corruption when archiving disks for others. That is until now. These disks tended not to be the your everyday usage disks but instead the ones that got written to once and were stored away. So I guess the probability of corruption was low. Sure, I've had corruption on my own disks but I've always been able to work around it by having backups or having software that is readily available from other sources such as online archives. As time goes by and disk media degrades further, having a bit of knowledge on disk corruption can't be a bad thing. This is a write up of my experiences in rescuing a particular software package. I hope it gives others confidence in attempting to repair their own disks when needed.
A few weeks ago I responded to a request that came through from the Australian Apple II enthusiast's email list. Matt was looking for someone local to archive software that had been written by the local state education department. He managed to obtain the physical disk by contacting the developer. This software was used locally in schools so it was not wildly dispersed. I had a decent search for information about this software but I could't find any references to it. Since I have an Apple II system setup that can easily generate images from disks or the reverse I didn't hesitate to call and offer my assistance.
The disk that Matt handed me is the game "Convict" by Paul Holland. It comes in two parts. The first being the research part which I could best describe as an interactive history book. The focus of which is on early European settlement in Brisbane, Queensland, Australia. The second part is a penal colony simulator where your objective is to allocate resources and punishment in the pursuit of setting up and growing a convict settlement. Even before evaluating the contents of the material I prefer to do the copying. That way if the disk is fragile and on it's last legs this will result in extracting the best possible information before the disk gets any worse. So far this hasn't been the case. Most 5.25 inch disks that I have dealt with have stood up pretty well.
Side A of the software copied without a hitch. I was on a roll and thought that within a few minutes I'd be finished. Well that didn't go to plan. Side B had corruption. At first three sectors were showing up as containing errors. To my surprise, the more I tried copying the disk the better it got. It got to a stage where only one sector was showing up as being corrupt. I then tried using different disk drives and different software packages to copy the software but with no luck. The bad sector was there to stay. I wondered if this was a copy protection issue so I tried to run the software to see what I got. The software would not run. Not in the sense that it was broken but in the way that it gave me a warning saying "Do no Write-Protect Program Disk!". The software was protected from being run off this floppy. This was a master disk. This disk was meant to be copied and yet it was now corrupt.
Looking at the bad sector can be done with the software like Copy ][+ however I prefer to use more modern solutions like CiderPress and Omnivore because these reduce the high ASCII codes into lower ones which results in being able to view readable characters. Looking at the corrupt sector and surrounding sectors I could see that this area is comprised of text data. This means that restoring the corruption was going to be easier than restoring program data. The text would be instant confirmation that the reconstruction was moving in the right direction. With the corrupt sector itself I could see that the first few lines looked fine but then a third of the way through the sector it reaches the corrupted area.
I ran with what I had imaged in an emulator and the game was playable but I knew that at some point into the game it would either crash or display unexpected behaviour. I wasn't confident in being able to restore the disk as I had no idea as to how corrupt the data might be. However I decided that until I ran out of options I would give it my best shot.
I exhausted all the copy programs that I had available and since I did not have any specialist archiving tools such as a KryoFlux or an EDD Plus card (not that these would guarantee success) I went with the only option I had left. That was with a logic analyser and some software I had written four years earlier for diagnosing CHED, a floppy disk emulator, I had built at the time. The logic analyser samples the disk data that is being transferred between the Apple II and the Disk II while the software that I wrote processes and evaluates that bitstream.
While using a sector editor on the Apple II I was able to reread the corrupt sector (Track $18 Sector $C) at the same time as using the logic analyser to record the bitstream. The analyser records multiple input lines such as the stepper motors pulses, read request and write data however we are only interested in looking at one line, the data read line. I had already loaded the "AppleIIDiskIIAnalyzer" analyser into the logic analyser software so it processes the Disk II data bytes as soon as the data is sampled. This shows up as disk bytes above the bitstream.
I exported the disk byte data into a text file then loaded it into a Microsoft Access database application for processing. I've built the processing around a database and not just a text file so that I could view the data at each of the conversion steps. It also makes it easy for me to modify the data in the database to test different scenarios, without having to reload the data each time. After processing the data I found that I had gone backwards. The corruption was worse then before. The reason for this was that "AppleIIDiskIIAnalyzer" was treating the timings as being close to optimum like all the other disks I had played around with. However the "Convict" disk was different. The timings had greater variances.
To understand what corruption is and how to fix it the above two diagrams are the most critical parts to understand. Since I'm only dealing with fixing a single corrupted sector I didn't need to worry about sync bytes, the address field or sector skew (Dos 3.3 reads a physical sector then skips two physical sectors to give itself time to processes the data hence Dos 3.3 sectors are numbered differently to physical sectors) so if you wish to cover these parts of the disk II protocol then the best book to read is Don Worth's "Beneath Apple DOS". http://mirrors.apple2.org.za/Apple%20II%20Documentation%20Project/Books/Beneath%20Apple%20DOS.pdf
I'll just be covering the data field. Each disk sector is just a stream of bits. "Convict" is a 16 sector per track disk (not the older 13 sector) so each disk byte is eight bits long, it always starts with a high bit and there can be no more than two consecutive zeros (absence of a high bit). Therefore all "AppleIIDiskIIAnalyzer" does to convert bits to bytes is to wait for a high bit at the start of the byte then process three different types of gap lengths (approx 3 microseconds to denote two consecutive high bits, 7 microseconds to denote a low bit and 11 microseconds to denote two consecutive low bits). Corruption happens when the magnetic structure of the disk is changed so that a high bit is read as a low bit or a low bit is read as a high bit (possibly less likely) or the gap length timings expand / contract. Once the disk bytes are calculated one has to look at the data field as a whole and follow the steps that the disk operating system would take to convert disk bytes into user bytes ie the memory bytes that the Apple II users would be used to when dealing with memory at the assembly language level or peeking / poking in higher level languages. The conversion process takes disk bytes, runs them through a lookup table with a one to one translation, exclusive ORs the result then performs a 6and2 prenibbleisation calculation. All these stages help in determining where the corruption is in the sector.
a. The one to one translation can help by determining if a disk byte is valid because it has to match only one of the sixty four valid disk bytes.
b. The exclusive OR step is probably the best at determining if corruption has happened because every byte is included in the calculation to determine the checksum.
c. The 6and2 prenibbleisation stage is helpful if corruption is contained to a small area because this stage splits the user bytes into different parts of the disk sector.
Because of the corruption in the data, when importing and processing the data bytes we end up with an invalid data field. The prenibble byte calculations get pushed past the checksum and field end bytes. To fix the field size I inserted in some dummy bytes to make the size equal 342 disk bytes plus the checksum.
I rewrote parts of the "AppleIIDiskIIAnalyzer" so that it was more forgiving on timings. What I ended up with was what had initially been produced by the copy programs.
To track the corrupt data all I needed to do was to find the bytes "6F 66 20 61 67" within the "Disk II Data Stream Processor". This I knew had been read without corruption. This gave me the start time of the data field within the bitstream recording. After counting 3 bytes for the data field header then 86 disk bytes which comprise of the compressed "0 0 C0 C1 B0 B1 A0 A1" bytes put me at the beginning of the user prenibble bytes. Traversing another eighty odd characters I arrived at the corruption area. Counting the bitstream bytes showed that indeed this was true. The "AppleIIDiskIIAnalyzer" has difficulty converting the bytes when it hits the bitstream containing the corruption. Looking further afield, the bitstream seems to suggest that the corruption is contained to one area and the protocol finally recovers towards the end of the sector.
To recover the bottom part of the sector I setup the processing to work backwards. If we assume the checksum exclusive OR result is zero and run the exclusive OR calculation in reverse we get the above result. Again I had to tinker with the "AppleIIDiskIIAnalyzer" timings to get a better outcome. The protocol does not recover straight way after the corruption area but a least we have a good chunk that can be worked out automatically. At this point we are left with just seventeen bytes to fix.
Still working from the end of the sector towards the corrupted area the rest of the calculations needed to be done manually. Out came the pen and paper. Working with two or three bytes at a time I was able to reconstruct the data all the way to the corrupted area.
The remaining eight bytes would end up being the most difficult to recover because the actual data was missing ie the disk was corrupt in this area. I placed a template of good looking bytes over the top of the corrupted bytes to give me an idea of how long normal byte lengths were. From this I continued to do byte recognition based from optimal timings. I was more confident over some bytes than others. I played around with different combinations until I stumbled across a solution that fulfilled the exclusive OR calculation. The displayed text was "caibu,ne". I didn't believe I had the right combination of bytes but the exclusive OR calculation worked. I could now turn off the reverse processing code and only deal with processing the data in one direction. The last two bytes made sense in the text so I was confident I was down to six bytes.
How hard can it be to brute force six bytes ie find every valid combination of data bytes? It turns out to be harder than it looks. The program is not just a bunch of "for" loops but instead something that knows the available disk bytes to use, 6and2 byte processing and the bytes it needs from the first 86 bytes of the data field. By brute forcing just two bytes I got a result that showed me sixty four possible combinations that satisfied the exclusive OR calculation. When four bytes were brute forced it generated a file that was half a gigabyte in size. It would have taken me months just to go through the results. I did notice that patterns were appearing in the output results and therefore I didn't need to process all six bytes at a time. In terms of just readable characters I determined that the lists (c,g,k,o,s,w), (a,e,i,m,q,u,y) and (b,f,j,n,r,v,z) could be used so I modified the program to output all the possible readable options. This didn't take too long and only took me about fifteen minutes to go through the results. The only word that worked well was "many" in terms of size and matching the existing converted text. Hooray that was the all the text worked out. I thought I was done.
Omnivore was used to fix the sector on the corrupted disk image. All I had to do now was to check and see if this worked. In the emulator I played the game several times and worked out where Track $18 Sector $C was being used. My celebrations were shorted lived. I realised that the fix did not work as expected. It didn't crash but it cut short the line in question ("build" was displayed instead of "buildings"). So even though the text was fixed, the character in front of the word "many" needed adjusting. I thought this character was a page or paragraph separator but in fact it must be some sort of length descriptor.
I changed the program to only look at solutions that contained the word "many" however it still produced a huge list of results that satisfied the exclusive OR test. I was lucky yet again. The corrupted sector and the text within it formed the part of the game where you were reading a letter (a message of your progress). After you read the letter you were asked if you wanted to read the letter again. I produced several disk images with what I thought may have been the correct replacement bytes. Back in the game I confirmed that I wanted to reread the letter and every time it showed me the letter I was able to swap another disk image and test out another option. Some options produced text but with incorrect line lengths, some options produced blank lines but one option got it spot on. Finally the right bytes were found and tested. I was able to use this same method to test the original disk and sure enough when the letter (message) was being displayed the hard drive generates that distinctive grinding sound and stops the program from continuing. When attempting to continue it keeps trying to reread the corrupted sector.
The images above show the six bytes used in the final cut and their results. I also generated a physical disk with the reconstructed image and then used the logic anlayser to read Track $18 Sector $C. The corrupted bitstream can now be compared with the reconstructed bitstream. The bitstreams do not differ by much. I knew that there were missing bits within the corrupted gaps however what threw me was the gap length variations. In hindsight it looks like I should have been able to reconstruct the bitstream without having to do the brute force option. Next time that may be the case.
You may be thinking this is all well and good but not everybody has a logic analyser. That's true but I think a device could be constructed for under $20 to do the job. I have used a Cypress FX2LP (a $10 breakout board) to sample video signals at 14MHz and send them over USB to a modern computer. To do the same with slow disk drive signals would be a push over. Think poor mans EDD Plus solution. This maybe a project I'll tackle in the future some day.
The "Convict" software can be located here. https://docs.google.com/open?id=0B5PVarmqxaOnT2pJbGZZUUtfMkE
My tools can be found here if one wishes to check out the source code. https://docs.google.com/open?id=0B5PVarmqxaOnYmJ0cTFfaldyMEk
Good luck with you own endeavours.