VOIP Tech Chat → [Other] Seagate DockStar: U-Boot/NAND-flash considerations

For anyone contemplating using either of (Jeff) Doozan's bootloaders, any of the 2-stage bootloaders available, and/or a distribution like OpenWrt's current official JFFS2 based build, I think it's important to know the implications that no one is telling.

A little bit of background is in order. Data on NAND flashes are organized in blocks. A block is the smallest unit of storage that can be erased at a time. Each block is further divided into "pages". A page is the smallest amount of data that is read from or written to flash storage. It is in essence a window into a location on flash. It contains both the user data, and some out-of-band (OOB) data used for storing bad block markers, ECC (Error Correcting Code) and FTL (Flash Translation Layer) information. The user data in a page is also logically divided into smaller units called "sectors", which are the units of data over which individual ECC is generated or correction performed. The ECC codes for each sector are grouped together and stored in the OOB area of each page.

On the Dockstar, blocks are 128KB, pages are 2KB user data plus 64B OOB. Sector size is dependent on the software that does the ECC, but is typically 512B. (Note: I refuse to use the obscene notation KiB, MiB, etc, that some elements are trying to force feed on us.)

NAND flashes are prone to developing bit errors, not only after repeated block erasures and write operations, but also after any read from a page on flash storage. When writing to a (previously erased) page, or reading from a page, other pages within the same block are exposed to voltages that can change the state of individual bits, leading to subsequent bit errors. This is called "read/write disturb".

There's a limit on the number of erase operations that can be performed on a block, as each such causes permanent deterioration that reduces the block's remaining usable life. The limit isn't fixed, but varies depending on flash architecture, and is on average in the order of 1000 for the flash chip installed on Dockstars.

Read/write disturb doesn't directly reduce the life span, but reprogramming the data once more bit errors develop than can be corrected, involves erasing (unless written elsewhere, but on aggregate still eventually leads to erasing blocks), which reduces the life.

Once a block becomes unusable, due to excessive, uncorrectable bit errors, it is usually marked bad by flash management software in the OOB area of the block's first page, and remapped somewhere else on the chip using FTL techniques.

To deal with the inevitable wear that comes from erasing and reprogramming blocks, and try to extend the usable life of the chip as a whole, NAND flash software often employ a combination of FTL and 'wear leveling' techniques, in an attempt to spread the wear evenly over all the data blocks on the flash chip. The JFFS2 and UBIFS flash file systems do this.

To cope with data bit errors, ECCs are used for detecting and correcting them. The default ECC algorithm typically found in and used by the open source MTD driver and U-Boot, is based on Hamming code, and is only capable of detecting and correcting a single bit error per sector (512B on the Dockstar flash). The makers of the flash chip installed on the Dockstar do not appear to consider that algorithm adequate, as they recommend using a better algorithm instead.

The boot program U-Boot on the Dockstar configures its boot-up run time properties from an environment it maintains in flash. It does not use FTL mapping on the location where the environment is stored, but can be configured at compile time to allocate multiple sequential blocks for the environments, such that if one block becomes bad, it marks it bad, and just skips to the next good one allocated. It can also be compiled to use a scheme of dual alternating environment copies, where it keeps a copy of the previous environment after any changes to it, something the U-Boot authors technically incorrectly refer as redundant (a truly redundant one would keep an identical copy of the current environment, not a copy of the previous one).

Neither the stock U-Boot nor any of the other U-Boots (mtd0 replacement or 2-stage) available use any such methods, but instead just allocate and use a single environment in a single flash block. Once that block becomes unusable, the Dockstar becomes essentially unbootable.

To mitigate this problem, and extend the life of the device to something reasonable, the company Cloud Engines (nicknamed by me and henceforth referred to as "Overcast Propulsion", or OCP) that produced the stock U-Boot that ships with the Dockstar, as they obviously heeded the manufacturer recommendations and didn't consider the default 1-bit ECC adequate for protecting the integrity of the boot environment, chose to replace it with one that can detect and correct up to 4 bits per sector. I don't believe it was done out of some conspiracy to spite the open source community, as many voices on forums out there would have it seem, but instead for quality purposes, as the environment can potentially be modified fairly extensively, especially in the production Pogoplug firmware to store non-volatile settings when used with their service, and the 2-stage setups where the bootup alternates between Pogoplug and an external distro.

Doozan's and the other U-Boots out there are content with just using the inadequate default 1-bit ECC. Personally, I wouldn't touch any of them with a ten foot pole.

There's been talk on this and other threads voicing desire to put the whole of the Linux distro on a writable file system in a single large MTD partition. Others voice a preference for having the root-fs in a read-only file system, with a separate writable file system.

In my opinion, putting everything on a single writable file system, such as JFFS2 or UBIFS, is not a good idea. Not withstanding the FTL and wear leveling techniques used by the aforementioned file systems, NAND flash is good for and reliable for storage of files that are accessed relatively little, but not for files that are constantly and repeatedly being accessed.

Modern operating systems, including Linux, use memory mapped file access when running programs. When the available RAM is exhausted, Linux will discard currently unused pages of loaded programs to try to satisfy any memory allocation requests. Discarded pages will then be reloaded from the underlying flash file system when again needed.

When a device is run in such a way that results in pages being discarded and reloaded often, it speeds up the corruption of such files with bit errors from read disturb. The same is true with files that are run often, for example from jobs that are started periodically. This is all true even with read-only file systems such as squashfs, so putting your OS on such as file system doesn't help much.

If all files have to be kept in the internal flash, the best way is to include all files that are accessed often in a root-fs that loads entirely into a RAM-disk without the possibility of backing access from flash. That way, access it limited to once per boot-up. Otherwise, it is preferable to keep the OS distro on an external device. Even though an external memory stick, being a NAND-flash device, is susceptible to the same wear, at least it's easily replaceable when symptoms of failure start appearing.

Edit: title changed to better reflect subject of thread.

Thanks for the detailed post. A question and a request.

The 2-stage boot loader is configured to boot an external USB drive. I understand that the if the external drive is a flash drive, and mounted rw, it would eventually wear out. However, once setup, there are no writes to the internal NAND flash. Is the internal flash prone to damage even in this configuration?

Could you (or the moderators) please post this as a new thread?

Thanks,
-m

The default configuration of the 2-stage setups rewrites the environment upon each boot-up (at least in the original instructions linked to by the OP), in order to alternate booting between the external OS and the internal Pogoplug firmware. That rewrite causes wear, and decreases the life expectancy of the environment block, although it is somewhat mitigated when keeping the stock U-Boot, as it uses the better 4-bit ECC. If configured to only boot from the external OS, then it would only read its environment (and secondary U-Boot) once per boot-up, thus maximizing the life span. It will eventually still error out due to read disturb, but these don't leave permanent damage to the block, as erases do. Note also, that the secondary U-Boot is only protected with a 1-bit ECC, even when loaded from the stock U-Boot.

The primary U-Boot's environment will typically last longer without (or with correctable) errors than the secondary's 1-bit ECC environment (if it uses its own writable environment). Once errors start popping up due to read disturb, it is possible to rewrite the environment, thus refreshing and fixing it for another while. One has significantly better chance to be able to refresh it before errors become uncorrectable, by using boot loaders with better ECC than the default 1-bit of the current alternative bootloaders.

As for moving this discussion to another thread, I have no objection. I don't think I can move it myself, though, other than by starting a new thread and reposting. You're welcome, however, to contact the moderator and requesting that it be done.

From Flash wiki:
Memory wear
"Another limitation is that flash memory has a finite number of program-erase cycles (typically written as P/E cycles). This limitation is meaningless for 'read-only' applications such as thin clients and routers, which are only programmed once or at most a few times during their lifetimes."

Do you mean Linux ro file system is not really Read Only?

A read-only filesystem is really read only. If I understood DogFace05 correctly, he informs us that even reading the NAND flash may eventually lead to errors ( read disturb ) on the erased locations cells.

The default 2-stage configuration toggles between two systems by writing to the uboot config area. Hence I have switched permanently to my external flash-drive.

cheers,
-m

No, read-only on Linux is read-only.

Both traditional and NAND flashes have a limit on erase cycles. However, unlike NAND flash, blocks on traditional flash can in practice be read from just about any number of times. Thus the life span of a block in a read only flash file system on traditional flash is not affected by reading from it.

Not so with NAND flash. Every time a NAND flash page is read from, it will disturb and eventually cause bits in other pages of the same block to change state and thus become corrupted. The more often you read from the read only flash file system on NAND flash, the sooner it will become corrupt. In other words, the error free life span of even a read only flash file system on NAND flash is affected by how often it is read from, unlike traditional flash.

I'm new to NAND flash. I did some search but could not find information about NAND life time is affected by read cycle, do you have some links?
This will affect my decisions to configure my Dockstar.

Look for read disturb in this document »download.micron.com/pdf/ ··· 2917.pdf

-m

It's a good paper, it mentioned:
"Performing a high number (hundreds of thousands
or millions) of READ operations on individual pages before an ERASE command
for the block containing those pages can exacerbate this error".
I believe those type of errors can be reduced by firmware, e.g., Jeff's U-boot reads kernel at boot time to SDRAM and boot from there.
That is one of the reasons to keep Dockstar Linux as "recovery system" or "Rescue System":
»forum.doozan.com/list.php?4

Yet I wish it would try attached USB media, then default to its own OS if none is found.
(Rather than alternate boots that likely do more writing on its internal flash.)

Which boot loader are you using?
After I installed Jeff's one stage mtd0 boot loader and Debian on mtd3, my Dockstar always try to boot from USB first, then Debian on internal mtd3 and then to Dockstar linux if installation of Debian on mtd3 fails, by the following bootcmd in env

bootcmd=run bootcmd_usb; run bootcmd_ubi; run bootcmd_pogo; reset

bootcmd_ubi=run set_bootargs_ubi; ubi part data; ubifsmount rootfs; ubifsload 0x800000 /boot/uImage; ubifsload 0x1100000 /boot/uInitrd; bootm 0x800000 0x1100000

bootcmd_pogo=fsload uboot-original-mtd0.kwb; go 0x800200

bootcmd_usb=run usb_init; run usb_load_uimage; run set_bootargs_usb; run usb_boot;

The "bootcmd_ubi" loads uImage and roo-fs to SDRAM (0x800000 and 0x1100000) and boot from there.

This is a really good explanation. Thank you DogFace05.

Additionally, we should consider Linux mount options atime and diratime, enabled by default in OpenWrt, which update (write to filesystem) files and directories access time every time these are accessed. If I understand correctly, relatime option will delay this update but it occurs later.

EDIT:

I guess we should set noatime and nodiratime options on jffs and ubifs file systems.

By the ways, this thread title should be changed to avoid confusion with another one started by mazilo.

The one that alternates.
So there is one that works the way I'd prefer?
Still, I'm waiting for the dust to settle. (I have a second device still in the box.)

Yes. Glad you are involved.
I feel like we have the professor from Gilligan's Island on our side

+1

The one stage boot loader to boot to USB is here:
»jeff.doozan.com/debian/uboot/

If you want to install one stage boot loader on mtd0 and Debian on mtd3, the following should do it:

»forum.doozan.com/read.php?2,72

After install Debian on mtd3, you should still have Dockstar Linux on mtd1 and mtd2.

You can change boot orders by modify the following bootcmd:

bootcmd=run bootcmd_usb; run bootcmd_ubi; run bootcmd_pogo; reset

I believe Dogface05's point is Jeff's boot loader is flawed because only 1 bit ECC is used for ENVs, it's more likely ENVs get corrupted making it impossible to boot from USB or internal Linux. When ENVs get corrupted, boot loaders will load default ENVs from boot loader, may or may not allow users to access from Ethernet port. Serial cables may be need to recover ENVs.

Dogface05 is developing a boot loader with 4 bit ECC and compatible with Dockstar ENV tools.

When that is done, I would like to test it on my Dockstar. I'm not a Linux expert, but I consider myself a "recovery expert" for Dockstar because I had recovered my Dockstar from completely erased NAND.

BTW, I have tested Dogface05' nice env tool which runs on Dockstar or internal/external Linux. The env tool uses 4 bit ECC and is compatible with Dockstar Uboot to print or modify boot loader environment variables.

The life time of the NAND device isn't directly affected by read disturb. I was referring to the "error free" (or data integrity) life time of a flash file system installed on it; in other words, how long it is usable before errors make it necessary to refresh and correct bad data.

With a writable file system (FS), you can just rewrite individual files to refresh them as they become corrupt, provided that the OS is still bootable, while for a read-only FS you have to rewrite the entire FS.

With either FS, depending on where the corruption happens, it may or may not render the OS unbootable. If it becomes unbootable, it can present quite a challenge for the average user of the device to recover, requiring accessing the boot loader over a serial connection.

For a R/O FS, you can just re-flash it from a backup. If you have a writable FS with critical data that were written to it since its virgin JFFS2 (or UBIFS, etc) FS image was installed, its quite a bit more challenging to recover without loss, but can be done.

In any event, use of a serial connection (or even J-TAG) is not a big deal for you and many other enthusiasts, but for the vast majority of users, it may not be so simple or, at the very least, is a hassle they might rather not have to deal with.

Knowing what factors cause the occurrence of bit errors, and what can be done to minimize their impact, allows a user to decide on the best combination of means to maximize the reliability of their device for their own particular uses and needs.

As for a link, you've already been given one by another member, but there are several other good papers on the subject available at Micron's site. I don't have any specific links handy, but you can just browse over to their web site and do a search on the topic.

I hope Micron paper said is true

"Performing a high number (hundreds of thousands
or millions) of READ operations on individual pages before an ERASE command for the block containing those pages can exacerbate this error".

If RO fs is only read at boot time, Dockstars can last long enough before they are replaced by better devices.

I believe RO system is still more preferred than RW system for NAND.

We have used Windows XP embedded for mission critical systems, where on board NADN flash is Read Only. Enhanced Write Filter is used to write NAND only when user issues:
"ewfmgr -commit". Otherwise the system will boot every time to the same state even after power failures.

quote:

---

quote:

---

said by voiceipuser See Profile :

This is a really good explanation. Thank you DogFace05.

---

Yes. Glad you are involved.
I feel like we have the professor from Gilligan's Island on our side

---

Since I am not good with English, I guess I am glad I didn't understand your comment. I just wanted to recognize DogFace05 explanation. I didn't want to be prepotent.

No. I was saying DogFace056 is really smart and its great he is here
A very good thing!

Gilligan's Island was an old TV show.

Erase cycles cause cumulative degradation of a block, and thus are statistically dependent events. That is, every time you erase/program a block, the probability becomes gradually greater that it will reach its end of life and fail permanently.

Read disturb bit errors, on the other hand, are random and statistically independent events. That is, the probability of a bit error occurring for a given page read doesn't depend on previous reads. It is essentially the same constant probability for each individual read.

Micron's NAND flashes are rated for an average of about 1,000,000 reads/bit-error for SLC, and 100,000 reads/bit-error for MLC type flashes. These figures indicate the average rate at which one may expect to run into a bit error, not how many times one has to read a page before bit errors start appearing. You could, although extremely unlikely, end up getting a bit error in a page the very first time you read from another page within the same block.

Although 1M reads sounds like quite a bit, depending on how you use the device, it may not be so much.

As an example, my latest OpenWrt build takes up just over 8MB including both the kernel and rootfs. It takes some 4000 page reads to bring it all into RAM, which means I should expect a bit error on average every 250 boots, provided the flash locations where those are stored are never accessed again during normal operation. This is the best possible case.

If instead, one runs a glibc based distribution like Debian, that isn't as frugal with memory use, one will likely end up loading a much larger image, with a proportional reduction in the number of boots between running into bit errors.

If instead of having the rootfs placed in non-paged RAM, it's placed in a flash file system partition (whether writable or not), then one runs the risk that reads will need to be done to re-fetch pages that are discarded by the virtual memory paging system during use. This is exacerbated the more memory taxing the distro. Debian, for example, would be far more likely to require frequent paging re-fetches than OpenWrt.

And then on top of it, if one runs memory hogging applications such as gcc (for example, to to build distros natively), it's pretty much guaranteed that there will be a great deal of paging re-fetches.

Gcc is an intense memory hog, especially when building itself. One of the c files that is compiled during the build, causes a nearly 2GB memory allocation. My computers don't have that amount of virtual memory available, so whenever I've built a gcc toolchain, I've had to edit the make file for that file to add a dependency rule with an awk script that splits the file in 3 pieces. Otherwise, the build would fail. For systems that have enough virtual memory, that single file can still take several hours to days to compile, depending on the amount of actual RAM available and CPU speed. Split in 3 with the awk script, these 3 files compile in a few minutes, at worst.

Anyway, doing such things as building gcc natively on a DockStar, with the rootfs residing in a flash file system, would certainly push the limits of the read disturb error tolerance. If one intends to use the device for development, it is certainly best to keep the rootfs loaded in non-pageable RAM, or on an external device or hard drive.

And there are plenty of other memory taxing applications that can cause a lot of paging discards/re-fetches, pushing the number of reads to the 1 million figure within fairly short order.

A bit error can occur anywhere. It may not necessarily always happen in a file that is critical for booting the OS, but the more often such errors occur, the likelier that eventually it will crash the OS. So reducing the amount of paging from flash, prolongs the length of trouble free use.

I'm not trying to sound alarmist. Just think people should benefit from knowing what factors are involved that affect the reliability of data stored on flash, and what measures can be taken for getting adequate reliability depending on particular use.

For some light uses, it is perfectly fine to run the whole shebang from a writable flash file system on the DockStar. However, I see people rushing to install such distributions as Debian (or even OpenWrt) on the MTD3 partition with nary a thought of any relevant implications, and many then using the device in applications that are essentially device suicide when run in such a manner. Just some thought put into how one configures and installs the distros, can significantly increase the trouble free life of the unit.

I am developing a replacement U-Boot, and yes, I am somewhat biased about my work, and think the results will be better than currently available U-Boot alternatives for the DockStar. It might therefore seem like I'm trying to prop my own over the others. However, the current discussion is really more about what practices can in general improve the longevity and reliability of the DockStar. Where one locates the rootfs is not specific to any U-Boot, but has the same relevance to flash reliability no matter which boot loader is chosen.

I have no current plans for making and distributing a Linux distro, nor modifying the current Linux 1-bit ECC MTD / flash file system drivers to use 4-bit ECC (can't afford continuing to spend much more time on this).Indeed, it is always a good idea to have a fallback/recovery system in place.

Very good point. There is an option when building OpenWrt (don't recall if it's in the OpenWrt menuconfig, or the kernel menuconfig), that allows controlling these these parameters.

You are absolutely correct.

I'm wandering how many those type of bit errors can be corrected by ECC and boot will be still successful after ECC.

As fall back, if Debian on mtd3 fails to boot, Dockstar will boot to Linux on mtd1 and mtd2 allowing Ethernet access to recover.

I believe some newer u-boots have net consoles which allow users to recover without serial cable.

Eric C.C. proposed additional "recovery kernel" which is only 512KB and can be put somewhere on NAND for recoveries.
»forum.doozan.com/read.php?3,1114
Unfortunately, his "recovery kernel" needs serial console access.

Only one bit error per 512B sector is correctable, unless one updates the MTD or file system drivers to utilize 4-bit ECC.Is this something he's proposing as a third recovery, or a secondary for when the Pogoplug Linux is no longer present?

Unfortunately again, Eric C.C. is the one who prefers to remove Pogo Linux and have a big NADN partition for OpenWrt.

However, his idea of small recover kernel, could be used in addition to other Linux on NAND, e.g., boot orders:
USB -> Debain (or OpenWrt)->Pogo->Recovery.

Since DockStar was designed relaying on network connections (no real time clock), may be NFS should be used as primary file system and/or recovery file system.

U-boot has "tftpboot" built-in, may be that's something to try.

If I got this right the best solution will be to leave Dockstar Linux in place, install Debian or OpenWRT on a USB stick or hard disk and have an U-Boot who will try to boot from USB and if it fails to boot Dockstar Linux.

Nothing wrong with replacing Pogo Linux with another "recovery" Linux. I would at least include networking ability in it, though, so one can connect to it without having to appart the unit to access the serial interface.

Another option would be to set up one of the USB ports as a gadget device emulating a USB serial port. I'm not sure if there are such Linux drivers available open source, but it would be ideal, especially if added to U-Boot.

There are some references in the U-Boot code for some drivers that seem to allow using a USB port with a cross-wired USB cable to do something of the sort, and redirect stdin, stdout and stderr to that USB port, but I haven't looked too much into detail about it. This would eliminate any need for using a normal serial connection.Nothing wrong with this scenario, but it would probably take alot for both of the first two to fail simultaneously, under normal circumstances. Under development and experimentation, well, that's another ballpark altogether.I have what I think is a better alternative in mind, but I don't want to comment about it until I've verified that it is feasible.

The best solution really depends on how one plans on using the DockStar. For my own purposes on my primary DockStar, I want a recovery Linux (could be Pogoplug Linux or a minimal distro ala Ecc Eric) on the DockStar's flash, and my main Linux running from an external hard drive. I expect this to be a fairly high utilization device, so I don't want it taxing the onboard flash. For my secondary DockStar, I haven't yet decided what I'm going to do with it, so all options are still open.

For other uses that are light on flash access, it's perfectly fine to run it all embedded.