Exchange 2000/2003 Database Files 101

Exchange data is stored in a database system known as Extensible Storage Engine (ESE) which is a variant of Microsoft Joint Engine Technology (JET). Some of you may be asking why Microsoft doesn’t use SQL Server as the database engine for Exchange. Microsoft have looked into this and ............



............ have even tested an Exchange Server with mailboxes on a SQL database, but for now they’re sticking with good old tried, trusted and reliable ESE. For more information on ESE see the following link: Everything You Ever Wanted to Know about ESE

Several files are used in an Exchange ESE database; we have EDB files, STM files, Transaction Log files and Checkpoint files.

EDB and STM Files
Exchange stores data in two files: an EDB file and an STM file. These files work together with uncommitted entries in transaction log files to form a current Exchange database. There are two types of databases available in Exchange which are: Private store databases which contain mailbox data and Public store databases which contains public folder data.

EDB files store content submitted by MAPI clients (Outlook) and data is stored in a proprietary format called Microsoft Database Encapsulated Format (MDBEF). These database files are organised into a B-Tree database structure using a 4KB page size. For more information on ESE’s B-tree database structure see the following link: ESE's B-tree Database Structure

Exchange 5.5 and earlier versions of Exchange converted all messages from non-MAPI clients to the native MAPI format for use by the Information Store and MAPI clients. If a non-MAPI client requested data, it was converted back to the client’s native format before being sent out. This back and forth conversion consumed processor bandwidth and slowed server performance.

The STM file was developed to alleviate this problem. When messages arrive from non-MAPI clients they are streamed directly to the STM file and data is added to it sequentially in its native format. The STM file does not store data in a B-Tree structure and is accessed via the Exchange Installable File System (ExIFS). For more information on the EXIFS see the following link: Exploring Exchange 2000's ExIFS

When a non-MAPI client requests a message stored in an EDB file the message is converted to the appropriate format for the requesting client. When a non-MAPI client accesses messages stored in an STM file there is no need for any conversion as the mail items are already in the proper state for delivery to the client.

When a MAPI client requests a message stored in an STM file, the Information Store converts the message to the native MAPI format and places it in the EDB file, after which the message remains in the EDB file and is never moved back to the STM file.

Storage Groups
Storage groups contain the Exchange databases or stores as they’re known. The standard edition of Exchange supports only one storage group but the enterprise edition supports up to four storage groups. Each storage group can contain up to five stores. A default installation of Exchange Server will create a storage group that contains a mailbox store, consisting of two files: priv1.edb and priv1.stm, and a public folder store consisting of pub1.edb and pub1.stm. All stores in a storage group share the same set of transaction log files.

Transaction Log Files
Exchange updates its databases through transactions. A transaction is a set of operations that must all be completed together. When an action is performed by an end user in Outlook, a remote procedure call (RPC) is made to Exchange, which then builds the transaction and passes it on to ESE for processing. ESE stores these transactions in volatile memory and then moves this data to the information store databases on the disk. As transactions are recorded in memory they are considered committed to the database even though nothing has yet been written to the database files on the disk.

But surely I would lose the ‘committed’ transactions stored in volatile memory should I experience a system failure I hear you cry. You would be correct if it wasn’t for Transaction Log Files.

Transaction Log Files are optimized for high-speed writes and record transactions as they are written to memory. They are used in the event of a system failure to recover data up to the last committed transaction before the crash. In normal operation the database engine never actually reads the log files. It reads from the log files only if the information store service stops abnormally or crashes and the database engine needs to recover by replaying transactions in the log files. Microsoft recommends storing the transaction log files on a separate volume from the Exchange database files for fault tolerance. Therefore if there is a problem with the disk containing your database files the transaction log files (which are required for a complete database recovery) are unaffected.

Exchange Server transaction logs are always 5MB. The current transaction log file is named edbxx.log (the xx represents the Storage Group identifier) and when this becomes full Exchange Server renames the log to edbxxxxx.log (the xxxxx portion of the name is a hexadecimal number starting with 00001) and creates a new edb.log file to accept transactions.

If the Exchange Server attempts to create a new log file and there is less than 10MB of available disk space it records the outstanding transactions to two reserve log files named res1.log and res2.log to enable it to continue logging long enough to stop the database cleanly.

Checkpoint File
A Checkpoint File is used to keep track of data that has not yet been written to a database file on the disk. The checkpoint file is used in soft recovery scenarios where there is a system failure that does not corrupt the database or the transaction log files. Each storage group maintains a checkpoint file named exx.chk (the xx represents the Storage Group identifier) which records which transactions have been committed to the databases for that storage group. When the database is mounted again, Exchange reads the checkpoint file and begins to replay uncommitted transactions into the exchange databases. If there is no checkpoint file, Exchange replay begins with the oldest log file in the transition log folder for the storage group.


Circular Logging
If you have circular logging turned on (it is disabled by default) Exchange reuses a subset (typically five) of transaction logs instead of creating new ones. When circular logging is enabled you can only perform a full backup. The disadvantage of enabling circular logging is seen in disaster recovery scenarios. For example let’s say you perform one full backup at the end of every day and during the day the disk containing your database files experiences a complete failure but the disk containing your transaction log files is intact. You will have to restore the databases from last night’s backup and you’re going to lose a lot of data as Exchange will only replay the transactions from the circular log files that hadn’t yet been committed to the database which will not contain all of the transactions since the last backup.

A full backup of Exchange backs up the database and transaction log files. A differential or incremental backup only backs up only transaction log files. Log files are purged from the disk after a full or incremental backup.

That's all for now on the topic of Exchange databases.
Marek