Operations Manager 2007 Data Retention and Grooming

By Juanita Wallace,2014-05-05 09:01
12 views 0
Operations Manager 2007 Data Retention and Grooming

Operations DB General

    A dedicated maintenance workflow is executed at Midnight Daily controlled by the RMS and performs the following actions in this order:-

    1. Data Partitioning

    2. Grooming

    Grooming is controlled by the Grooming Settings UI in the Administration console

    Unlike MOM 2005 there is no groom blocking, meaning data is groomed as per settings and does not have to be moved to DW first.

The Table InternalJobHistory shows the grooming history.

Any updates to the Grooming settings are applied immediately.

    In RTM there were no Index Optimization or Statistics update, in SP1 these workflows now exist and are executed daily at 2am relative to the RMS.

    For Operations DB installed on SQL Enterprise Edition, we perform an online re-index, for all other versions of SQL an index re-organize is performed.

    Automatic alert resolution is controlled by the Alert UI in the administration console.

DW Maintenance General

DW Maintenance settings exist in the table MaintenanceSetting. The DW contains 1 list of settings per

    DW instance even if DW contains multiple MGs.

    A dedicated maintenance workflow is executed every minute within the DW to perform the following actions in this order

    1. Allocate storage

    2. Index optimize

    3. Stats updates

    4. Grooming

    5. Aggregation

    In SP1 these tasks have a 30 second threshold. If Data aggregation takes 40 seconds to complete steps 3 and 4 will not execute until the next maintenance run.

    Data Aggregation Information and Grooming settings are stored in the StandardDataSetAggregation

    Table. Updates to this table take effect immediately.

    When grooming, if tables need to be dropped they are dropped one per Grooming Frequency Period. Therefore if grooming frequency is every 4 hours and 4 tables need to be dropped the grooming will be complete after 16 hours.

    Grooming looks at the StandardDataSetTableMap to determine what data exists in each table and will

    groom accordingly.

    SP1 ResKit has a tool called DWDATARP.exe that allows you to view/set the data retention policies for all configured datasets.

    Aside from the inbuilt maintenance we also recommend as part of your normal maintenance windows performing a DB Check Integrity and DB Backup.

    Data Retention & Grooming

    There are two places in the DW where we store data retention-related settings. Config & Instance Data

    For "config space" (your management packs, rules they contain, overrides you've created, etc) and "instance space" (objects discovered, their properties and relationships, etc.) we store setting inside the MaintenanceSetting table. Here are the columns of interest and their default values: Instance space settings:

    1. LastInstanceGroomingDateTime - the last time grooming operations were performed; 2. InstanceGroomingFrequencyMinutes - frequency of the grooming process start (default: 480) 3. (most important) InstanceMaxAgeDays - maximum age (since the day instance was deleted) for the instance space objects (default: 400)

    4. InstanceMaxRowsToGroom - maximum number of objects to delete in one run (default: 5000). Config space settings:

    1. LastConfigGroomingDateTime - the last time grooming operations were performed;

    2. ConfigGroomingFrequencyMinutes - frequency of the grooming process start (default: 60) 3. ManagementPackMaxAgeDays - maximum age for the management pack (since the day MP was uninstalled) (default: 400)

    4. NonSealedManagementPackMaxVersionCount - maximum # of non-sealed MP versions to preserve (independent of the age) (default: 3)

    Based on these settings for config space, sealed MP will be removed 400 days after it was uninstalled from all management groups that are members of the DW. Non-sealed MPs play by the same rules, but in addition we keep up to 3 old versions of non-sealed MP maximum.


    Each data type is stored in a separate structure called "dataset". There is Performance dataset for perf data, state dataset for monitor state transitions, event dataset for events, etc. etc. MPs may introduce new datasets as well. All datasets in existance known today are so-called "standard datasets". For those, we have a set of tables that hold description of the dataset including data retention policies. Non-standard datasets may be introduced (we do not know of one today though) and they don't have to follow the same rules - data retention settings for non-standard datasets are dataset specific. For standard dataset data retention is set at the "aggregation" level. Such that performance "raw" data (samples themselves) stored certain number of days which may be different from the number of days daily aggregates of performance counters are stored. These settings are stored in the StandardDatasetAggregation table. Here are the columns of interest. Note that "primary key" of the table is composite consisting of dataset id (you can lookup which dataset is which id in the Dataset table and AggregationTypeId which can be looked up at the AggregationType table). The defaults vary by dataset / aggregation type:

    1. MaxDataAgeDays - maximum number of days to store data;

    2. GroomingIntervalMinutes - grooming process frequency;

3. MaxRowsToGroom - max number of rows to delete per transaction (see note below);

    4. LastGroomingDateTime - last time grooming process run.

     One important not here is that we do not always groom data row-by row. If data inflow is high (which is usually the case in medium-to-large organizations for performance and event data) we create additional tables to store data. For example, we store first 10M performance samples in the first table. Once we get more data we leave the first table there, create second table and start inserting into it. At the same time we calculate min and max date for the data in the first table (and store it separately in the StandardDatasetTableMap table). Then the grooming process works like that (for certain dataset/aggregation type comnbination): Check to see if we have only one table. If one - delete records row-by-row using DELETE TOP and using MaxRowsToGroom parameter. If there is more then one table, find the table with the oldest "max date" for data in it. If the "max date" is older then retention period - drop entire table if not, leave everything there. So, we do not necessarily "up to date" on grooming all the time. If you have a table which spans one month, we will keep some records one month longer then relly needed, but performance gains of dropping whole table vs. row deletes is so huge that we think it is way better to store a bit more data for a bit longer then to pay the penalty.

Report this document

For any questions or suggestions please email