Merge: Merging QofBook structures
[Query Object Framework]

Detailed Description

Collision handling principles.

  1. Always check for a GUID first and compare. qofbookmerge only accepts valid QofBook data and therefore ALL objects in the import book will include valid GUID's.
  2. If the original import data did not contain a GUID (e.g. an external non-GnuCash source) the GUID values will have been created during the import and will not match any existing GUID's in the target book so objects that do not have a GUID match cannot be assumed to be MERGE_NEW - parameter values must be checked.
  3. If import contains data from closed books, store the data from the closed books in the current book as active. i.e. re-open the books.

More information is at

Each foreach function uses g_return_if_fail checks to protect the target book. If any essential data is missing, the loop returns without changing the target book. Note that this will not set or return an error value. However, g_return is only used for critical errors that arise from programming errors, not for invalid import data which should be cleaned up before creating the import QofBook.

Only qof_book_merge_update_result and qof_book_merge_commit return any error values to the calling process. qof_book_merge_init returns a pointer to the QofBookMergeData struct - the calling process needs to make sure this is non-NULL to know that the Init has been successful.

Prior to libqof2 rationalise internal variable names.


file  qofbookmerge.h
 API for merging two QofBook structures with collision handling.

Data Structures

struct  QofBookMergeRule
 One rule per entity, built into a single GList for the entire merge. More...
struct  QofBookMergeData
 mergeData contains the essential context data for any merge. More...

qof_book_merge API

typedef void(* QofBookMergeRuleForeachCB )(QofBookMergeData *, QofBookMergeRule *, guint)
 Definition of the dialogue control callback routine.
QofBookMergeDataqof_book_merge_init (QofBook *importBook, QofBook *targetBook)
 Initialise the QofBookMerge process.
void qof_book_merge_rule_foreach (QofBookMergeData *mergeData, QofBookMergeRuleForeachCB callback, QofBookMergeResult mergeResult)
 Dialogue Control Callback.
gchar * qof_book_merge_param_as_string (QofParam *qtparam, QofEntity *qtEnt)
 provides easy string access to parameter data for dialogue use
QofBookMergeDataqof_book_merge_update_result (QofBookMergeData *mergeData, QofBookMergeResult tag)
 called by dialogue callback to set the result of user intervention
gint qof_book_merge_commit (QofBookMergeData *mergeData)
 Commits the import data to the target book.
void qof_book_merge_abort (QofBookMergeData *mergeData)
 Abort the merge and free all memory allocated by the merge.


enum  QofBookMergeResult {
 Results of collisions and user resolution. More...

Typedef Documentation

typedef void(* QofBookMergeRuleForeachCB)(QofBookMergeData *, QofBookMergeRule *, guint)

Definition of the dialogue control callback routine.

All MERGE_REPORT rules must be offered for user intervention using this template.
Commit will fail if any rules are still tagged as MERGE_REPORT.

Calling processes are free to also offer MERGE_NEW, MERGE_UPDATE, MERGE_DUPLICATE and MERGE_ABSOLUTE for user intervention. Attempting to query MERGE_INVALID rules will cause an error.

For an example, consider test_rule_loop, declared as:

void test_rule_loop(QofBookMergeData *mergeData, QofBookMergeRule *rule, guint remainder);
void test_rule_loop(QofBookMergeData *mergeData, QofBookMergeRule *rule, guint remainder)
g_return_if_fail(rule != NULL);
g_return_if_fail(mergeData != NULL); printf("Rule Result %s", rule->mergeType);
qof_book_merge_update_result(mergeData, rule, MERGE_UPDATE);

The dialogue is free to call qof_book_merge_update_result in the loop or at the end as long as the link between the rule and the result is maintained, e.g. by using a GHashTable.
The parameters are:

If the dialogue sets any rule result to MERGE_INVALID, the import will abort when qof_book_merge_commit is called. It is the responsibility of the calling function to handle the error code from qof_book_merge_commit, close the dialogue and return. The merge routines in these files will already have halted the merge operation and freed any memory allocated to merge structures before returning the error code. There is no need for the dialogue process to report back to QofBookMerge in this situation.

Definition at line 321 of file qofbookmerge.h.

Enumeration Type Documentation

enum QofBookMergeResult

Results of collisions and user resolution.

All rules are initialised as MERGE_UNDEF. Once the comparison is complete, each object within the import will be updated.

MERGE_ABSOLUTE, MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE can be reported to the user along with all MERGE_REPORT objects for confirmation. It may be useful later to allow MERGE_ABSOLUTE, MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE to not be reported, if the user sets a preferences option for each result. (Always accept new items: Y/N default NO, ignores all MERGE_NEW if set to Y etc.) This option would not require any changes to qofbookmerge.

MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE are only actioned after conflicts are resolved by the user using a dialog and all MERGE_REPORT objects are re-assigned to one of MERGE_NEW, MERGE_DUPLICATE or MERGE_UPDATE. There is no automatic merge, even if no entities are tagged as MERGE_REPORT, the calling process must still check for REPORT items using qof_book_merge_rule_foreach and call qof_book_merge_commit.

MERGE_INVALID data should be rare and allows for user-abort - the imported file/source may be corrupted and the prescence of invalid data should raise concerns that the rest of the data may be corrupted, damaged or otherwise altered. If any entity is tagged as MERGE_INVALID, the merge operation will abort and leave the target book completely unchanged.

MERGE_ABSOLUTE is only used for a complete match. The import object contains the same data in the same parameters with no omissions or amendments. If any data is missing, amended or added, the data is labelled MERGE_UPDATE.

Every piece of data has a corresponding result. Only when the count of items labelled MERGE_REPORT is equal to zero are MERGE_NEW and MERGE_UPDATE items added to the existing book.
MERGE_DUPLICATE items are silently ignored. Aborting the dialogue/process (by the user or in a program crash) at any point before the final commit leaves the existing book completely untouched.

MERGE_UNDEF  default value before comparison is made.
MERGE_ABSOLUTE  GUID exact match, no new data - ignore
MERGE_NEW  import object does not exist in the target book - add
MERGE_REPORT  import object needs user intervention - report
MERGE_DUPLICATE  import object with different GUID exactly matches existing GUID - ignore
MERGE_UPDATE  import object matches an existing entity but includes new or modified parameter data - update
MERGE_INVALID  import object didn't match registered object or parameter types or user decided to abort - abort

Definition at line 125 of file qofbookmerge.h.

00126 {
00127     MERGE_UNDEF,     
00128     MERGE_ABSOLUTE,  
00129     MERGE_NEW,       
00131     MERGE_REPORT,    
00134     MERGE_UPDATE,    
00136     MERGE_INVALID    
00138 } QofBookMergeResult;

Function Documentation

void qof_book_merge_abort ( QofBookMergeData mergeData  ) 

Abort the merge and free all memory allocated by the merge.

Sometimes, setting MERGE_INVALID is insufficient: e.g. if the user aborts the merge from outside the functions dealing with the merge ruleset. This function causes an immediate abort - the calling process must start again at Init if a new merge is required.

Definition at line 1024 of file qofbookmerge.c.

01027     {
01028         param_string = qtparam->param_getfcn (qtEnt, qtparam);
01029         if (param_string == NULL)
01030             param_string = "";
01031         return param_string;
01032     }
01033     if (safe_strcmp (paramType, QOF_TYPE_TIME) == 0)
01034     {
01035         QofDate *qd;
01037         param_qt = qof_time_copy (
01038             qtparam->param_getfcn (qtEnt, qtparam));
01039         if (!param_qt)
01040             return NULL;
01041         qd = qof_date_from_qtime (param_qt);
01042         param_string = qof_date_print (qd, QOF_DATE_FORMAT_UTC);
01043         qof_date_free (qd);
01044         qof_time_free (param_qt);
01045         return param_string;
01046     }
01048     if (safe_strcmp (paramType, QOF_TYPE_DATE) == 0)
01049     {
01050         Timespec param_ts, (*date_getter) (QofEntity *, QofParam *);

gint qof_book_merge_commit ( QofBookMergeData mergeData  ) 

Commits the import data to the target book.

The last function in the API and the final part of any QofBookMerge operation.

qof_book_merge_commit will abort the entire merge operation if any rule is set to MERGE_INVALID. It is the responsibility of the calling function to handle the error code from qof_book_mergeCommit, close the dialogue and return. qof_book_merge_commit will already have halted the merge operation and freed any memory allocated to all merge structures before returning the error code. There is no way for the dialogue process to report back to qof_book_merge in this situation.

qof_book_merge_commit checks for any entities still tagged as MERGE_REPORT and then proceeds to import all entities tagged as MERGE_UPDATE or MERGE_NEW into the target book.
This final process cannot be UNDONE!

mergeData the merge context, QofBookMergeData*
  • -2 if any rules are tagged as MERGE_INVALID
    • mergeData will have been g_free'd).
    • note that this will be before any operations are done on the target QofBook.
  • -1 if mergeData is invalid or no merge has been initialised with qof_book_merge_init - the calling process must check the value of mergeData
  • +1 if some entities are still tagged as MERGE_REPORT - use qof_book_merge_update_rule and try again (mergeData is retained).
  • 0 on success - mergeData will have been freed.

Definition at line 1218 of file qofbookmerge.c.

01218 {
01219     struct QofBookMergeRuleIterate qiter;
01220     QofBookMergeRule *currentRule;
01221     GList *matching_rules, *node;
01223     g_return_if_fail (cb != NULL);
01224     g_return_if_fail (mergeData != NULL);
01225     currentRule = mergeData->currentRule;
01226     g_return_if_fail (mergeResult > 0);
01227     g_return_if_fail (mergeResult != MERGE_INVALID);
01228     g_return_if_fail (mergeData->abort == FALSE);
01229     qiter.fcn = cb;
01230 = mergeData;
01231     matching_rules = NULL;
01232     for (node = mergeData->mergeList; node != NULL; node = node->next)
01233     {
01234         currentRule = node->data;
01235         if (currentRule->mergeResult == mergeResult)
01236             matching_rules = g_list_prepend (matching_rules, 
01237                 currentRule);
01238     }
01239     qiter.remainder = g_list_length (matching_rules);
01240     g_list_foreach (matching_rules, qof_book_merge_rule_cb, &qiter);
01241     g_list_free (matching_rules);
01242 }
01244 /* ============================================================== */
01245 /* ============================================================== */

QofBookMergeData* qof_book_merge_init ( QofBook importBook,
QofBook targetBook 

Initialise the QofBookMerge process.

First function of the QofBookMerge API. Every merge must begin with init.

Requires the book to import (QofBook *) and the book to receive the import, the target book (QofBook *). Returns a pointer to QofBookMergeData which must be checked for a NULL before continuing.

  1. Invoke the callback qof_book_merge_foreach_type on every registered object class definition.
  2. Callback obtains the registered parameter list for each object type. This provides run time access to all registered objects and all object parameters without any changes to QofBookMerge - no registered object or parameter is omitted from any merge operation.
  3. Use qof_object_foreach to invoke the callback qof_book_merge_foreach, one object at a time on every instance stored in mergeBook. This is the first point where real data from the import book is accessed.
  4. qof_book_merge_foreach obtains the GUID for the object from the import book and runs the first check on the original book, checking for any exact GUID match. With the full parameter list, the rules for this object can be created. If there is a GUID match, the data in each parameter of the import object is compared with the same semantic object in the original book. If there is no GUID in the import object or no GUID match with the original book, the original book is searched to find a parameter match - checking for a MERGE_DUPLICATE result.
  5. qof_book_merge_compare sets the QofBookMergeResult of the comparison.
  6. Inserts the completed rule into QofBookMergeData::mergeList GSList.

NULL in case of error, otherwise a QofBookMergeData* metadata context.

Definition at line 987 of file qofbookmerge.c.

01000                           : 2005-01-01T10:55:23Z
01001 If you change QOF_UTC_DATE_FORMAT, change 
01002 backend/file/qsf-xml.c : qsf_entity_foreach to
01003 reformat to QSF_XSD_TIME or the QSF XML will
01004 FAIL the schema validation and QSF exports will become invalid.
01006 The QOF_TYPE_BOOLEAN is lowercase for the same reason.
01007 */
01009 gchar *
01010 qof_book_merge_param_as_string (QofParam * qtparam, QofEntity * qtEnt)
01011 {
01012     gchar *param_string;
01013     gchar param_sa[GUID_ENCODING_LENGTH + 1];
01014     QofType paramType;
01015     const GUID *param_guid;
01016     QofTime *param_qt;
01017     QofNumeric param_numeric, (*numeric_getter) (QofEntity *, QofParam *);
01018     gdouble param_double, (*double_getter) (QofEntity *, QofParam *);
01019     gboolean param_boolean, (*boolean_getter) (QofEntity *, QofParam *);
01020     gint32 param_i32, (*int32_getter) (QofEntity *, QofParam *);
01021     gint64 param_i64, (*int64_getter) (QofEntity *, QofParam *);

gchar* qof_book_merge_param_as_string ( QofParam qtparam,
QofEntity qtEnt 

provides easy string access to parameter data for dialogue use

Uses the param_getfcn to retrieve the parameter value as a string, suitable for display in dialogues and user intervention output. Within a QofBookMerge context, only the parameters used in the merge are available, i.e. parameters where both param_getfcn and param_setfcn are not NULL.

Note that the object type description (a full text version of the object name) is also available to the dialogue as QofBookMergeRule::mergeLabel.

This allows the dialog to display the description of the object and all parameter data.

replace with qof_util_param_as_string

Definition at line 1064 of file qofbookmerge.c.

01066     {
01067         numeric_getter =
01068             (QofNumeric (*)(QofEntity *,
01069                 QofParam *)) qtparam->param_getfcn;
01070         param_numeric = numeric_getter (qtEnt, qtparam);
01071         param_string = g_strdup (qof_numeric_to_string (param_numeric));
01072         return param_string;
01073     }
01074     if (safe_strcmp (paramType, QOF_TYPE_GUID) == 0)
01075     {
01076         param_guid = qtparam->param_getfcn (qtEnt, qtparam);
01077         guid_to_string_buff (param_guid, param_sa);
01078         param_string = g_strdup (param_sa);
01079         return param_string;
01080     }
01081     if (safe_strcmp (paramType, QOF_TYPE_INT32) == 0)
01082     {
01083         int32_getter =
01084             (gint32 (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01085         param_i32 = int32_getter (qtEnt, qtparam);
01086         param_string = g_strdup_printf ("%d", param_i32);
01087         return param_string;
01088     }
01089     if (safe_strcmp (paramType, QOF_TYPE_INT64) == 0)
01090     {
01091         int64_getter =
01092             (gint64 (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01093         param_i64 = int64_getter (qtEnt, qtparam);
01094         param_string = g_strdup_printf ("%" G_GINT64_FORMAT, param_i64);
01095         return param_string;
01096     }
01097     if (safe_strcmp (paramType, QOF_TYPE_DOUBLE) == 0)
01098     {
01099         double_getter =
01100             (double (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01101         param_double = double_getter (qtEnt, qtparam);
01102         param_string = g_strdup_printf ("%f", param_double);
01103         return param_string;
01104     }
01105     if (safe_strcmp (paramType, QOF_TYPE_BOOLEAN) == 0)
01106     {
01107         boolean_getter =
01108             (gboolean (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01109         param_boolean = boolean_getter (qtEnt, qtparam);
01110         /* Boolean values need to be lowercase for QSF validation. */
01111         if (param_boolean == TRUE)
01112             param_string = g_strdup ("true");
01113         else
01114             param_string = g_strdup ("false");
01115         return param_string;
01116     }
01117     /* "kvp" contains repeating values, cannot be a single string for the frame. */
01118     if (safe_strcmp (paramType, QOF_TYPE_KVP) == 0)
01119         return param_string;
01120     if (safe_strcmp (paramType, QOF_TYPE_CHAR) == 0)
01121     {
01122         char_getter =
01123             (gchar (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01124         param_char = char_getter (qtEnt, qtparam);
01125         param_string = g_strdup_printf ("%c", param_char);
01126         return param_string;
01127     }
01128     return NULL;
01129 }
01131 QofBookMergeData *
01132 qof_book_merge_update_result (QofBookMergeData * mergeData,
01133     QofBookMergeResult tag)
01134 {
01135     QofBookMergeRule *resolved;
01137     g_return_val_if_fail ((mergeData != NULL), NULL);
01138     g_return_val_if_fail ((tag > 0), NULL);
01139     g_return_val_if_fail ((tag != MERGE_REPORT), NULL);
01140     resolved = mergeData->currentRule;
01141     g_return_val_if_fail ((resolved != NULL), NULL);
01142     if ((resolved->mergeAbsolute == TRUE) && (tag == MERGE_DUPLICATE))
01143         tag = MERGE_ABSOLUTE;
01144     if ((resolved->mergeAbsolute == TRUE) && (tag == MERGE_NEW))
01145         tag = MERGE_UPDATE;
01146     if ((resolved->mergeAbsolute == FALSE) && (tag == MERGE_ABSOLUTE))
01147         tag = MERGE_DUPLICATE;
01148     if ((resolved->mergeResult == MERGE_NEW) && (tag == MERGE_UPDATE))
01149         tag = MERGE_NEW;
01150     if (resolved->updated == FALSE)
01151         resolved->mergeResult = tag;
01152     resolved->updated = TRUE;
01153     if (tag >= MERGE_INVALID)
01154     {
01155         mergeData->abort = TRUE;
01156         mergeData->currentRule = resolved;
01157         return NULL;
01158     }
01159     mergeData->currentRule = resolved;
01160     return mergeData;
01161 }
01163 gint
01164 qof_book_merge_commit (QofBookMergeData * mergeData)
01165 {
01166     QofBookMergeRule *currentRule;
01167     GList *check, *node;
01169     g_return_val_if_fail (mergeData != NULL, -1);
01170     g_return_val_if_fail (mergeData->mergeList != NULL, -1);
01171     g_return_val_if_fail (mergeData->targetBook != NULL, -1);
01172     if (mergeData->abort == TRUE)
01173         return -1;
01174     check = g_list_copy (mergeData->mergeList);
01175     g_return_val_if_fail (check != NULL, -1);
01176     for (node = check; node != NULL; node = node->next)
01177     {
01178         currentRule = node->data;
01179         if (currentRule->mergeResult == MERGE_INVALID)
01180         {
01181             qof_book_merge_abort (mergeData);
01182             g_list_free (check);
01183             return (-2);

void qof_book_merge_rule_foreach ( QofBookMergeData mergeData,
QofBookMergeRuleForeachCB  callback,
QofBookMergeResult  mergeResult 

Dialogue Control Callback.

This function is designed to be used to iterate over all rules tagged with a specific QofBookMergeResult value.

callback external loop of type QofBookMergeRuleForeachCB
mergeResult QofBookMergeResult value to look up.
mergeData QofBookMergeData merge context.
Note : MERGE_NEW causes a new entity to be created in the target book at commit which is then assigned as the targetEnt of that rule. If mergeResult == MERGE_NEW, the rules returned by qof_book_merge_rule_foreach will have a NULL set for the targetEnt. This is because commit has not yet been called and no changes can be made to the target book. The calling process must handle the NULL targetEnt and NOT call any param_getfcn routines for the target entity. The import entity is available for display.

Uses qof_book_get_collection with the QofBookMergeRule::mergeType object type to return a collection of QofEntity entities from either the QofBookMergeData::mergeBook or QofBookMergeData::targetBook. Then uses qof_collection_lookup_entity to lookup the QofBookMergeRule::importEnt and again the qof_book_mergeRule::targetEnt to return the two specific entities.

Definition at line 1270 of file qofbookmerge.c.

QofBookMergeData* qof_book_merge_update_result ( QofBookMergeData mergeData,
QofBookMergeResult  tag 

called by dialogue callback to set the result of user intervention

Set any rule result to MERGE_INVALID to abort the import when qof_book_merge_commit is called, without changing the target book.

The calling process should make it absolutely clear that a merge operation cannot be undone and that a backup copy should always be available before a merge is initialised.

Recommended method: Only offer three options to the user per rule:

  1. Allow import data to be merged into target data
  2. Allow import data without an exact match to be added as new
    • change MERGE_REPORT to MERGE_NEW IF mergeAbsolute = FALSE
  3. Ignore import data and leave target data unchanged

Handle the required result changes in code: Check the value of qof_book_mergeRule::mergeAbsolute and use these principles:

To ignore entities tagged as:

To merge entities that are not pre-set to MERGE_NEW, set MERGE_UPDATE.
Attempting to merge an entity when the pre-set value was MERGE_NEW will force a change back to MERGE_NEW because no suitable target exists for the merge.

To add entities, check mergeAbsolute is FALSE and set MERGE_NEW.
An entity only be added if mergeAbsolute is FALSE. Attempting to add an entity when mergeAbsolute is TRUE will always force a MERGE_UPDATE.

It is not possible to update the same rule more than once.

  1. MERGE_NEW is reserved for new objects and is only pre-set if all parameters, including GUID, have already failed to match any relevant object. qof_book_mergeCommit will create new entities for all rules tagged as MERGE_NEW.
    • if mergeAbsolute is TRUE and the user wants to import the data, requests to set MERGE_NEW will be forced to MERGE_UPDATE because an entity with that GUID already exists in the target book.
    • if MERGE_NEW is pre-set, requests to change to MERGE_UPDATE will be ignored because a new entity is needed.
  2. MERGE_UPDATE is reserved for existing objects - qof_book_mergeCommit will require a matching entity to update and will force a change to back to MERGE_NEW if none is known to exist, using the principle above.
  3. MERGE_INVALID will cause an abort of the merge process.
  4. MERGE_UNDEF and MERGE_REPORT cannot be set - the entity result will be unchanged.
  5. MERGE_DUPLICATE and MERGE_ABSOLUTE are handled identically but are semantically different - QofBookMergeRule::mergeAbsolute is used to dictate which to set:
    • if mergeAbsolute is TRUE but MERGE_DUPLICATE is requested, force a change to MERGE_ABSOLUTE.
    • if mergeAbsolute is FALSE but MERGE_ABSOLUTE is requested, force a change to MERGE_DUPLICATE.

qof_book_merge_commit only commits entities tagged with MERGE_NEW and MERGE_UPDATE results.
Entities tagged with MERGE_ABSOLUTE and MERGE_DUPLICATE results are ignored.

The calling process must check the return value and call qof_book_merge_abort(mergeData) if non-zero.

mergeData the merge context, QofBookMergeData*
tag the result to attempt to set, QofBookMergeResult
-1 if supplied parameters are invalid or NULL, 0 on success.

Definition at line 1186 of file qofbookmerge.c.

01186         {
01187             g_list_free (check);
01188             return 1;
01189         }
01190     }
01191     g_list_free (check);
01192     qof_book_merge_commit_foreach (qof_book_merge_commit_rule_loop,
01193         MERGE_NEW, mergeData);
01194     qof_book_merge_commit_foreach (qof_book_merge_commit_rule_loop,
01195         MERGE_UPDATE, mergeData);
01196     /* Placeholder for QofObject merge_helper_cb - all objects
01197        and all parameters set */
01198     while (mergeData->mergeList != NULL)
01199     {
01200         currentRule = mergeData->mergeList->data;
01201         g_slist_free (currentRule->mergeParam);
01202         g_slist_free (currentRule->linkedEntList);
01203         mergeData->mergeList = g_list_next (mergeData->mergeList);
01204     }
01205     g_list_free (mergeData->mergeList);
01206     g_slist_free (mergeData->mergeObjectParams);
01207     g_slist_free (mergeData->targetList);
01208     if (mergeData->orphan_list != NULL)
01209         g_slist_free (mergeData->orphan_list);
01210     g_hash_table_destroy (mergeData->target_table);
01211     g_free (mergeData);
01212     return 0;
01213 }
01215 void

Generated on Thu Jan 31 22:50:27 2008 for QOF by  doxygen 1.5.4