Migration » History » Revision 7

« Previous | Revision 7/10 (diff) | Next »
Sven Vanpoucke, 12/08/2010 15:33


This page gives an overview of the general architecture of the migration and the available migrations from Software XX to Chamilo.


The general idea is that it should be as easy as possible to hook a new platform in the migration system and just write code to actually convert the data. To accomplish this we have a generic framework which exists out of 2 parts. The migration starter and the migration platform integration. Each platform exists out of a number of blocks of data that needs to be migrated. Every block exists out of one or more data classes that need to be converted to chamilo 2. The architecture is based on a record by record migration because the architecture of chamilo 2.0 is so different to those of other platforms that in most of the cases this is the only solution to migrate the data.

The migration starter

The migration for a certain platform can be started through two methods

  • The wizard: A step by step wizard for each big block of data that needs to be migrated. After each step a user needs to click on the next button. Through this way a user can manually correct mistakes that were made during the migration for each block.
  • The command line: A sequential script that runs through every step without the need for user interaction.

The migration settings

1. The following settings are stored in the database
  • The selected platform
  • The location of the old platform
  • Whether to move physical files or to copy them
  • Whether to include deleted files or not (if the platform supports deleted files)
  • The selected blocks + whether they are migrated or not
2. Usage
  • When using the wizard, the settings are asked from the user through a form.
  • When using the command line interface, a user must make sure that there is a file included with the name settings.php

3. The settings are validated before the execution of the migration

The migration flow

The only difference between the wizard and the command line interface is the interface and the way it ask for settings, for all the other tasks it relies on a generic framework.

1. The system loads the settings and validates them.
  • To validate the settings we use a special class called MigrationProperties.
  • Each platform needs an extension of the MigrationProperties to validate the settings.
  • This class is also used to determine the order of the migration blocks.
2. For each selected migration block the function migrate() is called.
  • Each platform migration block should extend the general MigrationBlock class
  • The MigrationBlock class has a few methods to migrate the data so we don't need to reproduce this for each and every block. The migration of each block is very similar:
    • Prepare the migration: some logging, enabling the timer, prerequisites check to see if this block may be executed.
    • Migrate the actual data (See part 3)
    • Finish the migration: additional logging, stopping the timer.
  • Because this is always the same process, an integration of MigrationBlock only needs to integrate 3 very small methods
    • get_prerequisites(); returns an array of the blocks that need to be migrated before this block can be migrated.
    • get_data_classes(); returns an array of the data classes that are connected
    • get_block_name(); returns the name of the current migration block
3. Migrate the actual data
  • Each block exists out of several data classes which represent a table in the old platform. These data classes are an extension of MigrationDataClass
  • The general MigrationDataClass has a few generic methods for managing the following tables
    • Id references: keep track of the ids between the old and the new system so we can use them in other tables with a foreign key
    • Failed elements: a list with all the records that were not migrated
    • File recoveries: a list to keep track on the new locations of the files, if we select the move setting we should know where the files are to restore them manually if necessary.
  • The extension of the MigrationDataClass needs to integrate the following methods:
    • is_valid();
    • convert();
    • get_table_name();
    • get_class_name();
    • get_data_manager();
  • For each data class a dummy object is created
  • The data manager from the data class is asked from the dummy object and the method retrieve_all_objects() is called
    • Each platform needs to provide his own datamanager integration
  • The objects for each data classes are retrieved in chunks of 1000 to keep the performance as fast as possible
  • The data manager uses the table and class name to retrieve and map the objects to the correct data class
  • The object is validated by calling the is_valid() function of the dataclass
  • The object is converted by calling the convert() function of the dataclass

The integration

This is a overview of the minimal amount of classes and functions you need to get a working platform

  • A new folder with the name of the platform needs to be created in /migration/lib/platform/yourplatform/
    • Data classes are stored in /migration/lib/platform/yourplatform/data_class/
    • Migration blocks are stored in /migration/lib/platform/yourplatform/migration_block/
  • A data manager should be created as an extension of the platform migration data manager and should inherit the following methods
    • function retrieve_all_objects($data_class, $offset, $count);
    • function count_all_objects($data_class);
  • We need an extension on the migration properties class with the following methods
    • function validate_settings($settings, $selected_blocks, $migrated_blocks);
    • function get_migration_blocks();
  • We need at least one migration block with the following methods (extension of MigrationBlock)
    • function get_prerequisites();
    • function get_block_name();
    • function get_data_classes();
  • We need at least one data class with the following methods (extension of MigrationDataClass)
    • function is_valid();
    • function convert_data();
    • static function get_table_name();
    • static function get_class_name();



  • Refactoring of commmand line migration

Updated by Sven Vanpoucke over 12 years ago · 7 revisions