I am working on notes for a draft that should become a book about migration processes built with Drupal and its Migrate API. It is expected to be released in June 2020, and collecting, testing, and shaping the content has become quite a large piece of work.

There are still a few months left before launch, so, to preserve a little mental health and give these tasks some partial meaning, I thought I could publish a few small posts here, based on those working notes.

That way I can share something useful alongside the complementary notes and, if COVID-19 gets me before the book comes out, at least I will have shared something first (I guess).

Well, what do I want to talk about in this post? I would like to make a list of Drupal modules related to migration processes, available as contrib modules that can add functionality to a migration. This article is only a lightweight set of basic resources (I swear).

This article was originally published in https://davidjguru.github.io Picture from Unsplash, user Nils Nedel, @nilsnedel

Table of Contents

1- Introduction 2- Basic Resources - Core Modules 3- Other Basic Resources - Contrib Modules 4- Extra Resources - Contrib Modules for Plugins 5- Migration Runners - Contrib Modules Drush-Related 6- Authors you should know 7- :wq!

This article is part of a series of posts about Drupal Migrations:

1- Drupal Migrations (I): Basic Resources

• Core Modules for Migrations in Drupal.
• Contributed Modules for Migrations in Drupal.
• Contributed Modules for Plugins in Migrations.
• Contributed Modules Drush - Related.
• Authors you should know.

2- Drupal Migrations (II): Examples

• Introduction: Migration as code or as configuration.
• Arrangements: Migrating embedded data and CSV file.
• Approaches for the Migrations.
• Executing Migrations.
• Key Concepts for the Migrations.
• Resources for the Migrations.

3- Drupal Migrations (III): Migrating from Google Spreadsheet

• Remember ETL processes.
• Exposing Data Through Google Spreadsheet.
• Special Properties from the JSON transformation.
• Characteristics of the Migration.
• Custom Module for Migration.

4- Drupal Migrations (IV): Debugging Migrations First Part

• Basic Debugging of a Drupal Migration: files, database tables and configuration objects.
• Average Debugging with Migrate Devel Module.

5- Drupal Migrations (V): Debugging Migrations-II

• Introduction to Xdebug.
• Xdebug for DDEV.
• Steps for setting up the IDE.
• Errors launching Drush in containers.

1- Introduction

It is not easy to talk about migrations in general and, of course, it is not easy in the context of Drupal either. To run migrations, you need solid knowledge of the technology, the data models at both the source and destination, some experience with ETL processes, and a certain amount of Drupal plugin know-how. Migrations make heavy use of Drupal-style plugins.

In any case, since the topic is extensive and my time is now short, I thought of this article as a summary catalogue (for quick consumption) of tools and basic resources for working with migrations.

2- Basic Resources - Core Modules

• Migrate: The Mainframe for migrations, the migrate, that provides the base API for migrating in Drupal.

• Migrate Drupal: migrate_drupal. Focused on upgrades from Drupal 6 or Drupal 7. Allow reading of configuration entities in Drupal 8.

• Migrate Drupal: Multilingual migrate_drupal_multilingual. Experimental module in core (¿?). See: https://www.drupal.org/node/2959712.

• Migrate Drupal UI: migrate_drupal_ui. Interface for upgrading.

3- Other Basic Resources - Contrib Modules

• Migrate Plus: [migrate_plus](https://www.drupal .org/project/migrate_plus). Migrate Plus is an essential contrib module which extends the features and capabilities of the Migrate core module with a lot of plugins and extensions.

• Migrate Tools: migrate_tools. Another essential resource: provides a lot of Drush commands for running and managing Migrations.

• Migrate Status: migrate_status. This little contrib module lets you get a feedback about a migration process. Do you need to know if a migration is running? this module gives you a service that you can call to check the migration.

• Migrate Files: migrate_files. It’s such an interesting set of process plugins that you will want to move files and images with it.

• Migrate Commerce: commerce_migrate. General-purpose framework that extends to the main Migrate module from Drupal Core, for moving data in a Drupal Commerce scenario.

4- Extra Resources - Contrib Modules for Plugins

In Drupal migration processes, we’ll use different resources to process the ETL migration plan. One of these basic resources, as I mentioned in the introduction, is Drupal Plugins, which requires some knowledge and practice. In a migration scenario, plugins help us read information from the E: Source (Source Plugins), perform the T: Processing (Process Plugins), and save data in the L: Destination (Destination Plugins). Putting these three parts together usually gives us a working migration process.

Many modules of the core already bring their own Plugins to facilitate migration processes (as the user module). So let’s review some migration plugins packaged in contributed modules.

Source Plugins Migrate Source Plugin

• Migrate Source CSV: migrate_source_csv. Contrib Module for migrating data to Drupal 8 from a classic, simple CSV file.

• Migrate Source SQL: custom_sql_migrate_source_plugin. As a peculiarity of database plugins, this module lets you include SQL queries directly in the .yml file that describes the migration. Those queries will be executed against the source database.

• Migrate Source YAML: migrate_source_yaml. It’s just a simple tool for migrating content from YAML files.

Processing Plugins Migrate Process Plugins

• Migrate Process Geofield: [geofield](https://www.drupal .org/project/geofield). The contrib Geofield module comes with a custom process plugin for migrations. See Process Plugin in Geofield.

• Migrate Process XML: migrate_process_xml. Provides process plugins for xpath and xvalue.

• Migrate HTML to Paragraphs: migrate_html_to_paragraphs. Helps to transform HTML from a migration Source in a Paragraph item (managed by a Destination Plugin).

Destination Plugins: Migrate Destination Plugins & Examples

What kind of Drupal entities will be created in the migrating process? content entities? configuration entities? Take a look.

• Migrate Destination CSV: d8migrate. It’s a light custom module created by @jonathanfranks.

• Migrate Destination Config: Class Config.php. Offers a plugin for config migration.

• Migrate Destination Block: Class EntityBlock.php. Just like and example about the resources that every element can offers in a migration scene, in case of moving Block Entities (are Config Entities) see the PHP classes included in its own module for migrating (Source, Process and Destination).

5- Migrations Runners - Contrib Modules Drush-Related

• Migrate Scheduler: migrate_scheduler. This module offers integration with the Drupal Cron API to execute migrations under predefined schedules.

• Migrate Run: migrate_run. Drush commands for running migrations in a lightweight mode. More lean than Migrate Tools but not offer support for migrations groups. It also doesn’t depend on the Migrate Plus module. It’s just like a little maverick.

• Migrate Devel: migrate_devel. Provides Drush options to show debug info while executing migrations. Also provides the ‘debug’ process plugin. Here you can see an article about it. By the way, the Drush 9 compatibility is still unresolved, but a patch seems to be available.

6- Authors you should know

• Mauricio Dinarte: Mauricio is a developer, consultant, trainer and owner of his own business https://agaric.coop, wrote what is probably the mandatory reading guide for all people who want to learn how to migrate on Drupal: 31 Days of Drupal Migrations, a set of 31 articles published in https://understanddrupal.com/migrations with the most important aspects…examples, exercises, descriptions…to understand the whole internal world of migrations inside Drupal.

An essential training material. In addition, his company’s website, under the tag “migrate” also hosts many very good articles about migration topics: https://agaric.coop/tags/migrate.

Some examples from Mauricio Dinarte:

1. Introduction to paragraphs migrations in Drupal:

• https://agaric.coop/blog/introduction-paragraphs-migrations-drupal.

2. Using migration groups to share configuration among Drupal migrations:

• https://agaric.coop/blog/using-migration-groups-share-configuration-among-drupal-migrations

3. What is the difference between migration tags and migration groups in Drupal?

• https://agaric.coop/blog/what-difference-between-migration-tags-and-migration-groups-drupal.

His profile in Drupal.org: https://www.drupal.org/u/dinarcon.

• Tess Flynn: I heard about Tess Flynn reading articles by Mauricio Dinarte. That’s how I met this expert developer, speaker and communicator of the Drupal community. On her website https://deninet.com I found content of a different nature, but above all, a series of very interesting articles about migrations under the tag “drupal-migration”: https://deninet.com/tag/drupal-migration.

  Along the way I also discovered that it has several contrib modules related to Migrations and Processing Plugins.

Some examples from Tess Flynn:

1. Migrate Process URL: Provides Process Plugin to migrate link fields.

• https://www.drupal.org/project/migrate_process_url.

2. Migrate Process Vardump: Helping to debugging migrations.

• https://www.drupal.org/project/migrate_process_vardump.

3. Many Process Plugins:

• Migrate Process Array, Migrate Process Skip, Migrate Process Trim.

Her profile in Drupal.org: https://www.drupal.org/u/socketwench.

• Danny Sipos: Author of a well-considered Bible of Drupal -Drupal 8 Module Development, nowadays in its second edition- and writing also in his website https://www.webomelette.com/. Lecturer, trainer and regular attendee at various international Drupal events, has also written some interesting articles about Drupal migrations that you should read.

Some examples from Danny Sipos:

1. Your first Drupal 8 Migration:

• https://www.sitepoint.com/your-first-drupal-8-migration.

2. Dynamic migrations using “templates” in Drupal 8:

• https://www.webomelette.com/dynamic-migrations-using-templates-drupal-8.

3. Quickly generate the headers for the CSV migrate source plugin using Drush:

• https://www.webomelette.com/quickly-generate-headers-csv-migrate-source-plugin-using-drush.

His profile in Drupal.org: https://www.drupal.org/u/upchuk.

7- :wq!

Recommended song