Site Name

Kees Schollaart

Office 365 Migration Api using C#

10 Mar 2016

This blogpost describes how to use the Office 365 Migration Api using your own C# code.

The Office365 Migration Api enables you to upload lots of documents into SharePoint Online without being throttled.

What is the Migration Api?

I assume that you know what the Migration Api is and when to use it.

Steven Pogrebivsky has written an blogpost on this subject which I recommend to read first.

Also Benjamin Niaulin of Share Gate has written an blogpost, off course promoting their software which can help you with your migration.

In short, it boils down to the following flow:

The Microsoft PowerShell Scripts

Microsoft provides some PowerShell cmdlets to help you migrate data using the Migration Api. Using the SharePoint Online Management Shell Windows PowerShell environment you can use a set of Migration Api related operations.

What this cmdlets basically do, is:

This PowerShell scripts are easy to use but do not bring a lot of flexibility. For example:

Undocumented features

Some technical parts of this migration process are documented, like Site.CreateMigrationJob().

Other parts are not (well) documented, for example the structure of the manifest-package and the reporting-queue messageformat.

There’s also no example project on how to do advanced migration scenario’s and how to use the Api’s yourself.

Migration using C#

All the things the PowerShell scripts do, we can do ourselfs using .NET. I created a Proof Of Concept C# Console Application doing.

Lets check some code, the 4 steps refer to the image in the first paragraph:

class Program
{
	static void Main(string[] args)
	{
		var migrationApiDemo = new MigrationApiDemo();

		// Step 1, Create and upload some test-files to Azure Blob Storage
		migrationApiDemo.ProvisionTestFiles();

		// Step 2, Create and upload Manifest Package to Azure Blob Storage
		migrationApiDemo.CreateAndUploadMigrationPackage();

		// Step 3, Start the Migration Job using SharePoint Online Clientside Object Model (CSOM)
		var jobId = migrationApiDemo.StartMigrationJob();

		// Step 4, Monitor the Reporting Queue, persist messages/logs and wait for the job to complete
		migrationApiDemo.MonitorMigrationApiQueue(jobId).Wait();

		Console.ReadLine();
	}
}

The test files are provisioned to Azure but are generated in-memory:

public  ICollection<SourceFile> ProvisionAndGetFiles()
{
	var testfiles = new[]
	{
		new SourceFile
		{
			Filename = "test.txt",
			LastModified = DateTime.Now,
			Contents = Encoding.UTF8.GetBytes("Hi, this is a test text-file"),
			Title = "Title of file 1"
		},
		new SourceFile
		{
			Filename = "test2.txt",
			LastModified = DateTime.Now.AddDays(-1),
			Contents = Encoding.UTF8.GetBytes("Tesfile2"),
			Title = "Second title"
		}
	};

	_log.Debug("Removing all existing files on test blob...");
	_azureBlob.RemoveAllFiles();

	_log.Debug($"Uploading {testfiles.Length} files on test blob...");
	foreach (var testfile in testfiles)
		_azureBlob.UploadFile(testfile.Filename, testfile.Contents);

	return testfiles;
}

In a real life scenario this needs to be replaced with code getting the files from your source location, a network-drive, a legacy system or …? For now it’s important to know that the source-files can he loaded from any source but you have to write your own reader.

Now we need to create a manifest-package

public IEnumerable<MigrationPackageFile> GetManifestPackageFiles(IEnumerable<SourceFile> sourceFiles)
{
	Log.Debug("Generating manifest package");
	return = new[]
	{
		GetExportSettingsXml(),
		GetLookupListMapXml(),
		GetManifestXml(sourceFiles),
		GetRequirementsXml(),
		GetRootObjectMapXml(),
		GetSystemDataXml(),
		GetUserGroupXml(),
		GetViewFormsListXml()
	};
}

The package contains 8 XML files all of them are very small/static beside the ‘Manifest.xml’ file, this file contains all the references to our source files.

Now that we have to Azure Blob Containers we can start the Migration Job:

/// <returns>Job Id</returns>
public Guid StartMigrationJob()
{
	var sourceFileContainerUrl = _testDataProvider.GetBlobUri();
	var manifestContainerUrl = _blobContainingManifestFiles.GetUri(
		SharedAccessBlobPermissions.Read 
		| SharedAccessBlobPermissions.Write 
		| SharedAccessBlobPermissions.List);

	var azureQueueReportUrl = _migrationApiQueue.GetUri(
		SharedAccessQueuePermissions.Read 
		| SharedAccessQueuePermissions.Add 
		| SharedAccessQueuePermissions.Update 
		| SharedAccessQueuePermissions.ProcessMessages);

	return _target.StartMigrationJob(sourceFileContainerUrl, manifestContainerUrl, azureQueueReportUrl);
}

Office 365 will start this migration job and report any update (progress, fail of success) to the reporting-queue

public async Task MonitorMigrationApiQueue(Guid jobId)
{
	while (true)
	{
		var message = await _migrationApiQueue.GetMessageAsync<UpdateMessage>();
		if (message == null)
		{
			await Task.Delay(TimeSpan.FromSeconds(1));
			continue;
		}

		switch (message.Event)
		{
			case "JobEnd":
				Log.Info($"Migration Job Ended {message.FilesCreated:0.} files created, {message.TotalErrors:0.} errors.!");
				DownloadAndPersistLogFiles(jobId); // save log files to disk
				Console.WriteLine("Press ctrl+c to exit");
				return;
			case "JobStart":
				Log.Info("Migration Job Started!");
				break;
			case "JobProgress":
				Log.Debug($"Migration Job in progress, {message.FilesCreated:0.} files created, {message.TotalErrors:0.} errors.");
				break;
			case "JobQueued":
				Log.Info("Migration Job Queued...");
				break;
			case "JobWarning":
				Log.Warn($"Migration Job warning {message.Message}");
				break;
			case "JobError":
				Log.Error($"Migration Job error {message.Message}");
				break;
			default:
				Log.Warn($"Unknown Job Status: {message.Event}, message {message.Message}");
				break;

		}
	}
}

How to use this code

You first need:

After setting these variables in the appsettings-section of the App.config file you’re good to go, just run the application. After running this application succesfully, two test-files will be uploaded in the configured SharePoint environment. Log files will be stored in the application folder using log4net’s FileAppender.

This application is just to show you how to migrate files using .NET with a minimal set of code. It cannot be used as-is in real world scenario’s!

When you want to use this code, the first thing to change will be the origin of the files from in-memory-generation to (for example) a folder.

More information

Feel free to use this code for own/commercial use. For questions or information, contact me!