In this post we will take a detailed look at the Migration Manager, it’s capabilities, limitations and how to create pre-scan and migration tasks for the first time.
- Current limitations of Migration Manager
- Setup considerations
- File and folder permissions
- Demo: create a scan only task
- Demo: create migration task
So the good news is, if you have a Microsoft 365 subscription then you also have the Migration Manager too! The Migration Manager went into general availability in June 2020 for all customers as part of the SharePoint Online offering.
The Migration Manager is a part of the SharePoint admin center and is really simple and easy to get set up and begin using. Microsoft have this handy image to demonstrate how easy the process is:
Currently, the Migration Manager only supports file share migrations, this means that if you are planning to migrate any other content you hold, for example: on-premise SharePoint content or cloud based content you would need to use a separate tool.
Microsoft also offer several other tools that can facilitate the migration of that content too. Microsoft provide a table which recommends which tool to use here, but I’ve whittled them down to the main options below:
- Migration Manager: used for network and local file share migrations, easy to set up via the SharePoint admin center
- SharePoint Migration Tool: used for SharePoint Server 2010, 2013 & 2016 (Public Preview), network & local file shares, requires some prerequisites configuring before installation
- Mover: Service for cloud to cloud migration (Dropbox, Google Drive etc.), easy to set up via web platform
Another limitation of the Migration Manager is that it currently does not support third party multi-factor authentication.
So as the above breakdown of each Microsoft migration tool suggests, the Migration Manager is easy to get going. The key consideration when using this tool to migrate file share content is around the volume of data you are migrating.
If you are using the Microsoft FastTrack service, they will recommend that you set up a number of migration machines relative to the total number of files you are migrating.
The recommendation they offer for scoping this is below:
|Total file count||Migration machines required|
|Less than 200,000||2|
|Between 200,000 and 500,000||3|
|Between 500,000 and 1,000,000||6|
|More than 1,000,000||To be discussed with FastTrack|
Once you decide how many migration machines will be required, you will then need to setup the migration agents on each machine, check the prerequisites and required endpoints have been reviewed and met, and also have the relevant accounts being used as part of the migration process given access to the file share content and SharePoint admin roles.
File and folder permissions
When prepping for your file share migration, another consideration will be the permissions of a file once it is migrated. For most organisations it comes down to the following scenarios:
1. We want all file/ folder permissions preserved when they migrate to SharePoint Online
2. We only want specific file/ folder permissions mapped into SharePoint Online
3. We don’t want any of the permissions migrating, we want to start again!
Yes, I know the third option is highly unlikely, but I’ll include it anyway. When using the Migration Manager, the syncronization between your Active Directory on-premise and Azure Active Directory is key to how permissions migrate across.
It is important to note that if your organisation uses security groups in Active Directory to manage permissions for their file shares, they must be syncronized with Azure Active Directory in order for the user permissions to map across like-for-like.
If not, then you will be required to create a user mapping file to map user permissions for the relevant files or folders.
The Migration Manager also takes into account the same permissions conditions and results as the SharePoint Migration Tool. This table lists all the conditions and the corresponding results.
Demo: create a scan only task
Before you run any sort of migration task you will first want to get a handle on the current situation of your file share content, what areas will migrate easily and which will require remediation.
Both the scan and migration actions sit within Tasks in the Migration Manager and the initial setup for both actions is the same:
- Press add task > under Method select the default single source and destination (unless you wish to scan multiple sources)
- Under Source, enter the file share path that you wish to scan using the correct format
- Under Destination, leave the SharePoint site URL and location as the default as we are only performing a scan
- Under Settings, give your task a name > under common settings ensure perform scan only is checked
- Press Run Now
Demo: create a migration task
So once you have ran a pre-scan of your file share source and you are happy with the results, it is now time to create a migration task! This again is very similar to how we approached creating the pre-scan task but here are the steps:
- Press add task > under Method select the default single source and destination (unless you wish to scan multiple sources)
- If you wish to do a bulk upload, you will need to provide a CSV or JSON file which is well documented here
- Under Source > enter the file share path that you wish to migrate using the correct format
- Under Destination > select the application where you are migrating the data to. Press next
- Enter the URL of the location, then select the library or channel you wish to migrate to
- Under Settings > enter a name for you task and configure your migration task based on the following check boxes:
- Preserve file share permissions
- Migrate hidden files
- Migrate files created/modified after specified date
- Do not migrate files with specific extensions
- Migrate files and folders with invalid characters
- Migrate OneNote folder as OneNote notebook
- Azure Active Directory lookup
- User mapping file
- Automatically rerun failed tasks up to 4 times
- Press Run Now
Whether you have performed a pre-scan or migration in Migration Manager, each task once complete will provide a “task report” zip folder that is available to download. Microsoft break the reports down into Summary, Task level and Performance reports, but in fact all reports are included as part of the task report download.
The Microsoft breakdown of each report is very thorough, so I won’t bother adding any more detail, however I will highlight the reports that I found useful when using the Migration Manager:
- Summary Report: contains a single row of data that gives the total picture; including total size, number of files migrated, duration
- Item Failure Report: contains any errors found resulting in a file being unable or failing to migrate
- Item Report R1: detailed report with data on each file within a task. If large number of files migrated, split into separate, sequential reports (R1, R2, R3 etc.)
An interesting aside I did notice was on the task details pane for a pre-scan I ran it showed under files scanned with issues as 0, but within the task report ZIP there was, in fact several files and folders listed as having failed due to a variety of reasons. So I’m not really sure how accurate the files scanned with issues is, or what it actually defines as issues.
Microsoft documentation includes a page on troubleshooting Migration Manager issues and errors here, but typically during a recent migration I experiencing issues that weren’t including in the above.
- Error: the source file share does not exist
- Error: failed to duplicate token file due to error ‘There is not enough space on the disk’
Error: the source file share does not exist (but it does)
This error I received recently during a home drive – OneDrive migration I was conducting. I was using windows virtual machines with the migration agents installed previously and had logged into the machine and SharePoint admin center as accounts with the relevant permissions and roles as described here by Microsoft.
When I set up my migration tasks as demo’ed above they were instantly failing. The only errors I would receive were these:
Microsoft have this listed as an agent error message – which was the clue, but their action was to:
Make sure the source file share is an existing network file share. Confirm that the Windows account associated with the agent has read permissions to the file share you want to migrate.Troubleshoot Migration Manager issues and errors
What I found was in my case, even if the migration agents were showing as enabled, because I had originally installed and configured the agents some time ago (1-2 months prior), they need to be repaired and re-authenticated:
- Download the migration agent setup file
- Microsoft documentation hasn’t yet been updated, but to get to the agent download you need to open Migration Manager > Press Agents > + Add
- Run the agent setup file
- Press continue to reinstall the agent
- The installer will then begin the installation prerequisites
- Enter account credentials for the SharePoint Admin, then Windows account
- The installer will authenticate, then prompt you to test permissions with a file share, press OK to close
Now, when I re-ran my migration task with my agents repaired and re-authenticated, the tasks completed successfully.
Error: failed to duplicate token file due to error ‘There is not enough space on the disk’
This error occurred for me during a pre-scan of file servers locations. After kicking the scans off, I returned to find many of them failed. When looking at the task error details from the logs I found the following:
Error, Failed to duplicate token file due to error ‘There is not enough space on the disk.
This error was pretty much what it says on the tin, the VM had ran out of disk space. As referenced before, looking through Microsoft’s issues and errors post didn’t prove helpful as the error above is not listed.
So, through investigation of my own I found the following:
- There are a bunch of MTHost text files that appear to be log files of actions completed when tasks are ran within the Migration Manager.
- There is a Migration folder, within a subsequent MigrationTool folder, another folder with your tenant name will be found. In here will be folders with names beginning “WF_” which represents each task created within the Migration Manager.
- Each “WF_” folder contains a Log and Report folder, with the report folder containing all the migration reports we detailed earlier in this post.
- There is also a MigrationToolStorage folder which also contains a mirrored “WF_” series of folders related to tasks created within Migration Manager.
What’s crucial to understand here is you are able to delete the “WF_” folders from these locations, without it affecting the tasks in Migration Manager, or the ability to download the corresponding task reports.
I’ve tested deleted all the “WF_” folders from both the above folders and then refreshing and trying to download task reports for the tasks in the Migration Manager – and they download perfectly!
What I would question is what is the point of these “WF_” files downloading locally in the first place if they are also stored somewhere in the M365 cloud as well!
You cannot migrate to a deeper destination than first sub-folder
When migrating files from source to destination, in some cases you may want to migrate specific data that sits within a larger, nested folder structure. For example, lets say you have a folder that sits in a file share like this:
You only want to migrate folder b and nothing else except its associated folder structure. In Migration Manager, you are only able to select the first, top-level folder from a library. This applies to OneDrive, SharePoint or Teams and doesn’t offer any other sub-folders to be selected via the tool itself.
If you try to create migration tasks via CSV and include the sub-folders as additional columns in the spreadsheet, the Migration Manager ignores them and just migrates whatever is in the destination to the source library.
I would say generally this isn’t an issue as you can either skip the files you do not want to re-migrate and just set the source/ destination to be at the top-levels. This specific scenario came about because I was trying to remediate some files and folders that failed a prior migration, and once remediated migrate those folders only via Migration Manager into the corresponding folder structure in SharePoint. I would say this is an issue if you are doing what I described and don’t want to have to run another large migration task, even if it is skipping most of the files as the task will still take a while to complete.
RE: You cannot migrate to a deeper destination than first sub-folder
Using the SharePoint Migration Tool, it is possible to copy to a sub level, simply create the task as a teams or sharepoint (they both create the same job with only the ‘TargetListRelativePath’ being slightly different – teams, can select first sub folder, share point can only select the top level group for the output json), then save as json, edit like the below and reload into the tool and run!
Hi Joshua, thanks for commenting, I still think it’s bonkers this isn’t a feature within the UI but good to know you can workaround it
Hi! Much of what you went through, I have come across myself also. Just on the log files, do you know how much storage these generally take up or a percentage of the task perhaps? I am doing a big migration and I want to be able to leave it overnight without deleting log files to keep my space alive. I just need to know how much space I need to allocate.
Hi Michelle, thanks for commenting!
For me, it did depend on the size of the migrations we ran, but I saw log files build up to around 2GB following migrations. As an example, we ran a home drive – OneDrive migration for 1600 users and ran batches of 150 over a weekend – no more than a couple hundred GB of data.
I built all our VMs to spec, following this guide here: https://docs.microsoft.com/en-us/sharepointmigration/mm-prerequisites
I’ve also wrote a separate post about the homedrive – OneDrive migration which might be of interest: https://sharepointstuff.com/2021/01/13/tales-of-a-microsoft-365-migration-onedrive/
Yes, I performed a test migration using local shares with DACLs referring to local groups. I created a permissions matrix as documented by MS which mapped SERVER\Local Group to firstname.lastname@example.org (mail-enabled security group)
As I said, these must be MAIL-ENABLED groups in Office so that you can map to a full UPN (presumably), just like a normal user map. I tried the permissions file with a basic security group with a common name and it indeed failed. Creating mail-enabled groups instead was the only way I was able to get a group-to-group mapping to work. Luckily this is only necessary for environments without a preexisting corporate directory, otherwise you would simply sync your security groups over with AAD Connect and let Azure AD do the rest. Thanks,
“It is important to note that if your organisation uses security groups in Active Directory to manage permissions for their file shares, they must be syncronized with Azure Active Directory in order for the user permissions to map across like-for-like.”
^^^ I feel like I am able to work around this by using mail -enabled security groups in SPO. This was tested with a mapping file referencing local user groups mapped to the mail -enabled groups. What do you think?
Hi @NMF, thanks for the comment. I know you can use mapping files when running migration tasks in the migration manager as references here: https://docs.microsoft.com/en-us/sharepointmigration/mm-settings
Have you tested this in the context of a migration?