In this post we will look at how the image tags managed metadata column creates issues for ShareGate migrations and how to resolve them.
Intro
Image tags were introduced in July 2021 and are automatically added by Microsoft AI to improve search. When images get uploaded into document libraries, SharePoint will analyse them upon upload and tag them based on a set of 37 basic tags. SharePoint will also re-analyse each time an image is updated and update the image tags.
The image tags column is a managed metadata column that isn’t immediately available within a document library. you have to upload an image first, before the image tag column appears – which can take several hours.
The problem
When migrating files at scale using migration tools such as ShareGate desktop, you will start to see errors in your migration reports related to the image tags column. In the example below, the details of each failed item showed the same or similar errors:
“Property Image Tags: The following values are unavailable: ‘outdoor, sign, text’. Please specify another value.”
ShareGate migration errors
Example migration report showing the image tag error messages.
The solution
The way I resolved this issue was to change an option within the migration task itself to ignore the image tag column managed metadata values. To do this I did the following in ShareGate:
In ShareGate desktop > start a copy content migration
Select your source and destination sites
In the copy content screen > select Options
Select Metadata > then expand Document
You will see the image tags column here > change set mapped value to ignore
Tick apply for all version history
Set your copy content migration task to ignore the image tasks column metadata.
Wow it was over a year ago I wrote the first part in what was intended to be a series documenting my experiences of migrating to Microsoft 365! Better late than never I guess…well in this post we will explore the process of migrating “home” drives or personal user data to OneDrive in Microsoft 365.
The key tenants to the migration approach are fairly commonplace in most organisations, with four distinct phases:
Exchange email migration
Home drive migration
File share migration
SharePoint migration
The plan was migrate all users home drives from on-premise file servers into OneDrive, making the home drives read-only in the process and then finally removing the mapping once all migrations were completed.
Discovery & analysis
Discovery of home drive data
So naturally to start to flesh out how we would migrate all users home drive data to OneDrive we needed to do a little digging to understand our estate and figure out:
(L-R) How many home drives are there? What is the total volume size of all the home drive data? Do all the users have OneDrive to move to?
Also during this discovery phase we began working with Microsoft FastTrack. If you don’t know much about the service, it is something Microsoft offer to assist customers with their migration projects and best of all it’s FREE!
So once you are set up with the FastTrack service, you still need to do all the discovery and analysis as described above before you can provide the migration tracks to them, in order for them to schedule and run the migrations.
Fast Track also require that you create virtual machines (VM’s) to run the migrations. They calculate the number of VM’s you need based on:
The total file count of your file share data or SharePoint environment (if supported)
The total volume of your home drive data
We wanted to multi-purpose our VM’s to also run migrations for other parts of the M365 project. So we had to take the metrics above and provide an average based on what little we knew at the time in order to create enough VM’s (we ended up creating six).
In order to find answers to the above questions, I used the SharePoint Migration Manager. The Migration Manager can be used to create scan-only tasks that will output reports on all your home drive sources and basically enable you to begin scheduling the migrations.
The analysis part of the migration included looking at how we could effectively answer the three main questions above. For the first two questions; how many home drives are there, and how much data do they contain – running the scan-only tasks answered those questions.
How many home drives are there? and how much data do they hold?
To summarise what I did, I created tasks for each top-level home drive folder on each file server that contained the home drive data (for example: \\yourserver\home) and collated the answers to the questions above in one, master spreadsheet.
Example spreadsheet that I created to collate all the home drive info.
The scan-only reports included much more granular information about each home drive within the top-level file share, which I downloaded from Migration Manager into a working folder for further analysis if required.
I also found that the summary reports, or key information provided by Migration Manager around failures to be a bit misleading. For example, if the accounts used to read the home drives don’t have access to a particular folder, that doesn’t flag as a failure. You have to open the ItemFailureReport from each task to find out if there were any issues with access or files/ folders themselves.
Coming from on-premises? Be prepared to compare user data
As mentioned earlier, you are expected to provide FastTrack with a list of home drive locations and OneDrive URL destinations in a migration track in order for them to begin the migration.
Because our estate still has an on-premise Active Directory (AD) that didn’t have all users synced, we had to do some checks to make sure all the users from our on-premise AD had O365 accounts and also had OneDrive sites to migrate to.
We used an export of AD on-premise data, compared against Azure AD data and a OneDrive usage report that you can download from here. We used this to flag any accounts that either didn’t have Microsoft 365 accounts, or OneDrive sites, or both.
Use the usage reports in OneDrive to get a list of total and active accounts in OneDrive.
Migration
Once you get to this point there is no turning back! Well, that’s probably a bit dramatic but I sure wouldn’t want to after doing all the discovery and analysis on the home drive data…
Migration process
When it comes to FastTrack they are the one’s running the migrations for you. So with that there are some limitations that they set out up front:
FastTrack cannot complete more than 50,000 tasks
FastTrack cannot migrate more than 1TB of data in a 24 hour period
The migration process itself was a relatively simple one once your migration spreadsheet has been created for each batch of migrations and provided to the FastTrack team via their portal. With your migration dates scheduled with FastTrack, your role then becomes 1) making sure the user base is informed as to when they are being migrated and what to do, 2) liaise with FastTrack as the migrations complete, 3) then handover to the users after they’ve been migrated and finally 4) remediate any post-migration issues. Easy right?
Inform: make sure users know when they are being migrated. Migrate: work with FastTrack to migrate the date. Handover: handover to the users once migrated. Remediate: post-migration support of any issues.
Lessons learned
Aside from the issues and resolutions described below, the biggest thing I took away from the entire process is not to underestimate how much support your user base will require post-migration. We received lots of queries post migration around the change from home drives to OneDrive, some of which may of been addressed by more training, champions etc. but largely was just reacting to the change and not knowing what to do next.
Issues & resolution steps
#1 Not all users had OneDrive
As mentioned earlier in this post, during the analysis of the user data before migrating to OneDrive we discovered several users did not have OneDrive set up.
Before the OneDrive migration started in anger, we ran a few pilot migrations just to test the process out more thoroughly. We had setup the migration VM’s and migration agents some months earlier, so when we came to run the pilot migrations, all the migration tasks failed – pretty much immediately. In the logs for the failed tasks we received this error:
Error, the source file share does not exist.
Suggestion, Ensure the source file is an existing network file share and shared with the current user
SharePoint Migration Manager task error
Microsoft have an article on troubleshooting the Migration Manager here, but in this case the issue was actually related to do with expired account credentials running the migration agents. Although the error logs didn’t tell us this, once we re-ran the migration agent setup and “repaired” the agent, re-authenticating the accounts meant the migration tasks ran successfully thereafter.
#3 Permissions issues with files don’t show as “files scanned with issues”
Again, something I’ve touched on earlier but another issue we encountered during the migration was that any permissions issues that occur during the migration are not highlighted in the summary information for a given migration task.
So in the scenario post-migration, if a user gets in touch to say some files/ folders are missing you need to find the migration task, download the task report then comb through the ItemFailureReport to discover what data hasn’t been migrated due to permissions issues.
#4 “Missing” files and the hidden CSC folder
Kind of related to #3, we also encountered issues when switching off Windows Sync Center as part of our migration process. In some cases we had users who had files/ folders that hadn’t successfully synced from their local machine to their home drive, so following the migration and switch off of Sync Center it appeared their files had gone “missing”.
We resolved this by locating the hidden Windows CSC (client side cache) folder C:\\Windows\CSC, which is a location where Windows keeps offline copies of files and folders. Getting access into this location was a bit of a nightmare for us, but once you get access into the location we were able to find all of the reported “missing” files and copy them over to the users OneDrive.
Navigating to C:\\Windows\CSC with hidden folders enabled shows you this folder, which contains offline copies of files and folders.
In this post we will take a detailed look at the Migration Manager, it’s capabilities, limitations and how to create pre-scan and migration tasks for the first time.
So the good news is, if you have a Microsoft 365 subscription then you also have the Migration Manager too! The Migration Manager went into general availability in June 2020 for all customers as part of the SharePoint Online offering.
The Migration Manager is a part of the SharePoint admin center and is really simple and easy to get set up and begin using. Microsoft have this handy image to demonstrate how easy the process is:
Currently, the Migration Manager only supports file share migrations, this means that if you are planning to migrate any other content you hold, for example: on-premise SharePoint content or cloud based content you would need to use a separate tool.
Microsoft also offer several other tools that can facilitate the migration of that content too. Microsoft provide a table which recommends which tool to use here, but I’ve whittled them down to the main options below:
Migration Manager: used for network and local file share migrations, easy to set up via the SharePoint admin center
SharePoint Migration Tool: used for SharePoint Server 2010, 2013 & 2016 (Public Preview), network & local file shares, requires some prerequisites configuring before installation
Mover: Service for cloud to cloud migration (Dropbox, Google Drive etc.), easy to set up via web platform
Another limitation of the Migration Manager is that it currently does not support third party multi-factor authentication.
Setup considerations
So as the above breakdown of each Microsoft migration tool suggests, the Migration Manager is easy to get going. The key consideration when using this tool to migrate file share content is around the volume of data you are migrating.
If you are using the Microsoft FastTrack service, they will recommend that you set up a number of migration machines relative to the total number of files you are migrating.
The recommendation they offer for scoping this is below:
Total file count
Migration machines required
Less than 200,000
2
Between 200,000 and 500,000
3
Between 500,000 and 1,000,000
6
More than 1,000,000
To be discussed with FastTrack
Once you decide how many migration machines will be required, you will then need to setup the migration agents on each machine, check the prerequisites and required endpoints have been reviewed and met, and also have the relevant accounts being used as part of the migration process given access to the file share content and SharePoint admin roles.
File and folder permissions
When prepping for your file share migration, another consideration will be the permissions of a file once it is migrated. For most organisations it comes down to the following scenarios:
1. We want all file/ folder permissions preserved when they migrate to SharePoint Online
2. We only want specific file/ folder permissions mapped into SharePoint Online
3. We don’t want any of the permissions migrating, we want to start again!
Yes, I know the third option is highly unlikely, but I’ll include it anyway. When using the Migration Manager, the syncronization between your Active Directory on-premise and Azure Active Directory is key to how permissions migrate across.
It is important to note that if your organisation uses security groups in Active Directory to manage permissions for their file shares, they must be syncronized with Azure Active Directory in order for the user permissions to map across like-for-like.
If not, then you will be required to create a user mapping file to map user permissions for the relevant files or folders.
The Migration Manager also takes into account the same permissions conditions and results as the SharePoint Migration Tool. This table lists all the conditions and the corresponding results.
Demo: create a scan only task
Before you run any sort of migration task you will first want to get a handle on the current situation of your file share content, what areas will migrate easily and which will require remediation.
Both the scan and migration actions sit within Tasks in the Migration Manager and the initial setup for both actions is the same:
Press add task > under Method select the default single source and destination (unless you wish to scan multiple sources)
Under Source, enter the file share path that you wish to scan using the correct format
Under Destination, leave the SharePoint site URL and location as the default as we are only performing a scan
Under Settings, give your task a name > under common settings ensure perform scan only is checked
Press Run Now
Demo: create a migration task
So once you have ran a pre-scan of your file share source and you are happy with the results, it is now time to create a migration task! This again is very similar to how we approached creating the pre-scan task but here are the steps:
Press add task > under Method select the default single source and destination (unless you wish to scan multiple sources)
If you wish to do a bulk upload, you will need to provide a CSV or JSON file which is well documented here
Under Source > enter the file share path that you wish to migrate using the correct format
Under Destination > select the application where you are migrating the data to. Press next
Enter the URL of the location, then select the library or channel you wish to migrate to
Under Settings > enter a name for you task and configure your migration task based on the following check boxes:
Preserve file share permissions
Migrate hidden files
Migrate files created/modified after specified date
Do not migrate files with specific extensions
Migrate files and folders with invalid characters
Migrate OneNote folder as OneNote notebook
Azure Active Directory lookup
User mapping file
Automatically rerun failed tasks up to 4 times
Press Run Now
Reports analysis
Whether you have performed a pre-scan or migration in Migration Manager, each task once complete will provide a “task report” zip folder that is available to download. Microsoft break the reports down into Summary, Task level and Performance reports, but in fact all reports are included as part of the task report download.
The Microsoft breakdown of each report is very thorough, so I won’t bother adding any more detail, however I will highlight the reports that I found useful when using the Migration Manager:
Summary Report: contains a single row of data that gives the total picture; including total size, number of files migrated, duration
Item Failure Report: contains any errors found resulting in a file being unable or failing to migrate
Item Report R1: detailed report with data on each file within a task. If large number of files migrated, split into separate, sequential reports (R1, R2, R3 etc.)
An interesting aside I did notice was on the task details pane for a pre-scan I ran it showed under files scanned with issues as 0, but within the task report ZIP there was, in fact several files and folders listed as having failed due to a variety of reasons. So I’m not really sure how accurate the files scanned with issues is, or what it actually defines as issues.
Troubleshooting
Microsoft documentation includes a page on troubleshooting Migration Manager issues and errors here, but typically during a recent migration I experiencing issues that weren’t including in the above.
Error: the source file share does not exist (but it does)
This error I received recently during a home drive – OneDrive migration I was conducting. I was using windows virtual machines with the migration agents installed previously and had logged into the machine and SharePoint admin center as accounts with the relevant permissions and roles as described here by Microsoft.
When I set up my migration tasks as demo’ed above they were instantly failing. The only errors I would receive were these:
Microsoft have this listed as an agent error message – which was the clue, but their action was to:
Make sure the source file share is an existing network file share. Confirm that the Windows account associated with the agent has read permissions to the file share you want to migrate.
What I found was in my case, even if the migration agents were showing as enabled, because I had originally installed and configured the agents some time ago (1-2 months prior), they need to be repaired and re-authenticated:
Download the migration agent setup file
Microsoft documentation hasn’t yet been updated, but to get to the agent download you need to open Migration Manager > Press Agents > + Add
Run the agent setup file
Press continue to reinstall the agent
The installer will then begin the installation prerequisites
Enter account credentials for the SharePoint Admin, then Windows account
The installer will authenticate, then prompt you to test permissions with a file share, press OK to close
Now, when I re-ran my migration task with my agents repaired and re-authenticated, the tasks completed successfully.
Error: failed to duplicate token file due to error ‘There is not enough space on the disk’
This error occurred for me during a pre-scan of file servers locations. After kicking the scans off, I returned to find many of them failed. When looking at the task error details from the logs I found the following:
Error, Failed to duplicate token file due to error ‘There is not enough space on the disk.
This error was pretty much what it says on the tin, the VM had ran out of disk space. As referenced before, looking through Microsoft’s issues and errors post didn’t prove helpful as the error above is not listed.
So, through investigation of my own I found the following:
There are a bunch of MTHost text files that appear to be log files of actions completed when tasks are ran within the Migration Manager.
There is a Migration folder, within a subsequent MigrationTool folder, another folder with your tenant name will be found. In here will be folders with names beginning “WF_” which represents each task created within the Migration Manager.
Each “WF_” folder contains a Log and Report folder, with the report folder containing all the migration reports we detailed earlier in this post.
There is also a MigrationToolStorage folder which also contains a mirrored “WF_” series of folders related to tasks created within Migration Manager.
What’s crucial to understand here isyou are able to delete the “WF_” folders from these locations, without it affecting the tasks in Migration Manager, or the ability to download the corresponding task reports.
I’ve tested deleted all the “WF_” folders from both the above folders and then refreshing and trying to download task reports for the tasks in the Migration Manager – and they download perfectly!
What I would question is what is the point of these “WF_” files downloading locally in the first place if they are also stored somewhere in the M365 cloud as well!
You cannot migrate to a deeper destination than first sub-folder
When migrating files from source to destination, in some cases you may want to migrate specific data that sits within a larger, nested folder structure. For example, lets say you have a folder that sits in a file share like this:
\\contoso\fileshare\foldera\folderb
You only want to migrate folder band nothing elseexcept its associated folder structure. In Migration Manager, you are only able to select the first, top-level folder from a library. This applies to OneDrive, SharePoint or Teams and doesn’t offer any other sub-folders to be selected via the tool itself.
In Migration Manager you are only able to select the first. top-level folder from a library in OneDrive, SharePoint or Teams.
If you try to create migration tasks via CSV and include the sub-folders as additional columns in the spreadsheet, the Migration Manager ignores them and just migrates whatever is in the destination to the source library.
I would say generally this isn’t an issue as you can either skip the files you do not want to re-migrate and just set the source/ destination to be at the top-levels. This specific scenario came about because I was trying to remediate some files and folders that failed a prior migration, and once remediated migrate those folders only via Migration Manager into the corresponding folder structure in SharePoint. I would say this is an issue if you are doing what I described and don’t want to have to run another large migration task, even if it is skipping most of the files as the task will still take a while to complete.
SharePoint Stuff 