Tales of a Microsoft 365 migration – SharePoint

18 months on since my last post in this series, I have finally arrived at the last stop on the migration train! In this post I’ll describe how we migrated a SharePoint 2010 farm into SharePoint Online.

If you want to take a look back at my other posts documenting my organisations migration journey you can find them below:

Contents

Background

The aim of this part of the project was to migrate everything in scope from a SharePoint 2010 farm into SharePoint Online & Microsoft Teams, ultimately decommissioning the SharePoint 2010 environment. As part of the migration, all sites would be re-created as modern SharePoint sites, with the contents earmarked for migration moved from SharePoint 2010 to SharePoint Online.

Everything in scope includes site collections, sites, sub-sites, libraries, lists, forms, processes and workflows, with all forms and workflows were to be redeveloped using the Power Platform alongside the migration effort.

Discovery

Following a similar approach to the file share migration, we needed to find out:

How many site collections/ sites/ sub-sites are there? How large are they?
How many libraries/ lists are there and how many files/ items do they hold?
How many forms, workflows and custom solutions are there?
What does the permissions matrix behind the sites look like, are there any areas that require restricting post migration?

Due to our experiences following the file share migration, we were going to use Sharegate from the start for the SharePoint migration. As well as using Sharegate for the migrations themselves, we also used it to generate the inventory reports that we used as part of our migration tracking process. Most of the inventories were gathered using the reports provided by Sharegate, but in some cases I had to create custom reports due to the structure of our SharePoint environment, plus wanting to gather specific information around libraries and lists.

We also ran a Powershell script similar to the one provided by SharePoint Diary to get a list of all the Infopath forms within the SharePoint 2010 environment, this was added to another list of forms created using a 3rd party solution “Kintivo Forms”, plus any custom solutions deployed in the environment.

All this reporting data was collated and incorporated into a migration roadmap system using Lists, combined with links to each collated inventory report for each site collection.

*Example of the migration roadmap list, used to track progess and linked to the inventory reports.*

Analysis

As with the file share migrations we also continued to utilise Treesize to get a better understanding of the distribution of data within the libraries and lists, to best determine how to split larger ones up in order to migrate them successfully.

Once we had this information, we were able to incorporate a complexity score against each site based on what they contained with a view to target the complex ones first and remediate. From this point, we were able to contact colleagues to find site owners to work with to understand what resided within each site, with a view to make decisions on what needed to be migrated or removed.

In our case, the SharePoint 2010 environment was many things to many people – in my case I considered it more a Frankenstein’s monster! It was part intranet, part document management system, part business process, part team site. The problem this caused was around permissions, as we were not migrating the permissions groups from the old world to new. So as part of the inventory completion process, site owners had to specify whether the data being migrated from SharePoint 2010 was to be moved into an intranet site, or an organisational team site (restricted access to the relevant team only).

Migration

We had a lot of sites to migrate (over 15,000!), so as already mentioned having a migration roadmap was integral to the success of this project. The general approach to these migrations was to create the destination sites up-front in SharePoint Online, then use the inventories as a guide to create tasks in Sharegate to migrate the libraries and lists in scope.

As we had so many sites to create, we used another example provided by SharePoint Diary to bulk create sites from a CSV
We also had a large site collection that contained several thousand sub sites which also had custom site templates in use. For these, we decided to migrate site collection + sub sites, then modernize the site collection homepage and sub site homepages, again using PowerShell
As mentioned previously, we ruled out migrating any pages from SharePoint 2010, instead we encouraged site owners to copy/ paste the content from their old site pages where applicable, or just start from scratch

Intranet

There is already lots of information online about how to plan for an intelligent intranet, so I will just include some key takeaways I had from launching a new intranet:

Developing an information architecture (IA) was key to the success of the intranet. The IA gave the intranet development team something to work with when trying to structure the intranet in terms of hubs and hub site association
Having dedicated, named site owners was another key to success. All the newly appointed site owners for the intranet went on a tailored training session prior and all built there site homepages prior to launch
Having a steering group helped to make decisions by committee on design, templates, the landing page and develop a champion network

Issues

As with any large scale project, we had our fair share of problems. I’ve jotted down some of the more difficult ones, with details on how we overcame them which will hopefully help others:

Large libraries

Microsoft state you can store up to 30 million files or folders within document libraries, but you quickly run into issues when surpassing 100,000 items. Microsoft have documented the issues with permissions inheritance with more than 100,000 items, but that’s not the only issue you’ll encounter.

There is also the 5,000 item threshold limit puts a hard stop and the number of items displayed within a library view. Although, if you have used folders to separate your data, you can get round this limit.

Syncing large libraries can also be a disaster as the OneDrive sync client can only handle up to 300,000 files on a given device. Microsoft advice that you can occur “performance issues”, which I have personally seen to mean that OneDrive never completes syncing, crashes the laptop and results in it needing re-imaging or replacing.

Moving/ copying large amounts of data from large libraries using in-built copy/ move functionality can also be very problematic. Microsoft recommend:

No more than 100 GB total file size
No more than 30,000 files
Each file must be less than 15 GB

However, this doesn’t stop end users being able to copy/ move large folders containing way more than this. I’ve observed this to result in files not being moved to the destination and being deleted from their original location anyway. This proves a mammoth task to restore from recycle bin, or from cloud backup if you have one.

There were several issues we experienced during migrations with Sharegate. Here’s a summary:

Issues with image tags – see my post here for more
Migrating pages between modern sites – custom scripting needs to be enabled in order to do this. Include link to a blog post tbc
Duplicate lookup column values – this issue causes files not to migrate if it finds duplicate lookup values in a files metadata. This was typically caused by a lookup value being deleted, then re-added
Hidden required fields – several in-built, hidden or lesser known properties were being migrated but not mapped so were causing files to fail. Solution to this was to map everything required, even if its to a blank or NA value
Draft versions of files – If minor/ major versioning isn’t enabled in the destination library, files will fail to migrate
Random, inconsist errors, such as:
- Content database has ran out of free space
- An unexpected error occurred: Cannot access a closed Stream.
- The file could not be obtained from the source. This sometimes happens with large files, retry later.
- An unexpected error occurred: Specified method is not supported
- All of these were resolved by re-running the migration tasks

Site pages

The reason we decided not to migrate site pages was how difficult they are to migrate. Namely: incompatible page layouts (not modern), not always sitting in the same site pages library, can cause major issues when migrating into modern team sites by enabling classic features. Site Pages can be updated to become modern, but this requires using PnP page modernization, which in of itself is a steep learning curve.

Infopath forms

There is no direct way to migrate Infopath forms to Power Platform, so you already are looking at a re-development job per form, not a migration task. Even if you were able to migrate them, I personally don’t think it would be a good idea for a variety of reasons:

Power Apps isn’t a like for like replacement, as Infopath has things Power Apps can’t natively do (think repeating sections) and vice versa
Infopath forms can’t be lifted & shifted, taking forms on face value (or looking at the underlying rules/ XML schema) can be misleading
Infopath forms can be used in weird ways – for example blank ‘submitted’ xml forms saved to SP2010 were being opened, filled out and emailed to a shared mailbox without the attached template being present or even existing!
Review/ analysis meetings are needed with key business stakeholders to better understand the process behind the forms in order to create the best new solution

Lessons learned

Track your migrations: Make sure you have inventories of all the information stored within your SharePoint environment are key to a successful migration. You have to have a way to keeping track of what there is, where it is and what it’s migration status is
Adopt a robust information architecture: Strive to an information architecture to guide where data from SharePoint goes once migrated. Don’t let the old, stale content pollute your new intranet by ensuring intranet sites only have data that has been vetted by site owners
Named, trained site owners : Develop a training programme for site owners, have them be responsible for the sites and their content. Give them access to a training sandpit and resources to help them learn and share success
If in doubt, migrate: Site owners won’t always be present, or know much, if anything about the data. Although not desirable, it’s actually much easier to migrate the sites into classic sites in SharePoint Online. Migrating classic sites in SharePoint largely look and feel the same – all the content was there, including pages and everything was preserved. The key to this though was to ringfence these sites and clearly label them as “archive”, only making them read only to a select few, no links from the intranet and having a clear plan for getting the relevant data out of these sites into an official corporate resource
There is no such thing as a lift and shift migration! With SharePoint environments, the devil is most definitely in the detail. Large libraries/ lists, complex forms and processes all lead to solutions that have to be reworked as part of the migration effort