Tales of a SharePoint migration – part one

August 13, 2019August 12, 2019Anthony1 Comment

The path to a trial migration

I’ve been suitably inspired by Andrew Warland’s fantastic two-part series documenting his approach and migration to SharePoint Online, so much so that I thought it would be a fun series to write about my own experiences.

Take a look at Andrew’s blog series here.

It is’nt my intention to necessarily document Microsoft best practice in this series, rather just to explore some of the challenges, sucesses and experiences I notice along the way.

The current situation

My organisation has recently made the decision to move to to the cloud, with O365 being the naturally preferred destination. SharePoint has been well embedded, and heavily used within the business for several years, with on-premises SharePoint 2010 currently in production.

Finally, in terms of the SharePoint architecture and data volume, there are only three web applications to merge together as part of the migration effort. However, there are several site collections within our main intranet web app, plus many sub-sites nested within them, meaning the huge database sizes behind these site collections could prove difficult come migration time.

A note on the new, flat structure

Our current environment has a well established top-down structure in place that is generally consitent across the environment.

Having already made the investment in ShareGate, this will be the tool of choice for the migration. In the version 11.0 release of ShareGate, a new restructure option now allows you to promote sub-sites to top-level sites post inital migration from the source SharePoint environment.

*The new restructure option in ShareGate 11.0*

Considerations for a successful migration plan

One of the biggest issues to be resolved before we can start any sort of migration activity, is the fact that we have several content databases well over the 200GB recommended general use size limit.

Read about the recommended size limit here

Microsoft best practice suggests that any environment that has site collections, sites, content databases, libraries or lists that exceed the software boundaries and limits should be remediated prior to any migration activity. In this case, the main idea is to split each content database that exceeds 200GB into seperate content db’s, and where neccessary, move or promote sub-sites to site collections and attach new db’s.

Armed with the knowledge of the recent restrcuture functionality coming to ShareGate, plus my own personal feeling that any remediation activities to our current environment may in of itself carry adverse risk to the estate we proposed a different approach.

Trial migration begins

With all the reporting capabilities at our disposal via ShareGate, I was able to get a firm grasp of what resides within each site collection in our environment, in terms of:

The size of each sub-site underneath the top-level
Number/ size of libraries and lists
Number of items in each of the above
Any workflows running in any of the above

From this I ran a trial migration of a sub-site from SharePoint 2010 to a newly created team site in SharePoint Online.

Pre-migration

Before I kicked off the migration, I ran the source analysis tool within the Migration > Plan section of ShareGate. I noted the following obersavations:

The source analysis within “migration” in the ShareGate tool, although listed as only being able to analyze up to SharePoint 2013, does in-fact work for 2010

The source analysis cannot run at the sub-site level, meaning that you need to run it at the site collection level then just filter down to the sub-site in question through the report itself
Source analysis gives you a report of all checked-out files within a source site.From this, I created a simple view within each of the libraries that contained checked-out files to send to the site owners for action

Post-trial migration

The trial migration completed successfully as expected, however there were several interesting results I noted:

1. Everyone receieves a welcome email

If you migrate the permissions, once the source permission groups migrate each user will recieve a welcome email to the new SharePoint Online site.

There’s no GUI control for this as of this time of writing, but you can switch off the email notifications via PowerShell.

2. /Pages/ or /SitePages/…that is the question

Publishing sites seem to be the trickiest to migrate, especially those with custom master pages or page layouts. When migrating publishing sites, the Pages library is migrated wholesail, meaning the content won’t reside in the SitePages library (where new client-side pages are located).

3. Un-editable modern homepage

After the migration had completed, the new team site homepage threw up an error every time you tried to edit it.

I tried some of the documented resolution steps found here, but none of them worked for me. My solution was to just create a new page to replace the broken homepage, add all the relevant webparts and make this one the new default homepage.

Transforming classic publishing site pages to client-side pages

Publishing site pages will all be migrated as classic SharePoint pages, without the modern look and feel of a client-side page. My understanding is that for publishing pages with custom page layouts, additional metadata or custom content types will need to be transformed via PowerShell and creating a custom mapping file.

(I’m planning on writing a seperate blog post walking through an advanced publishing page transformation in the near future)

Its also worth considering that in the release notes for ShareGate 11.0 it makes mention of the fact they are researching the ability to transform classic to modern pages, so that could well simplify this process in a future release.

Conclusion

Overall, I was happy with our trial migration and believe it is a viable approach for us to move from on-prem to O365. Some lessons learned for myself would be to consider and SharePoint permissions audit prior to migration to remove any unecessary permissions, send an inventory out to site owners aswell as checked-out files, all in the name of reducing the migration effort.

This will be an ongoing series of posts, which i’ll focus more the on the nitty-gritty of the migration effort than anything else, but as always if there is any feedback or suggestions on how to improve this site, please let me know!

Configure a default view for Document Sets

July 1, 2019August 13, 2019AnthonyLeave a comment

(This post was written using a SharePoint 2010 environment)

Document sets are a pretty awesome way to convert people from traditional folders to using metadata in SharePoint, but retaining the ‘feel’ of a nice folder structure.

When you start using document sets you might want to play with the default view of your files, this isnt as straight forward as just setting a new default view. Here’s how to do it

Create and configure the view

Create your view
Under Folders > ensure show this view is set to In folders of content type: document set
Press OK

*Select the document set content type in the view settings*

Select the view for the Document Set Content Type

Navigate to the Library Settings
Under Content Types Select the Document Set
In Settings > Select Document Set settings

Under Welcome Page View > select your view
Press OK

Now the changes made to your custom view will be present inside each document set!

Bonus

For more on document sets, check out my write up of the quirks of document set routing.

How to show the folder path of a file in library views

May 20, 2019May 14, 2020AnthonyLeave a comment

Introduction

Firstly, I want to make a mention to this great blog post by Tom Benjamin for giving me the direction as to how to achieve this. His post is way more complex than what I was looking to do, so if you want to find out more about how to show files from a folder using CAML queries take a look:

Link: https://www.tombenjamin.ca/blog/designer/view/using-caml-query-show-files-folder/

The scenario

A common request I get is:

How do I see what folders/ sub-folders my files are in at a glance
– all users everywhere

Out of the box, there aren’t any columns available that you could potentially leverage to display this information in a standard SharePoint 2010 library.

The solution

So, just by adding one value in SharePoint Designer, here’s how you do it:

Navigate to the library you wish to change, create a new view under Library Tools > Library > Create View

Choose the relevant format of your view, give your view a name and press OK
Open SharePoint Designer > Open the site > open the library you were just working in
In the Views pane > click to open the view you just created

*In SharePoint Designer, clicking on the view name will open the view in edit mode*

In the code editor window, scroll down until you see something like the following:

<ViewFields>
	<FieldRef Name="DocIcon"/>
	<FieldRef Name="LinkFilename"/>
	<FieldRef Name="Modified"/>
	<FieldRef Name="Editor"/>
</ViewFields>

Add the following field reference in between the opening <ViewFields> and closing </ViewFields>
Add the field reference in the display order you would like it to appear in the view

*Add the field reference to the View Fields list*

Press the Save icon to save your changes
Press the Preview button to see your view in action in the browser

Now you will notice there is a new column being displayed “Path”, that is showing us the full location of the file or folder in the libary. You’ll also notice that this path will display data when at the library root, or in any folders or sub-folders in the library.

*File in sub-folder displaying relative location*

Bonus

Taking this one step further, what if we wanted to show files of a certain type, then create a view that groups these files by their folder location? Guess what, that’s exactly what I did!

Navigate to your library > create a new view as before, this time base your new view off the one you just created
If you wish to only show files of a particular type, use the filter by settings (for example below is filtered to only shows Word documents)
Make sure “show all items without folders” is selected
Press OK

*Filtering to only show word documents, also showing items without folders*

Back in SharePoint Designer > Open up the view you just created
Scroll down until you see the opening <Query> tag and add the following beneath it:

<GroupBy Collapse="FALSE" GroupLimit="30">
	<FieldRef Name="FileDirRef"/>
</GroupBy>

Save and preview your view, it should now be grouping by the Path field:

I know this has proven really useful for my company, so hopefully this helps out someone else too 🙂

Problems creating list or library views based on created date

April 16, 2019May 20, 2019AnthonyLeave a comment

The situation

Data retention and deletion…I’m sure this is a something that anyone involved in Office 365, SharePoint on information management in general gets fed up of saying since the recent GDPR legislation!

Recently we have been rationalising and cleaning up our data in preparation for moving to Office 365. We are starting with SharePoint as the first target repository or silo of content.

The general consensus is to delete files and folders over 7 years old unless there is a pre-existing data retention policy to adhere to. So the next task is to identify those files that fall within our threshold, and ultimately delete.

Luckily, we have Tree Size Pro and ShareGate so I was able to relatively easily identify the files in question (there were a lot!).

The setup

As our SharePoint environment is a) rather full; and b) rather old, I made the decision to incrementally delete files rather than en-masse to mitigate risk, targeting the lists/libraries containing the most out of date content. I started by creating a view in the first library – library A with the following parameters:

Standard library view
Filtered by Created Date if less than or equal to 01/01/2011
Folders or Flat: Show items inside folders
Show this view: In all folders
(all other settings are left default)

Results this returned looked good, I could see folders and files in this view that matched the criteria – brilliant! Based on my previous statement I decided to delete in batches out of working hours, again to mitigate risk. I deleted first from library A, then from the first stage and finally from the second stage recycle bin all in this fashion.

The problem

I had permanently deleted around 50% of the total volume of content to be deleted from library A when we started to receive reports of current files being ‘missing’ from library A…not a good day.

After these reports were investigated they were indeed true. It turns out that when folders are included within a library view, folders that match the filter will be shown in the view, regardless of whether the files inside match.

We tested the view exluding folders and all the files returned matched the filter criteria. The same results were demonstrated from a SharGate report of the same nature. The report of all files over 7 years old brought back folders over 7 years old, but they also contained files that were newer.

Conclusion

At present, we are not entirely sure as to why these filters are not able to drill down past a top-level folder. It appears to be difficult to specify via view settings to only show files within folders, including the folder itself that matches the criteria.

We have decided to omitt folders from our reports and views going forward and to solely focus on files as this is the most reliable way we can delete files.

Bonus: for those of you with ShareGate, heres an example of my report we created to bring back all files over 7 years old, excluding folders. I ran this report across the entire intranet application over a weekend and it worked a treat 🙂

Fix pages with no publishing options in SharePoint

March 18, 2019April 22, 2019AnthonyLeave a comment

(This post was written using a SharePoint 2010 environment)

So you’ve got a SharePoint site, it all looks good (well, as good as it can!) but you notice that the Publish tab isn’t available in the ribbon.

First things first you check the site settings to see if SharePoint Server Publishing is turned on.

If you get to this point and your still no further forward it’s likely that your site wasn’t set up as a publishing site, but if you follow the steps below and your pages will be able to be published in no time.

Open the site in question, then go to Site Actions > View All Site Content
Open the SitePages library
Under Library Tools > Library, select Library Settings
Under General Settings > Versioning Settings, turn on Create major and minor (draft) versions > press OK
Go back to the original page, you will now see the publishing tab has appeared!

SharePoint and Nintex workflows failing on start pt.2 FULLY RESOLVED

February 6, 2019February 6, 2019Anthony1 Comment

It’s back again…a few months ago a wrote about my experiences with workflows failing on start after a .NET security update that was applied. You can read that post here:

SharePoint and Nintex workflows failing on start

Recently, the same .NET security update was applied to our SharePoint 2010 farm, which in turn caused the failing on start error to present itself again across all the workflows in the farm.

After identifing the issue soon after the update was applied, we decided to follow the same tact as before and roll back the patches, restart the servers and re-test the workflows – However, this time the results were different.

What was different?

Previously, rolling back the security update and any other patches added during this time, plus restarting the servers “fixed” the issue. This time, the same process did not yeald the same results and the workflows were still broken.

After performing the steps above, we observed that standard SharePoint workflows with a pause started to run sucessfully again, but Nintex workflows with a pause step either failed on start, or completed but errored after the pause step and sent an error notification.

*Example 1 of nintex workflow with pause step failing on start*

*Example 2 of nintex workflow with pause step erroring, but completing*

How we fixed it…

So this time we followed the updated step-by-step guide provided below on how to update the web.config files and OWS timer files via Add-CodeDomAuthorizedTypeToOWSTimerConfig.ps1 PowerShell script on the SharePoint Application server.

https://blogs.msdn.microsoft.com/rodneyviana/2018/10/12/step-by-step-video-on-how-to-fix-the-sharepoint-workflow/

We ran the script as recommended, which re-added the assemblies and dependancies to the OWSTimer config file and the web.config files on associated web servers and this in fact fixed the issue! As the script does an IIS reset/ Timer Job recycle we didnt even need to restart the servers!

I hope the that tidbit regarding the nintex workflow pauses helps someone else 🙂

File naming restrictions in SharePoint

December 3, 2018February 6, 2019AnthonyLeave a comment

Firstly, full credit for this information goes to Mischa Oudhof and this sysadmin blog for this really useful post in 2010! I thought i’d write it up here just for my own sanity (and incase that page were to ever not exist) plus add a few extra bits of my own.

So recently a user was trying to upload multiple documents into SharePoint 2010 by copying/pasting using file explorer for the destination library. When copying across the files they received something like this message:

What are the restrictions?

There are a whole load of restrictions when it comes to file name, length, characters within SharePoint. This affects site collections, sub-sites libraries or lists. Here are some of the more general restrictions:

File and folder name lengths can’t exceed 128 characters in both WSS 2.0 as WSS 3.0
Link list items can’t exceed 256 characters
The entire path of files can’t exceed 260 characters

Here are some, more specific restrcitions again applying to site collections, sub-sites, libraries or lists:

Can’t be longer than 128 characters
Can’t use: ~ # % & * { } \ : < > ? / + | ”
Can’t use the period character consecutively in the middle of a file name (blah…blah.docx)
Can’t use the period character at the end of a file name
Can’t use the period character at the start of a file name
Can’t end with:
.files

_files

-Dateien

_fichiers

_bestanden

_file

_archivos

-filer

_tiedostot

_pliki

_soubory

_elemei

_ficheiros

_arquivos

_dosyalar

_datoteke

_fitxers

_failid

_fails

_bylos

_fajlovi

A deep dive into all the potential restrcitions can be found in the msdn blog here.

Things have changed in Office 365…

Microsoft announced in 2017 that the old maxpath limit has been expanded to 400 Unicode code units. Microsoft also announced more URL improvements for the support of # and % characters.

References

http://www.sysadminsblog.com/microsoft/file-name-length-and-character-restrictions-for-sharepoint/ – Original write up by Mischa Oudhof of Sysadminblog.com

Ways around the 10 item number order limit in choice columns

October 22, 2018February 6, 2019Anthony1 Comment

I was recently updating a view in a SharePoint List, the view was set up to use metadata fields to sort and group the content…lovely stuff. What I was required to do was to implement a choice field with a numerical order within it (i.e. 1. First step, 2. Second step, 3. Third step).

With sort order in List/Library views, it works with either alphabetical or numerical options ascending or descending. What I found was with choice fields operating as the number order, once you hit 10 the numbering system went out the window!

What you end up with is something like this:

1) First choice
11) Eleventh choice
12) Twelfth choice
2) Second choice
3) Third choice

and so on…

By default, SharePoint interperates the choice field as alphabetical so the way I got around this was to just use:

a)
b)
c)

This gets around any issues with numerical values over 10 or having to create lookup lists or anything else 🙂

SharePoint and Nintex workflows failing on start after .NET security update

September 27, 2018February 7, 2019Anthony2 Comments

Updated

I’ve wrote part two on this issue with my full resolution steps here:

SharePoint and Nintex workflows failing on start – part two

The problem

I had this issue myself in the last week where EVERY SINGLE workflow across the farm on premise stopped working. SharePoint Designer and Nintex workflows all reported “Failed to start” when triggered to run.

The workflows stopped working due to a series of .NET security updates Microsoft released in September 2018. Microsoft released a public KB article on this – with resolution steps which can be found below:

https://support.microsoft.com/en-us/help/4465015/sharepoint-workflows-stop-after-cve-2018-8421-security-updatehttps://support.microsoft.com/en-us/help/4465015/sharepoint-workflows-stop-after-cve-2018-8421-security-updatehttps://support.microsoft.com/en-us/help/4465015/sharepoint-workflows-stop-after-cve-2018-8421-security-update

But also this msdn blog post contains all the solution scripts and steps that includes Nintex workflows also (transcript below):

https://blogs.msdn.microsoft.com/rodneyviana/2018/09/13/after-installing-net-security-patches-to-address-cve-2018-8421-sharepoint-workflows-stop-working/

I noticed shortly after the fix was implemented that some of my SharePoint designer workflows were exhibiting odd behaviour. For example the screenshot below shows a SharePoint desinger workflow that previously worked without issue or errors in the history after the fix was applied:

Someone on reddit had already spotted this which drew my attention to the common issue, this only presents itself for workflows with pause steps!

https://www.reddit.com/r/sharepoint/comments/9koq2y/further_issues_around_paused_workflows_and/

I will update this post with my findings once this latest fix is applied.

Symptom

After applying .NET Security Only patch to resolve CVE-2018-8421 (Remote Code Execution Vulnerability) , all SharePoint out of the box Workflows fail to execute and the log will show an error like this:

09/13/2018 01:59:07.57 w3wp.exe (0x1868) 0x22FC SharePoint Foundation Workflow Infrastructure 72fs Unexpected RunWorkflow: Microsoft.SharePoint.SPException: <Error><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″…

The error suggest that System.CodeDom.CodeBinaryOperatorExpression is not in the authorized types.

Cause

Workflow Foundation (WF) will only run workflows when all the dependent types and assemblies are authorized in the .NET config file (or added explicitly via code) under this tree:

<System.Workflow.ComponentModel.WorkflowCompiler>

However, after the update, the following lines are necessary for SharePoint 2013 and beyond:

And for SharePoint 2007 and 2010, use these lines:

Solution

The solution is to add explicitly the types to all web applications’ web.config:

Or (for SharePoint 2007 and 2010):

Please notice that sometimes SharePoint Timer Service (SPTimerV4) runs workflows. If you notice that the application showing the error is ULS logs in OWSTIMER.EXE, you should also include the authorized types in [SharePoint Hive Folder]\bin\OWSTIMER.EXE.config. The Hive Folder will change by version of SharePoint. For SharePoint 2016, it is normally at c:\program files\common files\microsoft shared\web server extensions\16. For 2013, at c:\program files\common files\microsoft shared\web server extensions\15.

Additional Information

My colleague Joe Rodgers, who is Sr. PFE, put together this PowerShell script: https://gist.github.com/joerodgers/2302b394796c865818839d843bae2dad

There are two scripts. Normally, the only necessary script is:

Add-CodeDomAuthorizedType.ps1

Uncomment this line to make the changes:

Add-CodeDomAuthorizedType

If you have Nintex workflows you should run like this:

Add-CodeDomAuthorizedType -IncludeNintexWorkflow

To undo the changes, run:

Remove-CodeDomAuthorizedType

The script needs to run only once on any WFE. All web.config files related to SharePoint on all servers will be modified. New web applications created after that will also include the changes. Even if a new WFE is added to the farm, the entries will also be included in web.config. The change is a permanent requirement from now on since the WF patch. You do not need to undo the change before applying the SharePoint patch addressing it.

There is a second script to update OWSTIMER.exe.config. This one should only run if you see the symptoms in ULS logs with process OWSTIMER.EXE. Otherwise, you do not need to update. if you have the problem though, you need to rerun the script if a new machine is added to the farm. No line needs to be uncommented for this one. The script name is:

Add-CodeDomAuthorizedTypeToOWSTimerConfig.ps1

Note

Microsoft is aware of this issue and patches for SharePoint 2010, 2013 and 2016 are being worked as of 9/17/2018. I will update when we have an ETA. I had confirmation from the product team on 9/18/2018 that this information and solution on this post is in the line with the future patch and it is the recommended action plan until the patch is out. If anything change, I will update the post.

Note 2

Some people using third-party workflows (like Nintex) need to also include this:

Using the script, you need to add to the line defining types (line 24):

CodeTypeReferenceExpression

Example:
$typeNames = @( “CodeBinaryOperatorExpression“, “CodePrimitiveExpression“, “CodeMethodInvokeExpression“, “CodeMethodReferenceExpression“, “CodeFieldReferenceExpression“,“CodeThisReferenceExpression“, “CodePropertyReferenceExpression“, “CodeTypeReferenceExpression”)

Note 3

Joe updated his script to add a switch for Nintex workflows.

Call this way to include the extra type required by Nintex:

Add-CodeDomAuthorizedType -IncludeNintexWorkflow

(all credit to Rodney Viana for this information)

Save as site template missing or greyed out

September 6, 2018August 27, 2019AnthonyLeave a comment

(This post was written using a SharePoint 2010 environment)

I recently found myself needing to create a site template for a customer who required multiple sites created following a base format. So I decided to create a site template to achieve this to speed up the site creation process for said customer.

At first I went into the site settings of the site that the template would be based off to create it. I looked under look and feel in site settings, but alas the option to save as site template wasn’t there!

Next, I accessed the site via SharePoint Designer, looked under the site ribbon menu and would you believe the save as template button was greyed out there also!

After researching online, it appears there can be problems creating templates from publishing sites in SharePoint 2010, however I found that following the steps below enabled me to create a new site template and activate it not problem!

Navigate to the site collection/ sub-site you wish to create the template from
Add the following to the URL at the end of the site name

/_layouts/savetmpl.aspx

Go back to the site collection or sub-site you wish to create new sites under
Site Actions – Site Settings – Look and Feel – Page Layouts and site templates

Find your newly created site template within the subsite templates list
Press Add
Press OK

Please note: if your sub-site is inheriting from the parent site, you will need to navigate to the parent and add the subsite template there for it to be available.

Just to be clear. Using save-site-as-a-template with modern team sites or publishing sites is NOT supported. You can always find aways with tenant admin permissions to achieve things, but the fact that it’s doable does not mean it’s supported.

Any sites created with the template which has been produced with this method can have unexpected issues in the future, so provided steps should never be used.

Best regards,

Vesa Juvonen, Senior Program Manager, SharePoint Engineering, Microsoft.

UPDATE 27/08/2019

Saving as site template is not a supported feature of SharePoint Online for modern sites. You will find for modern sites that this option is not available, but you have the ability to re-instate it via activating a feature. The official stance from Microsoft is:

” Just to be clear. Using save-site-as-a-template with modern team sites or publishing sites is NOT supported. You can always find aways with tenant admin permissions to achieve things, but the fact that it’s doable does not mean it’s supported.

Any sites created with the template which has been produced with this method can have unexpected issues in the future, so provided steps should never be used. “

– Vesa Juvonen, Senior Program Manager, SharePoint Engineering, Microsoft.