Problems creating list or library views based on created date

The situation

Data retention and deletion…I’m sure this is a something that anyone involved in Office 365, SharePoint on information management in general gets fed up of saying since the recent GDPR legislation!

Recently we have been rationalising and cleaning up our data in preparation for moving to Office 365. We are starting with SharePoint as the first target repository or silo of content.

The general consensus is to delete files and folders over 7 years old unless there is a pre-existing data retention policy to adhere to. So the next task is to identify those files that fall within our threshold, and ultimately delete.

Luckily, we have Tree Size Pro and ShareGate so I was able to relatively easily identify the files in question (there were a lot!).

The setup

As our SharePoint environment is a) rather full; and b) rather old, I made the decision to incrementally delete files rather than en-masse to mitigate risk, targeting the lists/libraries containing the most out of date content. I started by creating a view in the first library – library A with the following parameters:

  • Standard library view
  • Filtered by Created Date if less than or equal to 01/01/2011
  • Folders or Flat: Show items inside folders
    Show this view: In all folders

(all other settings are left default)

Results this returned looked good, I could see folders and files in this view that matched the criteria – brilliant! Based on my previous statement I decided to delete in batches out of working hours, again to mitigate risk. I deleted first from library A, then from the first stage and finally from the second stage recycle bin all in this fashion.

The problem

I had permanently deleted around 50% of the total volume of content to be deleted from library A when we started to receive reports of current files being ‘missing’ from library A…not a good day.

After these reports were investigated they were indeed true. It turns out that when folders are included within a library view, folders that match the filter will be shown in the view, regardless of whether the files inside match.

We tested the view exluding folders and all the files returned matched the filter criteria. The same results were demonstrated from a SharGate report of the same nature. The report of all files over 7 years old brought back folders over 7 years old, but they also contained files that were newer.

Conclusion

At present, we are not entirely sure as to why these filters are not able to drill down past a top-level folder. It appears to be difficult to specify via view settings to only show files within folders, including the folder itself that matches the criteria.

We have decided to omitt folders from our reports and views going forward and to solely focus on files as this is the most reliable way we can delete files.

Bonus: for those of you with ShareGate, heres an example of my report we created to bring back all files over 7 years old, excluding folders. I ran this report across the entire intranet application over a weekend and it worked a treat 🙂

SG-report

File naming restrictions in SharePoint

Firstly, full credit for this information goes to Mischa Oudhof and this sysadmin blog for this really useful post in 2010! I thought i’d write it up here just for my own sanity (and incase that page were to ever not exist) plus add a few extra bits of my own.

So recently a user was trying to upload multiple documents into SharePoint 2010 by copying/pasting using file explorer for the destination library.  When copying across the files they received something like this message:

filename-error

What are the restrictions?

There are a whole load of restrictions when it comes to file name, length, characters within SharePoint. This affects site collections, sub-sites libraries or lists. Here are some of the more general restrictions:

  • File and folder name lengths can’t exceed 128 characters in both WSS 2.0 as WSS 3.0
  • Link list items can’t exceed 256 characters
  • The entire path of files can’t exceed 260 characters

Here are some, more specific restrcitions again applying to site collections, sub-sites, libraries or lists:

  • Can’t be longer than 128 characters
  • Can’t use: ~ # % & * { } \ : < > ? / + | ”
  • Can’t use the period character consecutively in the middle of a file name (blah…blah.docx)
  • Can’t use the period character at the end of a file name
  • Can’t use the period character at the start of a file name
  • Can’t end with:

    .files

    _files

    -Dateien

    _fichiers

    _bestanden

    _file

    _archivos

    -filer

    _tiedostot

    _pliki

    _soubory

    _elemei

    _ficheiros

    _arquivos

    _dosyalar

    _datoteke

    _fitxers

    _failid

    _fails

    _bylos

    _fajlovi

A deep dive into all the potential restrcitions can be found in the msdn blog here.

Things have changed in Office 365…

Microsoft announced in 2017 that the old maxpath limit has been expanded to 400 Unicode code units. Microsoft also announced more URL improvements for the support of # and % characters.

New SharePoint team site

References

http://www.sysadminsblog.com/microsoft/file-name-length-and-character-restrictions-for-sharepoint/  – Original write up by Mischa Oudhof of Sysadminblog.com

SharePoint and Nintex workflows failing on start after .NET security update

Updated


I’ve wrote part two on this issue with my full resolution steps here:

SharePoint and Nintex workflows failing on start – part two


The problem


I had this issue myself in the last week where EVERY SINGLE workflow across the farm on premise stopped working. SharePoint Designer and Nintex workflows all reported “Failed to start” when triggered to run.

The workflows stopped working due to a series of .NET security updates Microsoft released in September 2018. Microsoft released a public KB article on this – with resolution steps which can be found below:

But also this msdn blog post contains all the solution scripts and steps that includes Nintex workflows also (transcript below):

I noticed shortly after the fix was implemented that some of my SharePoint designer workflows were exhibiting odd behaviour. For example the screenshot below shows a SharePoint desinger workflow that previously worked without issue or errors in the history after the fix was applied:

Someone on reddit had already spotted this which drew my attention to the common issue, this only presents itself for workflows with pause steps!

I will update this post with my findings once this latest fix is applied.

Symptom

After applying .NET Security Only patch to resolve CVE-2018-8421 (Remote Code Execution Vulnerability) , all SharePoint out of the box Workflows fail to execute and the log will show an error like this:

09/13/2018 01:59:07.57 w3wp.exe (0x1868) 0x22FC SharePoint Foundation Workflow Infrastructure 72fs Unexpected RunWorkflow: Microsoft.SharePoint.SPException: <Error><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″ Text=”Type System.CodeDom.CodeBinaryOperatorExpression is not marked as authorized in the application configuration file.” /><CompilerError Line=”-1″ Column=”-1″…

The error suggest that System.CodeDom.CodeBinaryOperatorExpression is not in the authorized types.

Cause

Workflow Foundation (WF) will only run workflows when all the dependent types and assemblies are authorized in the .NET config file (or added explicitly via code) under this tree:

<configuration>

<System.Workflow.ComponentModel.WorkflowCompiler>

<authorizedTypes>

<targetFx>

However, after the update, the following lines are necessary for SharePoint 2013 and beyond:

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeBinaryOperatorExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePrimitiveExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodInvokeExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeFieldReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeThisReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePropertyReferenceExpression” Authorized=”True” />

And for SharePoint 2007 and 2010, use these lines:

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeBinaryOperatorExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePrimitiveExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodInvokeExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeFieldReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeThisReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePropertyReferenceExpression” Authorized=”True” />

Solution

The solution is to add explicitly the types to all web applications’ web.config:

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeBinaryOperatorExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePrimitiveExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodInvokeExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeFieldReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeThisReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePropertyReferenceExpression” Authorized=”True” />

Or (for SharePoint 2007 and 2010):

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeBinaryOperatorExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePrimitiveExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodInvokeExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeMethodReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeFieldReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeThisReferenceExpression” Authorized=”True” />

<authorizedType Assembly=”System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodePropertyReferenceExpression” Authorized=”True” />

Please notice that sometimes SharePoint Timer Service (SPTimerV4) runs workflows. If you notice that the application showing the error is ULS logs in OWSTIMER.EXE, you should also include the authorized types in [SharePoint Hive Folder]\bin\OWSTIMER.EXE.config. The Hive Folder will change by version of SharePoint. For SharePoint 2016, it is normally at c:\program files\common files\microsoft shared\web server extensions\16. For 2013, at c:\program files\common files\microsoft shared\web server extensions\15.

Additional Information

My colleague Joe Rodgers, who is Sr. PFE, put together this PowerShell script: https://gist.github.com/joerodgers/2302b394796c865818839d843bae2dad

There are two scripts. Normally, the only necessary script is:

Add-CodeDomAuthorizedType.ps1

Uncomment this line to make the changes:

Add-CodeDomAuthorizedType

If you have Nintex workflows you should run like this:

Add-CodeDomAuthorizedType -IncludeNintexWorkflow

To undo the changes, run:

Remove-CodeDomAuthorizedType

The script needs to run only once on any WFE. All web.config files related to SharePoint on all servers will be modified. New web applications created after that will also include the changes. Even if a new WFE is added to the farm, the entries will also be included in web.config. The change is a permanent requirement from now on since the WF patch. You do not need to undo the change before applying the SharePoint patch addressing it.

There is a second script to update OWSTIMER.exe.config. This one should only run if you see the symptoms in ULS logs with process OWSTIMER.EXE. Otherwise, you do not need to update. if you have the problem though, you need to rerun the script if a new machine is added to the farm. No line needs to be uncommented for this one. The script name is:

Add-CodeDomAuthorizedTypeToOWSTimerConfig.ps1

Note

Microsoft is aware of this issue and patches for SharePoint 2010, 2013 and 2016 are being worked as of 9/17/2018. I will update when we have an ETA. I had confirmation from the product team on 9/18/2018 that this information and solution on this post is in the line with the future patch and it is the recommended action plan until the patch is out. If anything change, I will update the post.

Note 2

Some people using third-party workflows (like Nintex) need to also include this:

<authorizedType Assembly=”System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089″ NameSpace=”System.CodeDom” TypeName=”CodeTypeReferenceExpression” Authorized=”True” />

Using the script, you need to add to the line defining types (line 24):

CodeTypeReferenceExpression

Example:
$typeNames = @( CodeBinaryOperatorExpression, CodePrimitiveExpression, CodeMethodInvokeExpression, CodeMethodReferenceExpression, CodeFieldReferenceExpression,CodeThisReferenceExpression, CodePropertyReferenceExpression“, “CodeTypeReferenceExpression”)

Note 3

Joe updated his script to add a switch for Nintex workflows.

Call this way to include the extra type required by Nintex:

Add-CodeDomAuthorizedType -IncludeNintexWorkflow

(all credit to Rodney Viana for this information)

Hide a SharePoint list or library from view all site contents

Have you ever been asked to hide a list or library from a SharePoint site? If so, you go straight for selecting ‘no’ to displaying the list or library on the Quick Launch or removing it from the navigation. However, your eagle eyed users notice the handy view all site contents option and see that it is still listed there – they want it gone!

Luckily, all you need is SharePoint Designer and it is as simple as a click of a button…

(These steps were created using SharePoint Server 2010)

  • Open the site that where list or library resides in SharePoint Designer
  • Under Lists and Libraries – Select the list or library you wish to hide
  • On the main list settings page – find the Settings section
    SPD
  • Check the Hide from browser option
    hidefrombrowser

Thats it! when you option the view all site content page now, that list or library will no longer be showing. Also, if you want to re-instate it at a later date, just un-check the box and it will re-appear.

This also works for SharePoint 2013, 2016 and SharePoint Online, under the site contents page.

Fun with PowerApps part 3: my first PowerApp

This is part three in my fun with PowerApps series where I’ll go through my personal experience of creating a new PowerApp from scratch and go through step-by-step for each part of the process…

You can read all parts of this series below:

Part 1: setting up the default gateway
Part 2: creating a data connection

Carrying on from my last post – creating a data connection, we are now good to go and begin to build our first PowerApp!

Just like setting up the data connection, creating a PowerApp can be a really simple thing to do.

However, I’ve found either creating a PowerApp from the SharePoint list or library is really quick and easy, but it doesn’t necessarily give you the most flexibility when it comes to wanting to do it your own way…

Creating a PowerApp

You can create a PowerApp from a list or library literally from the push of a button! Its as simple as:

  • Create your list or library, add all the associated columns and data you wish to be displayed in your PowerApp
  • Press the ‘PowerApps’ button on the list menu, then press ‘Create an app’

  • You’ll then be prompted to create your app, start by giving it a name and press ‘Create

After a few moments your app will be created! It will open in the PowerApps web editor and you should see your list columns and data displaying in your shiny new PowerApp!


You can also take a look at the app in the preview to get a good look at how it operates and how the data is displayed…

Conclusion…

That’s it! Your all set…but if you’re like me you’ll have some questions. Like, what if I want a desktop app and not a mobile one?

Well for that you’ll need to build an app from scratch, I’ll be going over creating a new PowerApp in the next part of this series so hold on as i’ll be posted the next part very soon!

Fun with PowerApps part 2: creating a new data connection

This is part two in my fun with PowerApps series where I’ll go through my personal experience of creating a brand new PowerApp from scratch and go through step-by-step for each part of the process…

You can read part 1 of this series below:

Part 1: setting up the default gateway

Continuing on from my previous post – setting up the default gateway, we should now be in a position to create a new data connection within PowerApps. On the surface this is pretty easy to do, all that’s needed is to choose the connection type – press create – authenticate – job done!

However, when we need to create a data connection for an on premise SharePoint environment, its a little bit of a different story. Here are the steps involved for setting up a new data connection for a SharePoint on premise environment:

  1. Go to web.powerapps.com
  2. Under Connections – press New Connection

  3. In the search bar, search for “SharePoint” – press the plus icon

  4. The SharePoint data connection popup box should appear, under “how do you want to connect to your data?” select Connect using on-premises data gateway
  5. Set the authentication type to Windows and then specify your username and password (you may need to include a domain name as part of the username)
  6. I used credentials of a user that had owner permissions for the SharePoint environment I connected to, it makes sense to me that the account you specify here would at the very least need to be able to read/write to the SharePoint list that acts as the data source.

  7. Choose the gateway we set up in part 1 of this series, if you can’t see it you might need to install it again or refresh the list until it displays.
  8. I actually had to create the default gateway using a different account than the one I created my first PowerApp with, so it wouldn’t show for me initially. I had to share the gateway I created and set my PowerApps user as an admin for it to show in this list.

  9. Press Save

You will now see your newly created data connection appearing in the connections list! Your able to edit and delete the data connection at any point and can also check who the owner is, the connection name and crucially the status of the connection under the details button.

BONUS

There is also this article from Microsoft talking through connecting to SharePoint from PowerApps that lists the known issues with certain column types:

https://docs.microsoft.com/en-us/powerapps/connections/connection-sharepoint-online

Configuring outgoing email SharePoint 2016

One of the first things a user may ask their shiny, new SharePoint 2016 site collection to do is to have the ability to send email. Luckily, this is pretty easy to do and will just rely on you getting to grips with a few things about SMTP server:

  • SMTP server address
  • SMTP server port number

If you don’t know either of these, then it might be worth getting friendly with your system admins as they should be able to help! To start configuring outgoing email in SharePoint 2016, we need to set it up at the central admin level:

  • Navigate to central admin – system settings
  • Under Email and Text Messages – Configure outgoing email settings
  • In Outgoing email settings – enter your SMTP server information, from, reply-to address and press OK

Now your SharePoint environment should be configured to send outgoing email, next we will check that outgoing email is set up correctly at the web application level:

  • Navigate back to Central Admin – under Application Management – Manage web applications
  • Select the web application you wish to manage
  • Press the General Settings drop down – Outgoing email

Check that SMTP server details are correct (based on the farm level configuration we did earlier), you can also personalise the from and reply-to addresses for the web application if you want them to be different to the top-level.

BONUS

One issue I’ve encountered when configuring outgoing email in SharePoint 2016 is where lists or libraries aren’t sending email alerts. The initial email is sent when alerts are setup, but the subsequent email alerts don’t fire.

The issue quite a specific one but worth mentioning, as at first I applied this security update which included an update for outgoing email, but didn’t solve the problem. It turned out that in Exchange Server, if anonymous Relay settings isn’t enabled, then your SharePoint servers need to be added to the allowed hosts relay list. So back to my earlier point, if this issue affects you – make sure you take your system admin to lunch to pick his brain about exchange relays!