Migrating a Multiterabyte Site Collection

The problem

A certain site collection grew too large (8TB, while the Microsoft-recommended limit is 100 GB), and it needs to be migrated to a SharePoint 2010 environment and split into many smaller site collections.

The solution

In this presentation, I show the approach my company (Exostar) took to solve this problem. The main idea is to use a third-party product (we recommend Metalogix) and to write automated scripts to be run in parallel.

Migration and Task Forms

The problem

We migrated a SharePoint 2007 site collection to SharePoint 2010, using the database attach method. This site collection contains a custom workflow with multiple task forms. After the migration, we found that one of the fields in the completed task forms is blank, even though it was a mandatory field that was indeed completed before the migration.

The solution

First of all, we needed to make sure that the data was there. The completed task form data is stored in the task's ExtendedProperty. So, we ran this script and made sure that the correct attribute was there:

cls

$w = Get-SPWeb "https://main.myveryownsite.com/sites/somesite"

$tasks = $w.Lists["Tasks"]

$iscs  = $tasks.Items | Where-Object { $_.Title -eq "Some task on Some List Item."}

$iscs | % {

Write-Host " "

$_.Title

$_["ExtendedProperties"]

}

After we found that the data was stored correctly, the next task was to make sure that the InfoPath form was displaying this field correctly. Unfortunately, the InfoPath 2010 form had some problems with this field and would not display its value, even though the task's ItemMetadata.xml was perfectly correct.

The solution involved renaming the field in the InfoPath form, and then changing the migrated data so that the field is mentioned by its new name. Here is the PowerShell script that does the trick.

cls

$site = Get-SPSite "https://main.myveryownsite.com/sites/somesite"

$webs = $site.AllWebs | Where-Object { $_.IsRootWeb -ne $true -and $_.WebTemplate.StartsWith("STS")} 

foreach ($web in $webs)
{
    $web.AllowUnsafeUpdates = $true

    Write-Host "Processing " $web.Url ", web template:" $web.WebTemplate
   
    $web.Lists | Where-Object { $_.BaseTemplate -eq [Microsoft.SharePoint.SPListTemplateType]::Tasks -and $_.Fields.ContainsField("ExtendedProperties") }  | % {
        Write-Host "  Task list" $_.Title " in web " $_.ParentWebUrl " contains ExtendedProperties - investigating..."
  
        $listItems = $_.Items
  
        $listItems | % {
       
                try
                {
                    $itemEP = $_["ExtendedProperties"]
             
                    if ($itemEP -ne $null -and $itemEP.Contains("ows_AuthoritiesComments"))
                    {
                        if ($_.Title.StartsWith("ISC Chairperson task on"))
                        {
                            $itemEP = $itemEP.Replace("ows_OldFieldName", "ows_NewFieldName");
                                                                                      
                            $_["ExtendedProperties"] = $itemEP
                  
                            Write-Host "      Trying to update " $_.Title " extended properties"
                            $_.SystemUpdate($false)
                            Write-Host "      Finished updating " $_.Title " extended properties"
                        }
                    }
                }
                catch [Exception] {
                        Write-Error $_.Exception.ToString()
                    }
            }
        }
        $web.AllowUnsafeUpdates = $false
        $web.Dispose()
}

Worked like a charm!

Programmatically Uploading a Large Number of Documents to a Library

The problem:

I need to see how well my document library performs when it has more than 5,000 documents. To do that, I need to generate more than 5,000 documents and upload them to the library. It would be nice to do it automatically.

The solution:

I have a simple Aha.docx file. First, I need to copy it 5,000 times. I chose to do it via PowerShell:

$path = "C:\Users\Administrator.TWS\Documents\TestDocs\Aha.docx"
$i = 1
do  { $newPath = "C:\Users\Administrator.TWS\Documents\TestDocs\" + $i + ".docx"; Copy-Item $path $newPath; $i++ }
while ($i -le 5000)

Now my TestDocs folder has the Aha.docx file plus 5,000 other files, and I need to upload them to the Documents library. Here is the corresponding PowerShell script:

$spWeb = Get-SPWeb -Identity https://main.my.server.com/sites/Boris/TheSiteIAmTesting
$spFolder = $spWeb.GetFolder("Documents")
$spFileCollection = $spFolder.Files
$path = "C:\Users\Administrator.TWS\Documents\TestDocs\Aha.docx"
$file = Get-ChildItem $path
$spFileCollection.Add("Documents/Aha.docx",$file.OpenRead(),$false)
#Now, let's copy 5000 other files
$i = 1
do { $path = "C:\Users\Administrator.TWS\Documents\TestDocs\" + $i + ".docx"; $file = Get-ChildItem $path;  $spFileCollection.Add("Documents/" + $i + ".docx",$file.OpenRead(),$false); $i++; }
while ($i -le 5000)

Now the Documents library contains the Aha.docx document plus 5,000 more documents, from 1.docx to 5000.docx.

SharePoint 2013 Public Beta - First Impressions

So, today I spent about an hour with the public beta of SharePoint 2013. My first impressions:

1) Blue top bar, having the Static word "SharePoint" (IMHO, it is just taking up valuable space!), followed by (on the right side) Newsfeed, SkyDrive, and Sites.
2) The long-running operations now have the "Sorry to keep you waiting." How cute.
3) However, too many errors result in the "Something went wrong" error messages - I wish they were more specific.
4) Borrowed a lot of ideas from NewsGator, such as Community Portal, Community Site, "following" sites, etc. I created a new Community Portal as well as another site collection (with "Team" being the top-level site and a "Community Site" being the only subsite.)
5) Navigation is not as easy as in SP 2010, the UI is a bit hard to follow, no breadcrumb navigation.
6) Icons where there used to be words. Sometimes it is helpful, but in many cases it is rather confusing.
7) Content is not just list, libraries, and sites, it now also has "apps" (a new buzzword from SharePoint 2013 lingo.) Oh, and lists and libraries are "apps," too.
8) Central Administration has not changed that much. Same UI as it was in SharePoint 2010. But when you create a new site, it has a little "Experience" dropdown box where you can choose whether you want the 2013 experience or the 2010 experience.
9) There are three or so more service applications (Machine Translation, Security Token, and App Management.)
10) Visual Studio 2012 now allows Visual WebParts in Sandbox solutions and provides designers for lists and content types, the Microsoft Fakes framework for testing SharePoint code (always a good idea), profiling tools for SharePoint projects, Intellisense for JavaScript, and a new item template for Project Item.

More About Expanding/Collapsing Web Parts

On June 19th, I posted about a suggested method to expand and collapse web parts on a page. In this method, a lonesome square with a plus or a minus is created, displayed in its own web part (CEWP, to be specific), and when you click on this little square, the corresponding web part is expanded or collapsed accordingly.

This is good, but what if you want some other content to be there next to that square with a plus or minus? Will this solution still work?

The answer is: it depends. If there is only static content, then the solution will definitely work, you just add more content to the CEWP. But if there is inline server code (or any server code), then this solution will not work, because CEWPs cannot process server code. In this case, the solution is a bit more complex.

Since you cannot put server code into a CEWP, we'll have to write a custom web part. This custom web part will have at least one personalizable (but not web browsable) property, named WebPartToHideID. The <div> element will have to be added as a panel, and the hyperlinks and images as the corresponding server controls, like this:

Note how the WebPartToHideID is passed as a parameter to the javascript methods.

Next, we need to generate the JavaScript containing the expand/collapse jQuery code. The GetClientScript() helper method generates the corresponding JavaScript, passing WebPartToHideID as a parameter when needed. Finally, both the panel and the script are added to the user control:

That's it. Now we have a web part that acts like the CEWP I posted about earlier, but is also able to contain server code.

Custom Pager for SPGridView - Part 3

Of course, our pager should be able to allow us to move to the previous page and to the next page. The following method accomplishes this:

Note that here, the target of the postback is the pager, not the gridview. The argument is "nextpage" for the next page and "previouspage" for the previous page. Here is how you call this method from the overridden Render() method:

Finally, our pager should have the dropdown list with the value of possible page sizes. So, we will have to trigger the postback from the <select> element.  This time, it will be trickier because the event argument will not be known until the element is actually selected - so, instead of calling the Page.ClientScript.GetPostBackEventReference() method, we will call __doPostBack() directly. The following chunk of code accomplishes this:

Now, all we have to do is to add our new gridview and pager to the user control of the visual Web Part. Make sure you set the pager's GridViewId property to the Id property of the gridview, and to implement the event handling method for the PageSizeChanged event (and for the PageIndexChanging event, of course.) And, in the OnPreRender() method of your user control, send the initial page size for your gridview (if the request is not a postback one.)

A little gotcha is that if you are putting your web part to a site page, make sure you assign that web part an ID in the <AllUsersWebPart> element. This will ensure your postbacks are always triggered for your gridview, since its client ID will not be changed.

Custom Pager for SPGridView - Part 2

Now that we have an enhanced gridview, we can associate a pager with it. SPGridViewPager comes to mind. The way it is displayed is not exactly what we want, because all it allows you to do is to move to the next page and to the previous page, and it displays the range of rows, not individual page numbers, like this:

There aren't any properties in SPGridViewPager to alter this display style. So, we need to create a new pager class, inheriting from SPGridViewPager.  Unfortunately, overriding the CreateChildControls() method in this class does nothing. So, our only hope is to override the Render() method:

This allows us to generate HTML on the fly, overriding the default rendering.

The problem we are having now is that we will have to trigger a postback from HTML elements. This is not so hard - that's what the Page.ClientScript.GetPostBackEventReference method is for. Here is how we can display individual page numbers (with the number in bold if the page is current, and a link to the correct page for non-current page numbers.)

In this method, a postback is triggered, with the gridview being the event target and "Page$" followed by a number being the event argument. The OOTB SPGridView knows how to raise a postback event for such an event argument.

Here I would like to remind everyone that the Page.ClientScript.GetPostBackEventReference(target,argument) method call is displayed in the HTML source as "javascript:__doPostBack(target, argument)."

To be continued...