Programmatically Uploading a Large Number of Documents to a Library

The problem:

I need to see how well my document library performs when it has more than 5,000 documents. To do that, I need to generate more than 5,000 documents and upload them to the library. It would be nice to do it automatically.

The solution:

I have a simple Aha.docx file. First, I need to copy it 5,000 times. I chose to do it via PowerShell:

$path = "C:\Users\Administrator.TWS\Documents\TestDocs\Aha.docx"
$i = 1
do  { $newPath = "C:\Users\Administrator.TWS\Documents\TestDocs\" + $i + ".docx"; Copy-Item $path $newPath; $i++ }
while ($i -le 5000)

Now my TestDocs folder has the Aha.docx file plus 5,000 other files, and I need to upload them to the Documents library. Here is the corresponding PowerShell script:

$spWeb = Get-SPWeb -Identity https://main.my.server.com/sites/Boris/TheSiteIAmTesting
$spFolder = $spWeb.GetFolder("Documents")
$spFileCollection = $spFolder.Files
$path = "C:\Users\Administrator.TWS\Documents\TestDocs\Aha.docx"
$file = Get-ChildItem $path
$spFileCollection.Add("Documents/Aha.docx",$file.OpenRead(),$false)
#Now, let's copy 5000 other files
$i = 1
do { $path = "C:\Users\Administrator.TWS\Documents\TestDocs\" + $i + ".docx"; $file = Get-ChildItem $path;  $spFileCollection.Add("Documents/" + $i + ".docx",$file.OpenRead(),$false); $i++; }
while ($i -le 5000)

Now the Documents library contains the Aha.docx document plus 5,000 more documents, from 1.docx to 5000.docx.

SharePoint 2013 Public Beta - First Impressions

So, today I spent about an hour with the public beta of SharePoint 2013. My first impressions:

1) Blue top bar, having the Static word "SharePoint" (IMHO, it is just taking up valuable space!), followed by (on the right side) Newsfeed, SkyDrive, and Sites.
2) The long-running operations now have the "Sorry to keep you waiting." How cute.
3) However, too many errors result in the "Something went wrong" error messages - I wish they were more specific.
4) Borrowed a lot of ideas from NewsGator, such as Community Portal, Community Site, "following" sites, etc. I created a new Community Portal as well as another site collection (with "Team" being the top-level site and a "Community Site" being the only subsite.)
5) Navigation is not as easy as in SP 2010, the UI is a bit hard to follow, no breadcrumb navigation.
6) Icons where there used to be words. Sometimes it is helpful, but in many cases it is rather confusing.
7) Content is not just list, libraries, and sites, it now also has "apps" (a new buzzword from SharePoint 2013 lingo.) Oh, and lists and libraries are "apps," too.
8) Central Administration has not changed that much. Same UI as it was in SharePoint 2010. But when you create a new site, it has a little "Experience" dropdown box where you can choose whether you want the 2013 experience or the 2010 experience.
9) There are three or so more service applications (Machine Translation, Security Token, and App Management.)
10) Visual Studio 2012 now allows Visual WebParts in Sandbox solutions and provides designers for lists and content types, the Microsoft Fakes framework for testing SharePoint code (always a good idea), profiling tools for SharePoint projects, Intellisense for JavaScript, and a new item template for Project Item.

More About Expanding/Collapsing Web Parts

On June 19th, I posted about a suggested method to expand and collapse web parts on a page. In this method, a lonesome square with a plus or a minus is created, displayed in its own web part (CEWP, to be specific), and when you click on this little square, the corresponding web part is expanded or collapsed accordingly.

This is good, but what if you want some other content to be there next to that square with a plus or minus? Will this solution still work?

The answer is: it depends. If there is only static content, then the solution will definitely work, you just add more content to the CEWP. But if there is inline server code (or any server code), then this solution will not work, because CEWPs cannot process server code. In this case, the solution is a bit more complex.

Since you cannot put server code into a CEWP, we'll have to write a custom web part. This custom web part will have at least one personalizable (but not web browsable) property, named WebPartToHideID. The <div> element will have to be added as a panel, and the hyperlinks and images as the corresponding server controls, like this:

Note how the WebPartToHideID is passed as a parameter to the javascript methods.

Next, we need to generate the JavaScript containing the expand/collapse jQuery code. The GetClientScript() helper method generates the corresponding JavaScript, passing WebPartToHideID as a parameter when needed. Finally, both the panel and the script are added to the user control:

That's it. Now we have a web part that acts like the CEWP I posted about earlier, but is also able to contain server code.

Custom Pager for SPGridView - Part 3

Of course, our pager should be able to allow us to move to the previous page and to the next page. The following method accomplishes this:

Note that here, the target of the postback is the pager, not the gridview. The argument is "nextpage" for the next page and "previouspage" for the previous page. Here is how you call this method from the overridden Render() method:

Finally, our pager should have the dropdown list with the value of possible page sizes. So, we will have to trigger the postback from the <select> element.  This time, it will be trickier because the event argument will not be known until the element is actually selected - so, instead of calling the Page.ClientScript.GetPostBackEventReference() method, we will call __doPostBack() directly. The following chunk of code accomplishes this:

Now, all we have to do is to add our new gridview and pager to the user control of the visual Web Part. Make sure you set the pager's GridViewId property to the Id property of the gridview, and to implement the event handling method for the PageSizeChanged event (and for the PageIndexChanging event, of course.) And, in the OnPreRender() method of your user control, send the initial page size for your gridview (if the request is not a postback one.)

A little gotcha is that if you are putting your web part to a site page, make sure you assign that web part an ID in the <AllUsersWebPart> element. This will ensure your postbacks are always triggered for your gridview, since its client ID will not be changed.

Custom Pager for SPGridView - Part 2

Now that we have an enhanced gridview, we can associate a pager with it. SPGridViewPager comes to mind. The way it is displayed is not exactly what we want, because all it allows you to do is to move to the next page and to the previous page, and it displays the range of rows, not individual page numbers, like this:

There aren't any properties in SPGridViewPager to alter this display style. So, we need to create a new pager class, inheriting from SPGridViewPager.  Unfortunately, overriding the CreateChildControls() method in this class does nothing. So, our only hope is to override the Render() method:

This allows us to generate HTML on the fly, overriding the default rendering.

The problem we are having now is that we will have to trigger a postback from HTML elements. This is not so hard - that's what the Page.ClientScript.GetPostBackEventReference method is for. Here is how we can display individual page numbers (with the number in bold if the page is current, and a link to the correct page for non-current page numbers.)

In this method, a postback is triggered, with the gridview being the event target and "Page$" followed by a number being the event argument. The OOTB SPGridView knows how to raise a postback event for such an event argument.

Here I would like to remind everyone that the Page.ClientScript.GetPostBackEventReference(target,argument) method call is displayed in the HTML source as "javascript:__doPostBack(target, argument)."

To be continued...

Custom Pager for SPGridView - Part 1

The problem:

My customer requirement is that the web part has to have a gridview whose pager allows access to any given page, go to the next page, go to the previous page, and select how many rows are displayed per page.

The solution:

This seemingly simple problem is actually a little more complex than it seems. The issue is that SPGridView does not have an event that triggers when the page size changes. So, the first step will be to create a new class that inherits from SPGridView and contains the following three things:

1)  The delegate PageSizeChangeHandler;

2) The public event PageSizeChanged;

3) the overridden RaisePostBackEvent method which triggers the PageSizeChanged event if the event argument is "Rows$" followed by the number of rows (e.g., "Rows$5" for 5 rows.) The dollar sign is used as a separator to follow the guidelines for event arguments for gridviews:

Of course, there should also be a new class PageSizeChangeEventArgs (inheriting from System.EventArgs) with a public property NewPageSize and a public constructor that sets this property based on the argument constructor:

Now we have a gridview capable of raising a postback event when the number of rows to be displayed changes.

To be continued...

Capturing Web Analytics Data in a Custom Web Part

The problem:

I need to display the number of daily unique visitors to a site collection on a custom web part. Using the following code results in an array with the only element being "The specified user or domain was not found." The Web application uses claims authentication.


The solution:

What is causing this issue?

The problem is that the GetWebAnalyticsReportData() function takes the user login name from Thread.CurrentPrincipal.Identity.Name and passes it to SPWeb.DoesUserHavePermissions() - but SPSecurity.RunWithElevatedPrivileges does not change Thread.CurrentPrincipal.Identity.Name to the application pool account name. It does, however, change WindowsIdentity.GetCurrent() to the application pool account.

So, it is necessary to explicitly set Thread.CurrentPrincipal to the correct identity. Since my Web application uses claims authentication, here is a correct way to do this:

System.Threading.Thread.CurrentPrincipal = 

 ClaimsPrincipal.CreateFromIdentity(WindowsIdentity.GetCurrent());

Works like a charm!

Expanding/Collapsing Web Parts

The problem:

My requirements state that a certain web part on my page should be expandable/collapsible. When the user presses the "-" sign this web part is to collapse, whereas when she presses the "+" sign this web part is to expand. When the page first loads, this web part should be collapsed.

The solution:

We should use jQuery for this purpose. First, we need to find out the ID of the web part in question (it should start with "MSOZoneCell_", something like "MSO_ZoneCell_WebPartctl00_m_MyCoolWebPart") - view your web part page in the source and find the ID there.

Then, add a new Content Editor Web Part to your page. In the CEWP, in the HTML source, type this:

<div align="right" class="content" id="jLoadDiv"><a href="javascript:show(&#39;MSOZoneCell_WebPartctl00_m_MyCoolWebPart&#39;);"><img title="Show" id="plus" alt="Show" src="/_layouts/images/EXPAND.GIF"/></a> <a href="javascript:hide(&#39;MSOZoneCell_WebPartctl00_m_MyCoolWebPart&#39;);"><img title="Hide" id="minus" alt="Hide" src="/_layouts/images/COLLAPSE.GIF" style="display: none"/></a> </div>
<script type="text/javascript">

$("document").ready(function()
{
$('td[id*="MSOZoneCell_WebPartctl00_m_MyCoolWebPart"]').css("display", "none");
$('img[id*="minus"]').css("display", "none");
});
function show(webpartID)
{
$('td[id*="'+webpartID+'"]').css("display", "block");
$('img[id*="plus"]').css("display", "none");
$('img[id*="minus"]').css("display", "block");
}
function hide(webpartID)
{
$('td[id*="'+webpartID+'"]').css("display", "none");
$('img[id*="minus"]').css("display", "none");
$('img[id*="plus"]').css("display", "block");
}</script>

Make sure you set the Chrome to None on your CEWP.

That's it. This does the job. You can even export the CEWP and then add its contents (with the contents of the Content element being HTML-encoded - use  this tool to perform the encoding) to the page or elements file.

DateTime Format With Time Zones

The problem:

In my Visual Web Part, I have to display a current date in the following format:

1:29 pm EDT June 11, 2012

The usual way to display the current date in this format is via <%=DateTime.Now.ToString("H:MM tt K MMMM d, yyyy"). The problem with this approach, though, is that the time zone is displayed in the wrong format, more like

1:29 pm -04:00 June 11, 2012

The solution:

First, we need to find the acronym current time zone. The following two functions take  care of it:

Next, we need to split the formatted date string (where the white space is the delimiter), and replace the second element (i.e., "-04:00") with the proper acronym (i.e., "EDT.") The following code takes care of it:

Finally, in the code-behind for the user control for the Web Part, create a protected method (e.g., ToProperFormat()) and make it return DateTime.Now.ToProperFormat(). Then, in the ASCX file, call <%=ToProperFormat() %> . Note that calling the extension method directly from the ASCX file results in a compilation error unless you import the extension namespace.

Site Pages in Non-Team Sites

The problem:

I am creating a site from my site collection feature receiver; the site is not a team site, but it does need to have the Site Pages library. You might recall that on every team site this library is automatically provisioned, because of  the WikiPageHomePage feature; however, the problem is that this feature cannot be activated in non-team sites.

The solution:

Since I have to activate the feature in code, the best way to do it is to obtain the reference to the site that is being created and then call the EnsureSitePagesLibrary():

SPWeb web = ...
SPList list = web.Lists.EnsureSitePagesLibrary();

Note that, in a similar fashion, you can provision a Site Assets library in a non-team site - just call EnsureSiteAssetsLibrary(), rather than EnsureSitePagesLibrary().

Caching Web Part Content

The problem:

Here is a common scenario. You have a web part whose contents include a gridview, which is populated by going through all (or many) subsites and gathering the information from each of these sites. The gridview has to allow sorting and possibly even filtering.

The problem, though, is that sorting and filtering typically cause postback, which means you need to perform the expensive operation of loading the data again. If the data happens to be static (or relatively static), you wind up loading the same data over and over again, which results in performance deterioration.

The solution:

The solution depends on whether the data displayed the gridview is the same for every user.

a) If it is the same, then it will be worthwhile to use the Cache object and add the datatable (or any other data source object) to the cache if it is not already there. This will speed up the loading time for every user, since the cache will be populated only once. A couple of gotchas, though:

1) Make sure you place a lock around the code that accesses cache. Otherwise, if it takes 15 seconds to load the data table and many users are trying to access this page at the same time, then all of these users will be simultaneously trying to update the same cache object - and this would result in performance problems.

2) If the data occasionally changes, it would be wise to implement cache dependency, so that the cache is invalidated whenever the data changes.

b) If it is not the same, then use Session instead of Cache. But, make sure the session state is enabled. Another gotcha is that there is no expiration of the Session object, nor there is anything equivalent to cache dependency for Session objects - so you'll need to find some other way to clear the session object once the underlying data table changes.

Provisioning Site Pages

Today I'd like to talk about provisioning pages in a feature. Sometimes your customer wants to add some content to the pages you provision, and/or to have some web parts on them.. If that's the case, application pages will not do - they live on the file system (more specifically, LAYOUTS on the hive, or some subfolder of LAYOUTS), and you cannot add web parts to them.

Provisioning site pages using Visual Studio 2010 is a little harder than provisioning application pages - after all, there is no such an item template as "Site Page" or "Web Part Page." So, you will need to use a workaround.

First of all, you need to remember that when you provision files, you do so via modules. Luckily, there is such an item template. When you create a module, an Elements.xml file and a Sample.txt files are added to your Visual Studio 2010 project.

Rename Sample.txt to YourSitePage.aspx (or whatever name you'd like to use for your page), and add the following:

Note that in the PlaceHolderMain, I am creating a new web part zone, called MainWebPartZone (feel free to rename it if you don't like this name.) This is where the web part will live on this page.

Also add a class to your module and name it YourSitePage.aspx.cs.  This will be the code-behind class for your page.

The next step is configuring the module. If you want your user to treat your page as a library item, you need to provision it as ghostable in library. In the CDATA section, insert the full contents of the .webpart file (except the processing instruction!) In my example, the new page is provisioned in the Pages library.



Changing Toolbars For List Web Parts

The problem:

I need to always show the toolbar, MOSS 2007-style, on some of the list web parts on the site I am automatically provisioning. Unfortunately, there is no straightforward way to do it via the SharePoint API, for example through the SPWebPart or the SPView objects.

The solution:

I am not sure why Microsoft chose to omit this functionality from the object model, as this seems to be a pretty common requirement. But, it is possible to change the toolbar type if you use reflection. The code below does the trick. Note that I am passing "ShowToolbar" as a parameter to the "SetToolbarType" method.

Long Running Operations

The problem:

I have a long-running operation that is supposed to transfer to an error page in case of an error.

This code is perfectly valid. However, the error page (SPUtility.ErrorPage, which is just a synonym for error.aspx) it transfers to is too generic ("An error has occurred on the server") and cannot be customized with a more informative error message.

The solution:

You don't have to call the End() method for the long operation; if the result is erroneous, make a transfer to the error page using SPUtility.TransferToErrorPage, which takes an error message as a parameter. This code does the trick. Note also that it catches ThreadAbortException.

Adding a Positive Integer Column To a List Via PowerShell

The problem:

I need to add a column to my list via PowerShell; this column must be a positive integer.

The solution:

Here is a simple script to accomplish this:

$site = Get-SPSite "http://my_site_collection"
$web = $site.OpenWeb("my_site")
$list = $web.Lists["my_list"]
$spFieldType = [Microsoft.SharePoint.SPFieldType]::Integer

#The new field is not required, so the value of the third parameter is 0
$spFieldName = "PositiveNo"

$list.Fields.Add($spFieldName, $spFieldType, 0)
$list.Update()

#Now that we created a new field, let's change it so that it has a minimal value of 1 and no decimal places

$spField = $list.Fields.GetField($spFieldName)
$spField.MinimumValue = 1
$spField.DisplayFormat = "0"
$spField.Update()

#Need to add the new column to the list

$views = $list.Views["All Items"]
$views.ViewFields.Add($spFieldName)
$views.Update()

Flickering Progress Box in Outlook 2010 While Opening a Task

The problem:

A workflow assigned me a task. I opened the task email in Outlook 2010 and then pushed the "Open This Task" button on the ribbon. All I got was a flickering progress box. The task was never opened and it took me quite a while to get rid of the flickering progress box.

The solution:

This seems like a security issue (and could have been handled more gracefully by Outlook.) While normally I don't advocate editing system files on the hive, this is the case when it is necessary. I opened the 14\LAYOUTS\FormsServer.aspx and added the following line right after the <body> tag:


<SharePoint:FormDigest runat="server" />

This creates a security validation for the FormsServer.aspx page and thus gets rid of the annoying flickering issue.

Threaded View in Discussion Boards

The problem:

Our customer created a discussion board and a custom threaded view for this list and made this new threaded view a default one. Now he is no longer able to access this list. When he tries to do so, he gets a nasty and cryptic "Attempted to use an object that has ceased to exist. (Exception from HRESULT: 0x80030102 (STG_E_REVERTED))" error message. 

The solution:

The root cause of this is the misconfigured threaded view. Unfortunately, Microsoft made it too easy to make an error here. The threaded view (and the flat view, too, for that matter) do not work on top-level folders, therefore the only proper way to configure folders for the threaded view or the flat view is this:

To fix the problem I mentioned, the view that is not working must be removed and re-created - to get to the list settings, follow the "Content and Structure" link in the Site Administration, and then follow the "Edit Properties" link in the hovering menu.

Search Service Application

The problem:

I have a search service application in my farm, My only content source contains several  claims-aware SSL Web applications. Unfortunately, for some reason, the gatherer was not able to crawl them, throwing a 404 status, for no apparent reason. Deleting and re-creating the search service application did not help matters at all. The ULS logs did not show anything unusual, and neither did the IIS logs.

The solution:

Actually, it was more like a workaround that my co-worker and I invented. He has another SharePoint farm on the same domain. We made our farms trust each other, and then I made my farm consume his search service application. See http://technet.microsoft.com/en-us/library/ee704558.aspx on how exactly to do this. Then, I created my own content source and added my own crawl rule and registered my own custom security trimmer. After all these manipulations, the search worked!

Cannot Delete Files From Document Library

The problem:

I needed to delete several files from my document library. I tried to do it using the ribbon. Alas, nothing was deleted. Fiddler brought to my attention that the request was nioved; the ULS log informed me that a new request to the AccessDenied.aspx page was created. I never saw the AccessDenied.aspx page, so the results were not clear at all.

The solution:

I looked at the log again and learned that, when you are deleting the files using the ribbon, the Web request goes to the WCF client.svc web service (http://site_root/_vti_bin/client.svc/ProcessQuery).

My next step was checking the authentication for my Web application. It turned out that for this particular web application, Anonymous Authentication was disabled. When I enabled it back, everything started working.

Lesson learned:  Disabling anonymous authentication can bring some unpredictable effects.

Site Templates vs. Solutions

The problem:

We migrated our farm from MOSS 2007 to SharePoint 2010. Yesterday, he called us telling us that in his new environment, his new site templates are now saved in _catalogs/solutions/Forms, rather than _catalogs/wt/Forms where he expects them to see. He manages to access the _catalogs/wt/Forms path, but finds this gallery empty.

The solution:

Remember that in SharePoint 2010, there are no STP site templates! They are now replaced with solutions (which have WSP extensions) that live in _catalogs/solutions/Forms . Two gotchas there, though:

1) On migrated sites, the _catalogs/wt/Forms path is still valid. But, it now just points to a list, not a site template gallery. The STP files that lived there in MOSS 2007 are not migrated to SharePoint 2010, though, because, I repeat, there are no STP site templates in SharePoint 2010.

2) Even though there are no STP site templates in SharePoint 2010, there are STP list templates. This can cause some confusion. So, remember, STP files are only used as list templates in SharePoint 2010, not as site templates!

Adding a Converted File to the Same Document Library

The problem:

Sometimes, it is desirable to convert some document to PDF (or some other format) and do it automatically, whenever the document appears in the document library.

The solution:

First of all, there is no easy way to convert a document to PDF without using a third-party product. I use Muhimbi Converter for SharePoint (http://www.muhimbi.com/).

If you want to create a converted document whenever the source document appears in the document library, you need a list item receiver. Override the ItemAdded() and ItemUpdated() methods.

You will need to add a service reference to the Muhimbi Document Converter WCF service. Then, you will need to create a document converter service client to interact with that service. For more information, read

http://blog.muhimbi.com/2009/12/converting-office-files-to-pdf-format.html .

A little problem with this example is that it reads a file from the file system, rather than from the SharePoint library. You will need to get the byte array for the source document file. A good way to do it is:

SPFile spFile = copyItem.File;
string sourceFileName = spFile.Name;
byte[] sourceFile = spFile.OpenBinary();

The Convert() method takes this byte array, the open options, and the conversion settings as parameters and returns the byte array for the new file. Now you will need to add it as a new file to SharePoint. A good way to do it is:

spFolder = (copyItem.Folder == null ? copyItem.ParentList.RootFolder : copyItem.Folder);
string newFileUrl = string.Concat(spFolder.Url, "/", Path.GetFileNameWithoutExtension(sourceFileName), ".", conversionSettings.Format);
SPFile newFile = spFolder.Files.Add(newFileUrl, convFile);
newFile.Update();