UPDATE: 4/25/17: PowerShell to increase timeout included in workaround section.

Scenario:

You create a PowerPivot workbook with one or more data sources and upload it to SharePoint 2016. You then schedule the workbook to refresh and the refresh fails with the following error: “The operation has timed out“.

Cause:

This issue is caused by a change to the design of the product in SharePoint 2016 that limits a single PowerPivot data source to a refresh duration of 100 seconds (or 1:40). Note that this can also be caused by slow or underperforming data sources if you think your data should be refreshing in under 100 seconds. If you are pulling a small amount or data from a complex data source that takes a long time to query, you may consider alternative workarounds as increasing the timeout may not be the best option.

Symptoms:

1. If you expand the refresh history, you will notice that the data source of which refresh failed on ran for exactly 1:40 (100 seconds).
   
2. You may notice that some of your data sources actually succeed, or that the refresh succeeds intermittently if all of the data sources refresh in less than 100 seconds.
3. In the SharePoint ULS logs, you will see the actual error reported, but it will not seem to correlate to anything meaningful as far as a cause (because it doesn’t….)

   00/00/2017 14:49:18.44 w3wp.exe (0x00000) 0x59D4 PowerPivot Service Data Refresh 99 High EXCEPTION: System.Net.WebException: The operation has timed out     at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)     <<truncated stack>>

4. You will likely see a correlating event in the Application Event logs (EventID: 5214).
5. If you have multiple workbooks pulling from the same source, and one workbook is slowing the data source down causing the other queries to wait or slow (past the 100s), you may see multiple workbooks fail that do not normally fail.

Potential Workarounds:

1. Optimize your queries to run faster.
2. Query less data.
3. Add hardware to your data source to process queries faster.
4. Use PowerShell to increase the default timeout setting.
   • From a SharePoint 2016 Administrator enabled PowerShell prompt:

     $farm = Get-SPFarm
     #The timeout value is in milliseconds, so be very careful to not set it too low!!!
     $farm.Properties.Add("WopiProxyRequestTimeout", <new timeout value>);
     $farm.Update();
     #to double check the setting
     $farm.properties

   • To set this to a different value in the future:

     $farm = Get-SPFarm
     $farm.Properties.WopiProxyRequestTimeout = <new timeout value>
     $farm.update()

     Disclaimer: Increase this timeout value at your own risk! We realize that the new default timeout is very low, but you should also be aware of the amount of data that you are pulling vs the time it should take to pull that data. Setting this timeout too high and allowing users to pull large amounts of data can cause performance issues for PowerPivot, SharePoint and Office Online Server. You will also be limited by the default timeouts for SharePoint, SQL and your external data sources. This timeout value may not always be the answer. There are more workarounds and optimization options depending on the type of data you are pulling.

NOTE: This blog applies SPECIFICALLY to the issue listed above and ONLY to PowerPivot for SharePoint 2016 Scheduled Data Refresh. This may not be the only context that you see this error and you could also run into it in other versions of the product (but it will not be for the same reason!!).

If you have been following the multiple threads  on issues and changes to the PowerPivot for SharePoint 2016 Gallery Snapshots, you may be aware of an issue where gallery snapshots do not function properly (infinite “hourglass”) for SSL sites when there are multiple sites bound to the same port. While not limited to port 443 we will use a generic port 443 site as an example here as it is the most common usage.

I found this issue back in December 2016 during testing and again in a really well researched blog post by Brian Laws (http://info.summit7systems.com/blog/bug-report-powerpivot-for-sharepoint-2016-thumbnails-not-generating) in February of this year. After that, we started to see some cases roll in and I had a basis to start engaging our product team on the issue.

I will not go into extensive detail on the snapshot process or the log entries (you can see Brian’s blog above for most of it or follow along in the ULS logs), but here is the 10,000 foot view:

1. A user uploads a new PowerPivot workbook to the Gallery.
2. Event receiver detects the file and triggers detect that it is a file valid for a snapshot.
3. SharePoint calls the ExcelRest.aspx for each relevant item in the workbook via a URL similar to the following:
   •  https://localhost/_vti_bin/ExcelRest.aspx/PowerPivot%20Gallery/GenericWorkbook.xlsx/Model/Sheets(‘Sheet1’)
4. As part of the above call the full URL of the document is sent with the request to OOS on behalf of the user that uploaded the document. (Obviously “localhost” is not a valid URL for OOS, so we also send the full URL as part of the request so OOS knows where to go to retrieve the snapshots)
5. Office Online Server loads the workbook, retrieves the snapshots, and returns them to the SharePoint WFE server.
6. The snapshots are applied to the document.

The Issue:

The new snapshot process causes an issue for SSL sites that share a port and use unique certificates. When you combine multiple unique URLs each with their own unique SSL certificates and localhost calls, you wind up with IIS getting very confused on what certificate to pin to the request (since it is localhost, it takes the first one in line generally). In an effort to avoid an authentication double hop in a scenario where more than one SharePoint web front end (WFE) is involved, the product team opted to utilize localhost. If this was not in place, Kerberos Constrained Delegation (KCD) would need to be configured between the SharePoint WFE servers to avoid an overcomplicated configuration just for gallery snapshots to function properly.

In the SharePoint ULS logs, you will see a set of entries similar to the following detailing the failure:

00/00/2017 00:00:00:00 w3wp.exe (0xXXXX) 0x18A8 PowerPivot Service Unknown 46 Medium Capturing Report Gallery snapshot information from the following URL: https://localhost/_vti_bin/ExcelRest.aspx/PowerPivot Gallery/GenericWorkbook.xlsx/model/Sheets 37abef9d-92a8-20b9-17bb-d369b513965b

00/00/2017 00:00:00:00 w3wp.exe (0xXXXX) 0x18A8 PowerPivot Service Unknown 46 High Snapshot Exception: Unable to take snapshots or get details of the file: https://ssl.contoso.com/PowerPivot Gallery/GenericWorkbook.xlsx from the uri: https://localhost/_vti_bin/ExcelRest.aspx/PowerPivot Gallery/GenericWorkbook.xlsx/model/Sheets. 37abef9d-92a8-20b9-17bb-d369b513965b

00/00/2017 00:00:00:00 w3wp.exe (0xXXXX) 0x18A8 PowerPivot Service Unknown 46 High Snapshot Exception: Ensure localhost uris are allowed. 37abef9d-92a8-20b9-17bb-d369b513965b

00/00/2017 00:00:00:00 w3wp.exe (0xXXXX) 0x18A8 PowerPivot Service Report Gallery 99 High EXCEPTION: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. <<Truncated stack>> 37abef9d-92a8-20b9-17bb-d369b513965b

During my multiple attempts to narrow this down, I could see the snapshot process attempting to load the first site bound certificate in my list (for my “secondary” SSL site) instead of the certificate for ssl.contoso.com. Because of the localhost call, IIS is confused (obviously) about what certificate to hand out for the request. This issue is specific to farms where everything else is configured properly and functional, but it is not limited to just this scenario. This just happens to be the most common (and basic) configuration.

The product team has determined this is by design and will not be changed due to product limitations with SharePoint and Office Online Server.

“The Workaround”

• You can use a wildcard certificate. This is the only possible workaround we have found so far. With a wildcard certificate for your sites, you can utilize multiple SSL host header sites and your gallery snapshots will function (this has been tested and confirmed working). Again, this is assuming that everything else is configured properly for snapshots to function. We realize this is not viable for everyone, but at this point it is only option to keep sites SSL and have functional snapshots.
  • Note: Be sure that you research all aspects of using Wildcard certificates before implementing this solution so you are fully informed of the risks and rewards of using them.

Other Options

• You can restrict your usage to one SSL site  with host header and certificate and no IP specific bindings (site specific SSL certs and IP specific bindings usually go hand-in-hand).
• Stop using the PowerPivot Gallery all together. You can schedule data refreshes for PowerPivot workbooks in any document library as long as the PowerPivot solution is deployed to the web app and the features are activated on the site.
• Convert your SSL sites to HTTP. (Maybe even just the sites that host workbooks?)
• Set “All Documents” to the default view in the PowerPivot Gallery.
  1. Navigate to the list a site owner.
  2. Click “Library” > “Library Settings” from the ribbon menu.
     
  3. Scroll down to the “Views” section and click “All Documents”
     
  4. Check the “Make this the default view” box and click “OK”
     
• You can take the above a step further by deleting the Silverlight views in the PowerPivot Gallery.
  • (Do the “Set “All Documents” to the default view” section above first)
    1. Navigate to the list a site owner.
    2. Click “Library” > “Library Settings” from the ribbon menu.
       
    3. Scroll down to the “Views” section and click on any of the 3 PowerPivot Silverlight views “Gallery”, “Theatre” and “Carousel”.
       
    4. Click “Delete”.
       
• You can also Delete/Hide the extra views with PowerShell (sample scripts attached below):
  • Hide Script
  • Delete Script
    NOTE: For your current PowerPivot Galleries, they will continue to attempt to take snapshots even if you disable/delete the views. The only way to stop them from attempting to create snapshots is to replace the gallery with a normal document library. As long as you keep the PowerPivot features deployed to the site, you will still be able to schedule data refreshes as usual. We STRONGLY recommend that you test these scripts extensively in a development environment before using them on a farm that matters.

Something else to consider here is that PowerBI Report Server is being released soonish (Q2 this year is the current timeline). You can check out more info and try out the preview version already here: https://powerbi.microsoft.com/en-us/report-server/

For more information about changes to Gallery Snapshots in SharePoint 2016, see our post here: https://blogs.technet.microsoft.com/excel_services__powerpivot_for_sharepoint_support_blog/2017/01/20/changes-to-powerpivot-gallery-and-snapshots-in-sharepoint-2016/

This article covers an issue specific to PowerPivot Gallery Snapshots in SharePoint 2016 and does not refer to or reference any other products. If you see similar errors in other versions of SharePoint, the solutions/recommendations list here will not help you and are not relevant to your issue. The PowerShell scripts provided in this article are provided as samples for testing with no guarantee or warrantee by Microsoft.

You may receive the following error when running a scheduled data refresh of a PowerPivot workbook in SharePoint 2016:

The virtual path '/......xlsx' maps to another application, which is not allowed.

After this error occurs, the schedule gets disabled and the schedule settings are deleted. The schedule keeps getting disabled even if the following scheduled refreshes run successfully.

This error is caused by a bug in PowerPivot. The fix will be released in the sppowerpivot16.msi file in the Feature Pack for the upcoming release of SQL Server 2016 SP2.

You may encounter an issue with email notifications not getting sent for scheduled data refresh failures of PowerPivot workbooks in PowerPivot for SharePoint 2016.

This error is caused by a bug in PowerPivot. The fix will be released in the sppowerpivot16.msi file in the Feature Pack for the upcoming release of SQL Server 2016 SP2.

You may see the following error on when trying to schedule a data refresh of a PowerPivot workbook in SharePoint 2016:

Sorry, something went wrong.

The SharePoint ULS logs might show the following error:

UserProfileDBCache_WCFLogging :: ProfileDBCacheServiceClient.GetUserData threw exception: Access is denied.

This can occur when the account running the PowerPivot System Service in SharePoint (PowerPivot application pool) does not have permissions on the User Profile Service Application.

Add the required permissions using the following steps.

In SharePoint Central Administration, go to Application Management > Manage service applications, and select the User Profile Service Application. Then click on the Administrators button:

Then add the PowerPivot system service account and give Full Control:

Click OK, and then click the Permissions button:

Add the PowerPivot System Service account and give Full Control:

Click OK, and restart the PowerPivot System Service on all the SharePoint servers running this service.

We recently encountered an issue with the Excel Web Access Web Part we thought we needed to make the community aware of as the cause of the issue is not super evident.

Scenario:

You are attempting to render Excel workbooks in the Excel Web Access web part in SharePoint 2016 and are encountering the following error on page load: "We couldn't find the file you wanted".

You will notice that workbooks are rendering in the browser (view, edit and preview) are all working fine assuming there are no other issues.

ULS logging will not have anything helpful, but you may see errors similar to the following while SharePoint is attempting to engage the workbook:

04/18/2018 10:16:28.92 w3wp.exe (0x0000) 0x2D8C SharePoint Foundation WOPI ajrrk Medium ShouldProcessWopiTarget: WOPI is disabled and the request will be redirected to home url. 0000000-0000-0000-0000-000000000000
04/18/2018 10:16:28.92 w3wp.exe (0x0000) 0x2D8C SharePoint Foundation WOPI al0ix Medium GetWOPIRestrictedTargetUrl: Failed to get WOPI target. Error message: ' 0000000-0000-0000-0000-000000000000
04/18/2018 10:16:28.92 w3wp.exe (0x0000) 0x2D8C SharePoint Foundation General a5b15 Medium WacProxyHelper.ExecuteGetProxyUrl: Url was null or empty, throwing a path exception 0000000-0000-0000-0000-000000000000
04/18/2018 10:16:28.92 w3wp.exe (0x0000) 0x2D8C SharePoint Foundation General a5b16 Medium WacProxyHelper.GetExceptionForPathError: received an invalid path 'https://site/Documents/Excel Services Sample Workbook.xlsx', exception: ' 0000000-0000-0000-0000-000000000000

OOS ULS will have no messages related to this issue, because the request doesn't even make it to the server.

Cause:

The cause for this specific issue is due to missing or custom bindings for Office Online Server. Specifically, you are missing the "syndicate" binding. This binding is specific to all Excel extensions and allows them to be rendered in web parts. In a stock binding scenario, you would see this when you run get-spwopibinding:
Note that your "WopiZone" and "ServerName" may differ

Application : Excel
Extension : ODS
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : internal-https

Application : Excel
Extension : XLSB
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : internal-https

Application : Excel
Extension : XLSM
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : internal-https

Application : Excel
Extension : XLSX
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : internal-https

Application : Excel
Extension : ODS
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : external-https

Application : Excel
Extension : XLSB
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : external-https

Application : Excel
Extension : XLSM
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : external-https

Application : Excel
Extension : XLSX
ProgId :
Action : syndicate
IsDefaultAction : False
ServerName : <SERVERNAME>
WopiZone : external-https

Solution:
Add the "syndicate" action to your WOPI bindings by running the following from one of your SharePoint servers as a farm admin in an administrator enabled SharePoint PowerShell console:

new-spwopibinding -server oos2016ocsi -action syndicate

Note: This issue only affects the above mentioned products and may or may not be the only way to experience this specific error. At the time of this writing, we have only seen this happen in this specific scenario.

Scenario:

You install PowerPivot for SharePoint 2016 (or PowerBI Report Server) and use SQL 2017 as the data model server (Analysis Services in SharePoint mode). You have configured Office Online Server and all of the other dependencies to properly render workbooks and slice data, but you still receive the following error when you attempt the action: "We cannot locate a server to load the workbook Data Model"

Analysis:

In the ULS logs on the Office Online Server, you will see errors similar to the following while reproducing the error:

00/00/2018 11:11:11.94 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Monitorable Uncaught CLR exception crossing the Interop boundary: Microsoft.AnalysisServices.Streaming.ServerNotFoundException: There are no servers available or actively being initialized. at  <<TRUNCATED STACK FOR LENGTH>><<CORRELATION ID>> 

00/00/2018 11:11:11.94 w3wp.exe (0x0000) 0x0000 Excel Online External Data by6b8 Medium OLEDBConnection::InitConnection: There are no servers available or actively being initialized. <<CORRELATION ID>> 

00/00/2018 11:11:11.97 w3wp.exe (0x0000) 0x0000 Excel Online Excel Calculation Services akdn Verbose OperationSite.PrepareComplete: Caught Exception when running operation Microsoft.Office.Excel.Server.CalculationServer.Operations.ApplySlicerSelectionOperation, WebMethod: ApplySlicerSelection. Error was Id=NoStreamingServers; Microsoft.Office.Excel.Server.CalculationServer.OperationFailedException: We cannot locate a server to load the workbook Data Model. at Microsoft.Office.Excel.Server.CalculationServer.Operations.Operation.RunOperationAsync() at Microsoft.Office.Excel.Server.CalculationServer.Operations.OperationSite.PrepareComplete(PrepareAsyncArgs args) <<CORRELATION ID>>

Unfortunately, these errors will not help very much with the diagnosis other than the fact that it is telling you that data models cannot be created on the PowerPivot Analysis Services server. To find a root cause, a deeper look at the log is required and some background on Excel Services is needed.

When Excel Services boots up for the first time, it performs a series of checks against the defined PowerPivot instances prior to initialization. If one fails, that server is taken out of the pool of available servers. After the 3 main checks, it does a query for available memory. This is based on the performance counters applied to the server during the SSAS install. Because PowerPivot is so reliant on performance based counters, if the counters are missing and the memory check fails, the initialization of the server will also fail. This goes back to one of the errors we saw above: "There are no servers available or actively being initialized."

So, you may or may not see this list of checks in the logs at any given time, but if you run a restart-service wacsm command and are logging during the restart you will see the following (you may need to engage Excel somehow to get this to pop up):

00/00/2018 00:00:00.69 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Medium Checking Server Configuration (SERVERNAME\POWERPIVOT)
00/00/2018 00:00:00.69 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Medium --> Check Administrator Access (SERVERNAME\POWERPIVOT): Pass.
00/00/2018 00:00:00.69 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Medium --> Check Server Version (SERVERNAME\POWERPIVOT): Pass (14.0.1.439 >= 11.0.2800.0).
00/00/2018 00:00:00.69 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Medium --> Check Deployment Mode (SERVERNAME\POWERPIVOT): Pass.
00/00/2018 00:00:00.69 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Medium Check Server Configuration (SERVERNAME\POWERPIVOT): Pass.
00/00/2018 00:00:00.73 w3wp.exe (0x0000) 0x0000 Excel Online Data Model 27 Medium SSPM: Initialization failed on server SERVERNAME\POWERPIVOT: Microsoft.AnalysisServices.AdomdClient.AdomdErrorResponseException: The '\MSOLAP$POWERPIVOT:Memory\Memory Limit High KB' performance counter could not be found. System error -1073738824 <<TRUNCATED STACK>>

Cause:

The missing performance counters is a known product issue with SQL Analysis Services 2017 RTM, CU1 or CU2  if you install it in Tabular mode. Subsequently, if you install another instance on top of it or even uninstall/reinstall, the performance counters will still not be installed on the server.

Resolution:

This issue was resolved in SQL 2017 CU3. Our recommendation is just to update to the latest CU to bypass this issue all together:

https://www.microsoft.com/en-us/download/details.aspx?id=56128

Here is the official post from the SQL PG regarding the issue for more info:

https://support.microsoft.com/en-us/help/4056328/performance-counters-are-missing-after-the-installation-of-ssas-2017

Note: This article applies ONLY to the above mentioned products in this specific scenario. This product issue was found and resolved with the above mentioned patch and does not apply to any other versions of Office Online, PowerPivot or SQL Analysis Services. You may experience the same error in other forms for other reasons.