Hybrid Email Moderation Part 1


This article talks about the importance to configure Remote Domain both on Exchange on premise and Exchange Online to control the format and type of email messages exchange between hosted mailbox users and on premise mailbox users.

This post is available on my new blog:


Config Manager SCCM 2012/R2 SUP Fix

Hi everyone,

I want to share with you a strange behavior that happened that day when you uninstall WSUS and SUP from a server and then you try to install them again.

Say for example that you have a dedicated server running WSUS and SUP on it, and you got to a problem that you could not solve, and you decided to remove WSUS and SUP and then install them again to solve the problem. It can happen.

The moment you remove WSUS and SUP, at the Config Manager console, all updates will appear as expired. The strange thing is that when you install WSUS and SUP again, the synchrronization between your SCCM server and WSUS/SUP will work without errors but nothing get synched. No single error.


  • Uninstall SUP and confirmed from the supsetup.log to that it’s been uninstalled.
  • Run “select * from WSUSserverlocations” and confirm that it has removed entry for SCA.
  • Backup/Export HKLM\SOFTWARE\Microsoft\SMS\Components\SMS_WSUS_SYNC_MANAGER and renamed it.
  • Install SUP and again get the SUPSetup.log to verify.
  • New HKLM\SOFTWARE\Microsoft\SMS\Components\SMS_WSUS_SYNC_MANAGER registry key created automatically.
  • Restart SMS_EXECUTIVE.
  • Synchronization started.

[Resolution provided by Microsoft Support]

SharePoint Workflow Designer Tips P5

View other parts:

I want to start this part by talking about the rule of abstraction. Again, it is best practice to not hard-code any values inside your workflow code. Your workflow code should read at execution time all needed configuration values from external source. This is what we talked about in the previous part of workflow designer tips.

There is another place where abstraction can be applied. Inside your workflow code, you may need to interact with user values and groups. Usually you interact with users and groups in two places:

  • When you assign a task, you usually assign it to user or a group of users.
  • When you send email notification, you send it to a mail enabled user or group.

First tip is to avoid working with users and instead work with groups. This is the first level of abstraction. When you work with groups, you can externally modify the membership of the group and the magic happens right away. Even if you are assigning a task for a single user, you can assign it to a group that contains only that user. Later when that user leave the company and there is a need to change the task assignee, you can just change the membership of that group.

The second part of abstraction is to create a separate SharePoint list called Workflow Subscriptions for example, that contains two columns:

  • Subscription. (Data Type: Single line of text)
  • Subscriber. (Data Type: Person or Group)

Now, you can fill that list with values you wish. For example, if your workflow is sending email notifications to a group of people upon finishing the process of a new hire, you could create for example a new SharePoint or Active Directory group called (New Hire Notification Group), that contains the people that should be notified upon a new hire, and then create an entry in the Workflow Subscriptions like this:

  • Subscription: New Hire Notification.
  • Subscriber: New Hire Notification Group.

Now inside your workflow designer, when you want to send email to notify people for a new hire, you do the following:

SharePoint Workflow Designer Tips P1 65554.JPG

The same applies if you want to assign a task to a group, you create an entry in the Workflow People list after you create a group called (New Hire First Approval Group):

  • Subscription: First Approval.
  • Subscriber: New Hire First Approval Group.

You can see that we talked in the previous part about the Configuration List and now we are talking about Workflow People list. The main difference between them is the data type of the columns. The Configuration list has two single line of text columns, while the Workflow People has one column with single line of text column, and the other is People of Group column. Together, these two lists will give you enough abstraction to remove any hard-coding from your workflow code, and makes your workflow code safe from direct changes.

SharePoint Workflow Designer Tips P4

View other parts:

In this part of the tips, I want to talk about abstraction concept. When you write a good workflow code and everything works just fine, you should not face a situation where you open the SharePoint designer to do some changes unless there is a major change in the logic of how the workflow works.

If you have a change management request list that is served by your workflow that requires three approvals, and you wrote the workflow code, you test your code, you put the workflow in production, and everyone is happy with it, then theoretically speaking, you should not have a case where you need to open the workflow code again. Even if one of the approvals is changed to someone else, this should not be a cause to change the workflow code.

This can be accomplished by introducing a level of abstraction, by not hard-coding values inside the workflow code, and by maintaining all configuration values in separate SharePoint list, that your workflow will read from in real time.

SharePoint Workflow Dashboard Tip 651681

So now, you shall create a SharePoint List called (Workflow Configuration) for example, with two columns:

  • Configuration Name (Single line of text)
  • Configuration Value (Single line of text)

You then start working on populating this list with any value that you may use in your workflow. Examples are:

  • If you assigning tasks, then you can put the task due date and task title in the configuration list.
  • If you are calling web service, then you can put the web service URL in the configuration list.
  • If you have any static values or counter values, you put them here also.
  • Any switch values, for example, you may have a variable that is named (SendNotificationEnabled) and if it is true, your workflow will send notification. You can read this variable from the configuration list. That way, you can change the variable value from the configuration list without opening the workflow code.

Inside the Workflow Designer, you can read values from the Configuration List:

SharePoint Workflow Dashboard Tip 4538

Just thing of this case with me. Your workflow is calling a web service, and you kept the web service variables in a confirmation list like we did here. If the URL of the web service is changed for any reason, then you can ask anyone to change it from the configuration list. No need for you to go and open the Workflow SharePoint designer, and do some changes, and hit the scary Publish button.

I usually do not keep any single hard-coded value in any of my workflow code. Usually, when I write a workflow code, I do not open the code again until there is major changes the affects the way the workflow logic is happening. Any other changes, or customization are all kept and maintained in a separate configuration list. This even include the subject of workflow email notifications.

Web Application Proxy P1

We all know that Microsoft is not investing in TMG and UAG the way it did before and those products are going out of support soon. On the other hand, information worker are now using Laptops and devices all the time and from everywhere.

The way i see it is that end users now have their BYOD devices, have a cool looking apps, and they want to use those cool apps to connect to corporate data without the need to RDP into the corporate network for example and use VDI or RDS Session Host experience.

On the other hand, while VPN and DirectAccess are great technologies for remote access, for IT administrators, those type of technologies will let the user IN or OUT. In other words, VPN solutions will either let you IN or OUT and it is hard to control where the user can go once he is IN.

Microsoft came with a new way for remote access called “Conditional Access”, and added flexible authentication methods to the solution. The solution is called “Web Application Proxy”.

Web Application Proxy (a.k.a WAP) is a new Windows Server 2012 R2 role service under RRAS server role, integrated into Windows Server Manager and RRAS admin experience.

Microsoft now announced another new category, which is conditional access reverse proxy called “Web Application Proxy”. This is not a new technology for Microsoft as they have TMG and UAG before. Whats new about this Web Application Proxy, is its unique interaction with Active Directory Federation Services and devices. Web Application Proxy is part of Windows Server and not a separate installation like TMG or UAG.

From Information worker perspective:

  • Access corporate apps from anywhere, on any device, Windows and non-Windows
  • SSO and native device/app experience

From IT Pro:

  • Selectively publish apps
  • control access per app, user, device, location
  • Better protection with pre-authentication (optional)
  • No change required in existing apps
  • No change required on devices (client-less)

WAP: Fundamental Services

  1. Reverse Proxy Services:
    • Network Isolation
    • Basic DOS : Throttling, queuing, session establishing before routing to backend
    • URL Translation
    • Selective Publishing: Per internal application endpoint
    • ADS Proxy Services
    • Web Protocols Only: HTTP,HTTPS
  2. Pre-authentication services:
    • Rich Policy : user + device identity, application identity, network location
    • MFA Options (multi factor authentication)
    • SSO

Network Topology

Web Application Proxy is usually located in your corporate DMZ with one network card or two network cars.

You can choose not to join it to your domain if you like, but if you want to use Kerberos constrained delegation method, then you have to join it to the domain.

Web Application Proxy can authentication requests before forwarding them to the back end applications (this is called Pre-Authentication), or it can just pass the traffic to the back end application without authentication (this is called Pass Through)

Web Application Proxy cannot live without ADFS becuase ADFS provides the following for Web Application Proxy:

  • Configuration Storage: WAP is a stateless firewall, and its configuration files are stored in the ADFS
  • Pre-Authentication: WAP uses ADFS for authentication.

Web Application Proxy 1

SharePoint Workflow History Data and Logs Tips – P2

We have talked in part one that the workflow logs can be classified as Audit Logs or Debug Logs. Audit logs have long retention and are used by auditors and security teams as a proof of a controlled process, while Debug logs have low retention and are used by the people maintaining the workflow for troubleshooting purposes and to track the execution state of the workflow at different execution times.

SharePoint Workflow Designer gives you a built-in way to through some log data to a history list using an action (Log to History List). I am not fan at all of using this way of logging for different reason.

Let us talk a little bit about the Log to History List and about the History list itself. By default, there is a hidden list that get created by default when you first create your first workflow in a site called History List. You can use this history list for different workflows or you can choose or create different history list for each workflow.

One of things I do not like in such history lists is that it accept only single string at a time. There is no other columns in that history list that you can benefit from. Not that this is a big limitation, but I do not like to be restricted with only sending a string at a time. It makes it hard to filter and analyze the log data as I will described later because of the lack of other columns.

Second thing is that a workflow can be associated with only one history list to be used for logging. Take this example: You have a workflow that calls a web service, and you want to log the status code or perhaps the returned value from that web service call. The only option you have here is to use the Log to History List. You also want to log other events during the workflow execution. Now, someone came to you and ask you to give a report of all web service calls and their return code for analysis. You have to go and open that hidden history list from SharePoint designer, and then what you will see? You will see a lot of lines without any ability to filter the logs related to the web service calls and to track them back to the list item that cause the service to start. My point is it is very hard to look at the history list and track certain events, or do any kind of filtering.

Also we talked about two types of logs, audit and debug logs. Your only option here is to use the Log to History List and through logs related to auditing and other log entries (debug data) for you to troubleshoot the workflow. Now when the auditors ask you to extract a report for a certain item, the logs are mixed between audit and debug. Also, you may want to keep log entries used for auditing for longer time than those used for debugging, which you cannot do in this case because both are saved in the same history list.

Things become more interesting when you read about the “Workflow Auto Cleanup“. It is not best practice to disable this job or change its duration by the way. This job will remove the association of the workflow tasks and history data after 60 days by default. You can read more about this here. But what about the need to keep audit data for one year for example?. What will do then? People will disable this timer job, but Microsoft keep saying it will affect the performance of the product or something. If Microsoft implemented this timer job then there is a reason.

I think for a professional workflows, especially those that requires auditing, you should not use the built in way (Log to History List). I will describe the way I prefer in the next part.

SharePoint Workflow Designer as RDP APP

If you are responsible of writing SharePoint workflows using the SharePoint designer, I want to share with you a small tip when it comes to using the SharePoint designer console.

Usually, you have SharePoint servers and perhaps Workflow Manager in your data center, and you may have installed the SharePoint designer at your machine and connect remotely in order to start coding workflows.

SharePoint Workflow Designer Tips P1 233622

What I do not like in this case is the dependency on the link.  Sometimes you work remotely from a hotel room connecting to unreliable wireless network, connecting VPN to your corporate network, opening the SharePoint designer console from your machine and opening a very big workflow definition code, do some modifications and hitting Publish. You do not know what will happen if the network connectivity is not reliable.

What I prefer is to have a Remote Desktop server in the data center to do administrative tasks for different thing. You can then install the SharePoint designer in that remote desktop server, and then you can log remotely to that server and open the SharePoint designer from there. That way, when you hit Publish workflow, the changes will be pushed from the terminal server to that SharePoint farm without depending on any unreliable connection. Furthermore, I also have exported the SharePoint designer as a remote web app and copy it to my desktop. Whenever I want to use the designer, I just open the RDP file in my machine, which will connect using RDP to the RDS server in the data-center and give me a great experience.

SharePoint Workflow Dashboard Tip 4269

Even if you connect from your hotel room, connecing via unreliable wireless network via VPN to your corporate network, you will RDP to that RDS server, use the SharePoint designer from there, open your big workflow code, do your changes, hitting Publish, and you do not worry about anything. In fact, you can close your VPN connection and the background, the SharePoint designer will take its time publishing your workflow changes without any networking issues.

SharePoint Workflow Designer Tips P3

View other parts:

Entities and flow charts

After planning the whole process with the right people, sit and start drawing flow charts of how the workflow will act and behave. This visualization will help you a lot when you start coding and testing your workflow implementation.

Sometime your solution will require couple of additional lists to host some data. Try to write down the columns names and types for such lists from the beginning and adopt a naming convention if possible.

What I do usually when I am planning a solution that requires workflows, that i draw the workflow flowcharts, and then for each list that I need to create, I draw an entity drawing. I like to think of Lists as Classes in .NET programming. So for example, If i need to create a list for auditing and one to hold some analysis information, I will draw the following entities, for each entity, I am listing the properties and types [column names]:

SharePoint Workflow Dashboard Tip 3 3232

Do not test with Privileged Accounts

When you start writing your workflow code, do testing along the way for each step. My tip here is to test your workflow process with a normal user account and not with your admin or any high privileged account. I cannot emphasis how important this is.

I worked on a complex workflow once and everything worked fine as I was testing with my admin account. As the deadline become closer, I asked someone to test the workflow process with his account, and NOTHING worked. I figured out what was the problem later of course. The point here is never to test with your account or any privileged account.

Create testing environment

This is also a very good thing to do and it is a must in the development world. If you are coding a simple workflow for a list or document library, then try to create a test list or document library and code your workflow on that test list/library first.

This becomes handy when you later want to do enhancements or change something on the workflow after people start using it for long time. You will be afraid to change a working and live workflow because you cannot know if your change will affect current working users.

Instead, if you have a test environment, you could introduce your change there, test it, and then carry it safely to the production list/library.

I worked once on a solution that requires a separate sub-site, with many lists and workflows. So I created a test sub-site, create all lists and workflows there, I then did my whole testing there, and once I confirmed everything is working fine, I started to create the live sub-site with the same lists and workflows I had in the test one.

Later on, when I want to introduce new changes, I would test them in the test sub-site first so that I do not interrupt the live environment. Sometimes, when you did bad changes, the whole workflow won’t run and people start calling you. So test your workflow changes in test environment first.

SharePoint Workflow Designer Tips P2

View other parts:

Use Comments

One of my favorites features in SharePoint Designer is the ability to add notes for you and others to later open the workflow code and understand what is happening.

You can add comments inside SharePoint Designer by adding an action called Add a Comment.

SharePoint Workflow Designer Tips P1 2322

It is always good practice to make sure to add comments that describes what you are doing inside your code.

Workflow Variables

When writing workflows using SharePoint Designer, you usually need to create many variables.

Make sure that all your variables have names that are descriptive. Adopt naming convention for your variables. If you are using web service calls, try to prefix the variables needed for the web service with something like (WebService…).

When you create an item in a list, workflow designer will automatically create a local variable of type GUID called (create) that represents the GUID of the item created by the workflow designer on that list.

SharePoint Workflow Designer Tips P1 2326

What I usually do is to create a variable with type GUID, and called it (MyReports GUID) and use it instead of the automatically created on. I then go and delete the variable called (Create). This way, my variables will be more descriptive.

SharePoint Workflow Designer Tips P1 23622

Sometime, when you are beginner in writing workflows using SharePoint designer, you will end up with many variables that are created and not used inside your code. Keep an eye on those orphan variables and delete them as soon as you can so that you do not confuse yourself. This will make it easier for you and others to review your code later and get an idea about what’s going on.

Office 365 and Group Moderation Tips

In the process of testing out Office 365 and Exchange hybrid configuration, an interesting thing happened that I want to share with you.

I have an on-premise Exchange 2010 implementation and couple of users are hosted at Office 365. All hybrid configurations are set and connectors are configured to route emails between the two spaces.

Everything is working fine, and mailboxes hosted on Office 365 are working just fine. Things started to get interesting when people start to send emails to moderated distribution groups.

When someone sends email to a moderated groups, and the moderator is hosted on Office 365, the buttons for Approve and Reject are not showing at his email client.

It turned out that a setting called TNEF (Transport Neutral Encapsulation Format) is causing this to happen. We need to make sure TNEF format is enabled when sending emails out to Office 365 tenant.

The TNEF setting is configurable per remote domain (Get-RemoteDomain) and (Set-RemoteDomain).

By default, there is a default RemoteDomain configured in your Exchange environment called (Default). If you hit (Get-RemoteDomain), you will see all settings that controls the behavior of email communications and format when sending emails to external parties. One of the settings is TNEFEnabled.

Now that we have Office 365 hybrid setup, the HCW creates for us a remote domain in the on-premise organization to allow TNEF (TenantName.mail.onmicrosoft.com).

That is great. So all what we need to do is to configure that remote domain (Set-RemoteDomain -TNEFEnabled ….) and all is done, right?

There is a small thing left to say. When Office 365 sends emails regarding moderated groups, the messages come from a system mailbox in the tenant with email address SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}@TenantName.onmicrosoft.com. So we need to add a new remote domain called TenantName.onmicrosoft.com

So let us start typing some PowerShell commands:

New-RemoteDomain -Name “Hybrid Domain –TenantName.onmicrosoft.com” -DomainName TenantName.onmicrosoft.com

Now we have two remote domains:

  • TenantName.onmicrosoft.com
  • TenantName.mail.onmicrosoft.com

We have then to configure both remote domains to allow TNEF format. I also recommend configuring many other settings on the way.

Set-RemoteDomain -Name “Hybrid*” -IsInternal $true -TargetDeliveryDomain $true -AllowedOOFType InternalLegacy -MeetingForwardNotificationEnabled $true -TrustedMailOutboundEnabled $true -TrustedMailInboundEnabled $true -UseSimpleDisplayName $true -TNEFEnabled $true

That’s great. Now we have configured both remote domains to enable TNEF format. I have also noticed that when on premise mailboxes try to communicate with the Office 365 system mailbox for moderation actions (Approve,Reject), they are receiving authentication errors. To fix that, add the TenantName.onmicrosoft.com to the address space of the Office 365 connector:

Set-SendConnector “Outbound to Office 365″ -AddressSpaces @{Add=”TenantName.onmicrosoft.com”}

Finally, it makes sense to instruct the Office 365 tenant to treat the on premise Exchange organization the same way. Suppose my on premise domain domain is contoso.com, then connect to your office 365 Exchange PowerShell, and type:

New-RemoteDomain -Name “Hybrid Domain – Contoso.com” -DomainName Contoso.com

Set-RemoteDomain -Name “Hybrid*” -IsInternal $true -TargetDeliveryDomain $true -AllowedOOFType InternalLegacy -MeetingForwardNotificationEnabled $true -TrustedMailOutboundEnabled $true -TrustedMailInboundEnabled $true -UseSimpleDisplayName $true -TNEFEnabled $true.

Reference Link 

Workflow Suspended HTTP 500 Security Token Issue

Hi everyone,

Recently, I was working on interesting case. Someone told me that workflows in SharePoint are all suspended.

The environment is SharePoint 2013 and separate Workflow Manager in different server.

Without any introductions, workflows are all showing suspended state with error:

Retrying last request. Next attempt scheduled in less than one minute. Details of last request: HTTP InternalServerError to https://lol.contoso.local/_vti_bin/client.svc/web/lists/getbyid(guid’da2febde-ff47-4b11-bd71-785b004dbcdb’)/Fields?%24select=EntityPropertyName%2CTypeAsString Correlation Id: 18a9b439-f602-e9f9-a2eb-97474f55dc86 Instance Id: 12de845f-9ebe-49d7-bfdb-7802199fcb89

Or something like:

HTTP 500 ID3242: The security token could not be authenticated or authorized.. bla bla bla.

I restarted every single SharePoint server, i even restarted the workflow manager server, go through event viewer on all servers without any luck. I checked if there are any new windows updates installed, and the answer is no.

The SharePoint farm is running Nov 2015 CU by the way.

Many people are talking about making sure that the Claims to Windows Token Service is started in SharePoint server. Well it is started.

After digging around, i found this event log on one of the back end SharePoint servers:

An operation failed because the following certificate has validation errors:

Subject Name: CN=SharePoint Security Token Service, OU=SharePoint, O=Microsoft, C=US
Issuer Name: CN=SharePoint Root Authority, OU=SharePoint, O=Microsoft, C=US
Thumbprint: B6036254C45D04F8173CD9598A44DD1EF1CF9C39


PartialChain: A certificate chain could not be built to a trusted root authority.
RevocationStatusUnknown: The revocation function was unable to check revocation for the certificate.
OfflineRevocation: The revocation function was unable to check revocation because the revocation server was offline.

So I went to certificate console on the server, Computer Store, SharePoint ,Certificates:

Sharepoint STS Token error 232523

All these three SharePoint certificates when you open them, it shows that it cannot verify the identity of the certificate. This means that the SharePoint cannot locate the trusted root certificate for those three certificates.

SharePoint has a PKI Root Certificate called (SharePoint Root Authority). All what I need is to locate it and then import it to the Trusted Root Certification Authority node in the Certificate Console that I am already opening.

To pull that Root Certificate (Reference Link), open SharePoint PowerShell session and type:

$rootCert = (Get-SPCertificateAuthority).RootCertificate
$rootCert.Export("Cert") | Set-Content C:\SharePointRootAuthority.cer -Encoding byte

That’s it. Everything start working fine after that. I had to import that trusted root certificate to all my SharePoint servers’s Trusted Root Authority node.

Charts and PowerShell !! PowerShell Wrapper “Full Edition”

I think one of the most interesting things when writing PowerShell scripts, is to make the results appear in a nice chart or couple of them. If you are going to show disk space info, nothing more than a nice chart will worth looking at. If you are sending a report to your management, nothing will take their attention more than charts. I love getting charts as  a high level output, with more tables and data for extra details.

Moving to a New Blog Platform

This post is now moved to my new blog platform at https://blog.ahasayen.com. To continue reading this blog post, please click here