.. _ms-syncs: Microsoft syncs ----------------- .. _25.1|EKB-21976: Overview .......... This topic provides an approach to syncs with *auto filtering* for Microsoft. Using filters in syncs ........................ It is recommended that you define a filtering strategy employing filtering criteria that can drive filtering on the Microsoft Graph API side. Applying the filter on the Microsoft side in this way returns matching users only, thus minimizing the number of records that need to be processed, and reducing sync time and load on the system. For example, let's say there are three hundred thousand users in your tenant and you only need fifty thousand users in the system to be managed. In this case, it is recommended that you apply API filtering to retrieve only the relevant fifty thousand users, while preventing import of the additional two hundred and fifty thousand records that you don't need. .. rubric:: Supported fields At the time of writing (for 25.2) the following fields are supported for Microsoft Graph API filtering: **CsOnlineUser** * UserPrincipalName (UPN) - also supports the *in* condition and is implemented to allow for as many UPNs as needed in the filter * SAMAccountName * City * Country * UsageLocation * Department * FeatureTypes - *equals* is interpreted as *contains*; that is, this featuretype is among the user's featuretypes * AccountEnabled **MsolUser** * UserPrincipalName (UPN) - currently only supports up until the maximum size of a URL and cannot be used for too many UPNs * City * Country * IsLicensed - returns users with any licenses * Licenses.Skuld - *equals* is interpreted as *contains*; that is, this license is among the user's licenses Building your MsolUser model instance filter (MIF) with these fields and using the required criteria ensures that you're using the API filtering in Microsoft Graph where possible. .. note:: Filtering on the Microsoft Graph API is case-insensitive, which helps to prevent errors where, for example, User Principle Names (UPNs) are not uniform. If filtering on a field or using criteria that is unsupported on the Microsoft Graph API, then this is applied within Automate after retrieving all records from Microsoft. .. rubric:: Filtering methods There are two approaches to filtering: * :ref:`auto-filter-csonlineuser` * :ref:`traditional-syncs-independent-filters` .. _auto-filter-csonlineuser: Auto filter - filter CsOnlineUser based on MsolUser ..................................................... The *Auto-filter feature for Teams/CsOnlineUser sync* filters CsOnlineUser based on MsolUser. Auto filter is the recommended default if filtering is used. Auto filter simplifies setup and operations and should be reviewed as a path to move to for existing customers/systems that might have implemented other filtering approaches prior to Automate 25.1. Auto filter provides the ability to focus filtering around the Microsoft Entra User (MsolUser) resources, which means that the system only pulls users from Teams (CsOnlineUser) if the user exists in Automate (filtering based on UPNs). As a result, you won't sync in any Teams users that haven't first been synced from Microsoft Entra. Sync setup and performance is streamlined, making it easier for admins to set up and manage filters on the syncs. Admins can build effective filters for various needs, based on the rich set of fields available on Microsoft Entra User. The Teams sync utilizes this data set rather than another filter, and excludes any non-essential data. Syncing with auto filter impacts only the ``device/mstteamsonline/CsOnlineUser`` model. No other Teams models are affected. Model instance filters and related filtering capabilities are supported for all other Microsoft Teams models, but not *CsOnlineUser*. To use the auto filter, you apply filters to the O365/MSOL user sync, which has more fields to filter on, and use the *Auto-filter feature for Teams/CsOnlineUser sync*. This will *not* support additional filters on the CsOnlineUser. The table describes the advantages and disadvantages of the *Auto-filter feature for Teams/CsOnlineUser sync*: .. tabularcolumns:: |p{9cm}|p{6cm}| +---------------------------------------------+----------------------------------------------------------+ | Advantages | Disadvantages | +=============================================+==========================================================+ | * Focuses filter activity on the user model | * It is not possible to apply additional CsOnlineUser | | that has the most fields to filter on - | filters to the sync if you're using the *auto filter*. | | * Easy to create a plan to "stamp" users | This is due to how filtering happens on the PowerShell | | if the base populated fields are | side, and the risk of unpredictable results. | | inappropriate. For example, end customers | | | can "stamp" users for managed services to | | | their MSP. | | | * Provides significant control for | | | managing the dataset size for various | | | reasons, such as performance or security | | | (to only see relevant data, for instance) | | | * Provides a more effective method for | | | limiting the Teams sync and improving | | | sync performance (not limited to | | | filtering based on CsOnlineUser fields) | | +---------------------------------------------+----------------------------------------------------------+ .. _traditional-syncs-independent-filters: Traditional syncs - independent filters ......................................... Traditional syncs (using independent filters) refer to the way that syncs have typically been set up and managed in Automate. In this method, the sync is set up with separate model instance filters for the Microsoft Entra and Teams side, and you would try to align these model instance filters where possible. This sync works when a field you want to filter on is the same on each model (for example, *City* or *Country*), but are often not the ideal fields to filter on. .. tabularcolumns:: |p{9cm}|p{6cm}| +---------------------------------------------+----------------------------------------------------------+ | Advantages | Disadvantages | +=============================================+==========================================================+ | Can offer more control in scenarios where | * Difficult for admins to set up, to ensure accuracy, | | there is a need to have different sets of | and to maintain | | users from Microsoft Entra compared to | * Inefficient performance | | Teams. | | | | Many filters need to be applied on Automate, which | | | means all users are pulled then filtered. | | | | | | * Often requires trade-offs (pulling in too much | | | data due to lack of effective fields to filter on) | +---------------------------------------------+----------------------------------------------------------+ Filtering and user types .......................... The following user types should be considered when defining filters, regardless of the approach you take: * Real users * Service accounts The table describes filtering considerations for the user types: .. tabularcolumns:: |p{5cm}|p{10cm}| +-------------------+------------------------------------------------------------------------------------+ | User type | Considerations | +===================+====================================================================================+ | Real users | Real users need to be in MS Entra to have Teams, so should not be an issue to have | | | the user come in via Entra and identify an appropriate filter. | | | | | | The goal is to minimize the users being pulled in for performance/security | | | reasons, so identifying a filter based on your data to best achieve this. | | | | | | This may be considered the base filter. | +-------------------+------------------------------------------------------------------------------------+ | Service accounts | The potential challenge with this user type is how these users fit into the | | | MsolUser filter. | | | | | | * **Resource accounts** - these often won't have the same field(s) populated as | | | real users (for example, City), as they are auto-created and not effectively | | | managed. Criteria therefore needs to be added to the filter to incorporate these | | | if they don't fit into the "real user" filter. At the highest level, these can | | | be filtered on the *Department* field to be synced in. This means all resource | | | accounts are pulled in instead of a potential relevant sub set. However, if the | | | Entra users for these accounts are tagged correctly then the filter can be | | | refined. | | | | | | If all resource accounts are pulled in, this is typically a far smaller set of | | | data and less sensitive than real users so the lack of potential filtering is | | | probably acceptable considering the admin effort of tagging those Entra users | | | correctly. | | | | | | * **Meeting Room and Common Area Phone users** - These instances will also have | | | Entra users for each instance. | | | | | | Typically, these are more like "real users" as they need to be explicitly | | | added, they usually have a site (location), and other business context that can | | | be added in Entra. Thus, if they align with the "real users" filter, nothing | | | further is required. However if they don't align with "real users", your base | | | filter may need to be extended to ensure these users are encapsulated. | | | | | | .. important:: | | | | | | The CsOnlineUser record for service accounts are pulled in from Teams to | | | enable accurate inventory and management of the user accounts. | +-------------------+------------------------------------------------------------------------------------+ .. rubric:: MsolUser filter logic in the model instance filter Your MsolUser filter logic in the model instance filter (MIF) should consider the following: * Criteria required for "real users" based on the fields supported on MsolUser. This can include fields populated naturally in the directory (such as City or Country), or by using a specific tagging approach (such as *ExtensionAttribute1 = Managed*) * Criteria for service accounts (resource accounts, meeting rooms, and common area phones) Where these can align to the "real user" filter, this is preferred. However, if alignment is not possible, additional criteria may need to be added to the filter (for example, *Department = Microsoft Communication Application Instance*). Recommended setup ...................... This section describes the recommended, baseline setup to get started with your sync scenarios, and may be applied in various scenarios, for example, Enterprise with a tenant, or Provider with multiple end customers. You can adjust the recommendations as needed. Setup recommendations are provided for: * MS365 syncs * Teams syncs MS365 syncs '''''''''''''' MS365 syncs focus on the resources that the system pulls via the MS Graph API, including tenant resources, such as Entra users, licenses, or groups, and limited Teams elements. This section will give some overall guidance on the suggested baseline sync setup and considerations. .. rubric:: User filtering (MsolUser) Unless you really need to see all Entra users in the system or the user base is very small, then some filtering should always be applied to the syncs for Entra users. This is to minimize the time and resources spent on syncing as well as to improve security as only relevant user data is in the system. Common scenarios to consider in your Model Instance filters (MIF) for MsolUser (Entra Users) to pull into the Automate system: * If you're handling Microsoft licensing outside of the Automate system, or if you're only applying incremental licenses, such as Calling Plan, MCOEV, or extra group licensing: * Apply the *isLicensed = true* option This option only syncs in users that have some sort of license assigned from Entra. Users without any licenses are excluded. * If you intend to use Automate to fully license users from scratch, then leave out this filter so that users without licenses are synced in to be visible for provisioning. * Additional criteria to match users required - some examples: * Specific license(s) - for example, E5 or MCOEV * City(s) * Country(s) * Field identified to be used for filtering - ExtensionAttribute1, Custom Attribute, other schema field - populated with some specific value, and so on. * System Accounts * Resource accounts - if the previous filters won't match your resource accounts, then adding additional filters for this. For instance, *Department = Microsoft Communication Application Instance* * Meeting Room/Common Area Devices - attempt to match your additional criteria for users above, but if not, include any additional criteria to identify these (could be by specific license, for instance). .. note:: * A default filter is applied on the API to only sync in users with *UserType = 'Member'* - this excludes external/guest accounts from the sync. * Consider the length of your filter when using fields that are filtered on the API (recommended). There is no documentation from Microsoft on the maximum length of the filter so testing will be key to ensuring your filter doesn't cause an error from Microsoft (typically indicates invalid filter in error message). If you need a long filter and it causes an error on the Graph API, you will need to break your MIF up into multiple syncs. For this reason, it's recommended that you avoid very long MIFs that have multiple different criteria, if possible. If using multiple criteria, ensure you match the logic needed - *AND* conditions compared to *OR* conditions. Multiple top-level criteria are treated as an *OR* condition where multiple inner level criteria are treated as *AND*. You can also set up multiple syncs around users to have the criteria separate, if needed. In this case, carefully test the *remove* and purge users that don't match the settings on the sync to ensure the behavior is as expected. The "auto-filter" capability will work even if you have multiple different user syncs set up around users. .. note:: Refer to the documentation on model instance filters for more information. .. rubric:: Recommended sync setup for MS365 The system automatically creates a *Full* and *User* sync variant. It is recommended that you ignore these syncs and instead set up the specific syncs needed for your setup. Auto-created syncs will reappear if you delete them and can potentially be updated during upgrades. The table describes the recommended syncs to build as a baseline, and for scheduling purposes: =============================== ================================================================================== Sync Description =============================== ================================================================================== M365 Sync Excluding users Use this sync for all elements except users as that will be handled via User sync. This can be run on a longer schedule (for example, once a week or month), or even adhoc when needed (for example, a new license is added to the system). M365 Sync User Add/Remove This will be the main sync setup that runs regularly to bring in new users or to remove users offboarded outside the system. As this focuses on just adding and removing users it keeps the sync load and runtime minimal. =============================== ================================================================================== .. note:: A sync that updates users is not been included since, typically this does not need to be scheduled. When opening users in the system, the latest settings are pulled in from Microsoft so it can be viewed that way. If you find a need to run update syncs regularly (for example, you want to utilize flow through provisioning to move users between sites based on updates, or many changes are being made outside the system), this can be set up and scheduled as needed. However, note that update scenarios are the most time-consuming in the sync as each individual record needs to be compared for changes. Additional syncs may be set up for adhoc administration use. In this case, ensure they have similar settings and the appropriate Model Instance Filter (MIF) if the sync involves users. .. rubric:: M365 Sync (excluding Users or excluding Users and Groups/Teams) This sync handles elements such as licenses, Teams, and Groups. The main purpose would be to bring in new instances or to remove old instances of these elements. Updates can be handled by opening the entity in the Admin Portal if or when the entity is managed. If there are additional elements that you don't want to sync in, you can build an appropriate Model Type List for those elements. However, the result should at least be that MsolUser is not part of this sync. The table describes the settings to use: .. note:: You can clone the auto-created M365 sync as a starting point. .. tabularcolumns:: |p{5cm}|p{10cm}| +----------------------------+------------------------------------------------------------------------------------+ | Setting | Description | +============================+====================================================================================+ | Model Type List | This should be the list of elements you want to exclude, and should be at least | | | Excluding Users OR alternative Model Type List | +----------------------------+------------------------------------------------------------------------------------+ | Refresh Existing | Set to *true* (checked) | +----------------------------+------------------------------------------------------------------------------------+ | Model instance filter | Unlikely to be required unless for some specialized need. | +----------------------------+------------------------------------------------------------------------------------+ | Disabled Actions | Update *true* (checked) | +----------------------------+------------------------------------------------------------------------------------+ | Purge Unmatched Records | *true* (checked) | +----------------------------+------------------------------------------------------------------------------------+ .. rubric:: M365 sync user Add/Remove: This sync pulls in new users and removes users that no longer match the filter (or were removed from Entra AD). The table describes the settings to use: .. note:: You can clone the auto-created M365 user sync as a starting point. .. tabularcolumns:: |p{5cm}|p{10cm}| +-------------------------------------+---------------------------------------------------------------------------+ | Setting | Description | +=====================================+===========================================================================+ | Model Type List | M365 User Data (with just msgraph/MsolUser in it) | +-------------------------------------+---------------------------------------------------------------------------+ | Refresh Existing | Set to *true* | +-------------------------------------+---------------------------------------------------------------------------+ | Model instance filter | The user MIF you set up with your user filter criteria | +-------------------------------------+---------------------------------------------------------------------------+ | Include the standard user workflows | | +-------------------------------------+---------------------------------------------------------------------------+ | Disabled Actions | Update *true* (checked). Note that in this case, you will need to | | | uncheck *Remove*. | +-------------------------------------+---------------------------------------------------------------------------+ | Purge Unmatched Records | Set to *true* (checked) | +-------------------------------------+---------------------------------------------------------------------------+ Teams '''''' These syncs focus on the resources that the system pulls via PowerShell from MS Teams, that is, most Teams and voice elements. This section provides overall guidance on the recommended baseline sync setup and considerations. .. rubric:: User filtering For user filtering, it is recommended that you use the *auto-filter* that will automatically only pull Teams users that the system has a corresponding MsolUser record for. This enables you to use filtering capabilities available on the MS 365 sync (see above) to also filter the Teams users. In this case, there is no additional filtering required. If you're not using the *auto-filter*, then, as for the MS365 sync, you will need to define an appropriate model instance filter to match the Teams users to sync in. This should only be done if you require different filter criteria for Teams than you have for Entra users. .. note:: The system automatically applies an additional filter over the *auto-filter* or Model Instance Filter (MIF) that is configured: * *AccountType*, for example, User or ResourceAccount - this ensures only these account types are synced in, which filters out elements such as external/guest accounts And; * *softdeletiontimestamp=null* - this eliminates the accounts that have been removed/deleted but still exist in Teams as 'soft deleted'. .. rubric:: Recommended sync setup for Teams The system automatically creates a *Full* and *User* sync variant. It is recommended that you ignore these sync variants and load/set up the specific syncs required for your setup. The auto-created syncs reappear if you delete them and can potentially be updated during upgrades. For this reason, they should be ignored. The table describes the recommended syncs to build as a baseline, and for scheduling purposes: =============================== ================================================================================== Sync Description =============================== ================================================================================== Teams Sync Excluding users This will be the sync to use for all elements except users as that will be handled via User sync. This can be run on a longer schedule (for example, once a week or month) or even adhoc when needed. Teams Sync User Add/Remove This is the main sync setup that runs regularly to bring in new users or to remove users offboarded outside of the system. As this focuses on just adding and removing users it keeps the sync load and runtime minimal. =============================== ================================================================================== .. note:: * A sync that updates users has not been included as typically this doesn't need to be scheduled. When opening users in the system, the latest settings are synced in from Microsoft so it can be viewed that way. If there is a need to run update syncs regularly (for example, when many changes are being made outside the system), this can be set up and scheduled as needed. However, update scenarios are most time-consuming in a sync as each individual record needs to be compared for changes. * You can set up additional syncs for adhoc administration use, if required. In this case, ensure they have similar settings and the appropriate Model Instance Filter (MIF) if the sync involves users. * For planning, (if not using *auto filter*), if you're using fields that can be filtered on the PowerShell side (for example, *City*), there is a limit to the length of filter that can be applied. This is an undocumented PowerShell limitation, and failure messages may be unclear (for example, they may refer to relevant messages about the filter to a message containing "Expected literal (number, boolean, or null)". Internal system testing indicate 97 conditions for a single field is the limit. For instance, 97 cities passed, whereas 98 cities caused a failure; 97 cities and 97 countries passed and 98 of either failed. For this reason, it is recommended that you test and validate scenarios for your tenant if you're going to require long filters. .. rubric:: Teams sync excluding users This sync handles all Teams elements, except users. The main purpose of this sync is to sync in new instances or to remove old instances of these elements (updates can be handled by opening the entity in the Admin Portal if/when they manage it). If there are additional elements that you don't want to sync in, you can build an appropriate Model Type List for this purpose. However the result should at least be that CsOnlineUser is excluded from this sync. The table describes the settings to use: .. note:: You can clone the auto-created Teams sync as a starting point. .. tabularcolumns:: |p{5cm}|p{10cm}| +-------------------------------------+---------------------------------------------------------------------------+ | Setting | Description | +=====================================+===========================================================================+ | Model Type List | To define the list of elements that should be excluded; should be at | | | least Excluding Users OR alternative Model Type List | +-------------------------------------+---------------------------------------------------------------------------+ | Refresh Existing | Set to *true* (checked) | +-------------------------------------+---------------------------------------------------------------------------+ | Model instance filter | Unlikely to be required, unless for some specialized need. | +-------------------------------------+---------------------------------------------------------------------------+ | Disabled Actions | Update *true* (checked) | +-------------------------------------+---------------------------------------------------------------------------+ | Purge Unmatched Records | Set to *true* (checked) | +-------------------------------------+---------------------------------------------------------------------------+ .. rubric:: Teams sync user add/remove This syncs in new users and removes users that no longer match the filter (or were removed from Teams). The table describes the settings to use: .. note:: You can clone the auto-created Teams user sync as a starting point. .. tabularcolumns:: |p{5cm}|p{10cm}| +-------------------------------------+---------------------------------------------------------------------------+ | Setting | Description | +=====================================+===========================================================================+ | Model Type List | Teams user data (contains just msteamsonline/CsOnlineUser) | +-------------------------------------+---------------------------------------------------------------------------+ | Refresh Existing | Set to *true* | +-------------------------------------+---------------------------------------------------------------------------+ | Model instance filter | If using *auto-filter*, leave blank. If not using *auto-filter*, select | | | your MIF for user filtering. | +-------------------------------------+---------------------------------------------------------------------------+ | Include the standard user workflows | | +-------------------------------------+---------------------------------------------------------------------------+ | Disabled Actions | Update *true* (checked). Note that you will need to uncheck *Remove* if | | | using a MIF. | +-------------------------------------+---------------------------------------------------------------------------+ | Purge Unmatched Records | Set to *true* (checked) | +-------------------------------------+---------------------------------------------------------------------------+ Scheduling syncs ................. Consider the following with regards to scheduling syncs: * MS365 user sync * Run and complete the MS365 user sync *before* running the Teams user sync. This ensures that any new or removed users in Entra are in the system and can drive the *auto-filter* behavior during the Teams user sync. * Run the MS365 user sync more frequently than the Teams sync to minimize delays for user onboarding. The MS365 sync brings users into Automate and can then be run through the onboarding process (whether admin initiated or via flow through provisioning). Typically, the Teams user is not required in the system to make onboarding possible. The onboarding workflow target syncs the Teams user in during the workflow. * Teams user sync * May not need to be run as frequently as the MS365 sync. * If you're performing many user transactions outside the system (such as assigning numbers), the sync may need to be more frequent, for example, to ensure the accuracy of the number inventory. Typically, updates won't need to be scheduled as the system updates data when records are opened. However, update syncs can be set up and scheduled as needed.