.. _microsoft-sync-overview: Microsoft Sync Overview ------------------------- .. _21.4-PB1|EKB-15520: .. _21.4-PB5|EKB-18219: .. _21.4-PB5|VOSS-1313: .. _24.1|VOSS-1368: Overview .......... Automate interfaces with and syncs with a range of Microsoft applications, depending on the specific setup and use in your environment. The table describes how these applications sync in to Automate: .. tabularcolumns:: |p{3cm}|p{3cm}|p{9cm}| +--------------------------------+---------------------+-------------------------------------------------+ | Application | Synced in via | Description | +================================+=====================+=================================================+ | Microsoft Entra ID (Office365) | Graph API | Syncs in Msol users. Used for managing | | | | licensing. | +--------------------------------+---------------------+-------------------------------------------------+ | Microsoft Teams | PowerShell | Syncs in Csol users that have a matching Msol | | | | entry. Used for managing voice, voicemail, and | | | | collaboration. | +--------------------------------+---------------------+-------------------------------------------------+ | Microsoft Exchange Online | PowerShell | | +--------------------------------+---------------------+-------------------------------------------------+ .. note:: Microsoft has changed the name of Azure Active Directory to Microsoft Entra ID. .. rubric:: Related Topics * .. raw:: html Introduction to Data Sync .. raw:: latex Introduction to Data Sync in the Core Feature Guide * .. raw:: latex Controlling a Data Sync with a Model Type List in the Core Feature Guide .. raw:: html Controlling a Data Sync with a Model Type List * .. raw:: html Create a Targeted Model Type List .. raw:: latex Create a Targeted Model Type List in the Core Feature Guide * .. raw:: html Model Instance Filter .. raw:: latex Model Instance Filter in the Core Feature Guide * .. raw:: html Microsoft Quick Start Guide .. raw:: latex Microsoft Quick Start Guide in the Core Feature Guide * :ref:`azure-ad-o365-msoluser-filter-examples` Sync from Microsoft Entra ID / Office365 ......................................... Syncs from Microsoft Entra ID (Azure Cloud) sync in Msol users and are used to manage licensing. This sync fetches users and licenses from the Microsoft Entra ID instance for the tenant. This can either be the source AD or it could be synced with an on-prem AD which is the real source. A combination of sources is also possible depending on the setup. The licensing for cloud services is controlled on this user object in Microsoft Entra ID. Primary purpose of syncs with Microsoft Entra ID: * Provide the user objects and details that are in the Microsoft Entra ID environment, including existing settings and licenses. * Fetch license details for the Microsoft tenant - which licenses are valid for the tenant, current available/used statistics, as well as the service plans for each License SKU. The setup and timing of the sync can depend on how much of this data is required for your environment and how often you're viewing and using the data. For example, if you're using Automate to assign licenses to users as part of user onboarding, then syncing in users and licenses is important and needs to be timely in order to pick up new users in the system that need licenses. However, if licensing is being handled outside of Automate, this sync is less critical in terms of frequency. As this is a sync with the corporate AD, it is recommended that you apply a filter to control what's being synced in to the system. Filters can limit the number of users synced in to save load on the system, to reduce the time it takes for a sync to execute, and to enhance data security by ensuring that only relevant data (required for Automate's management purpose) is synced in. These types of filters are set up via model instance filters (MIFs) in Automate. The table describes common use cases for filtering in syncs: =================== =============================================================================== Geography Filter users by geography - certain cities and/or countries - potentially for a limited rollout of voice services in some geographies/sites, but not others. Licensing Filter users based on O365 licensing. You can have a simple filter that only pulls in licensed users (with any license), or you can have a more specific filter to identify licenses you want to match. The "IsLicensed" field is a filterable field for MsolUser and is the easiest filter (any license). Building a list of license IDs and permutations is more complex, but can be useful to narrow the sync down to a specific service (for example, those with licenses related to voice services such as E5, Phone System, or Calling plan). When contemplating this condition, consider whether you're managing users without a Phone System - for example, MS Exchange, or MS Teams users without Phone system, and so on - in which case a specific license filter may not be appropriate. Combination You can filter also use a combination of filters, that is, users from a certain geography, with an appropriate license. =================== =============================================================================== Automate includes example model instance filters for the scenarios outlined above. You can clone and use these model instance filters, and adjust as needed - including your City name(s), country name(s) or License SKU ids, for instance. Plan Your First Sync '''''''''''''''''''''''''' If your environment has more than a few thousand users, it is recommended that you plan your sync strategy and not just do a full sync. Use these guidelines to plan an approach based on the number of users you wish to pull in, and adjust the sync before you start pulling any users in - for example: 1. Start with a sync of licenses only (without users) to determine which licenses are in the environment, and the corresponding IDs, to set up in your model instance filters (if filtering by license). 2. Consider if your deployment will only include certain geographic regions and you can set up filters for those regions (maybe even starting with a single region initially to test and finalize setup). 3. Once you have your filters in place, run the sync, then adjust as needed. Sync from Microsoft Teams .......................... This sync pulls in a range of elements from the MS Teams tenant to allow Automate to manage the Microsoft Teams tenant. Fetched elements include, for example, users, voice, voicemail, collaboration, resource accounts, numbers, policies, and teams. .. note:: When syncing in data from ``msteamsonline/CsOnlineUser``, Automate sets the default maximum number of records fetched at 130 000. Paging is supported, and records are sorted on the ``UserPrincipalName`` (UPN). You'll need to contact VOSS support to reduce this value since a lower value may impact performance. Users are the main consideration for setting up this sync since creating and updating users is related to the licensing of users in Microsoft Entra ID. A user won't appear in MS Teams unless they have a relevant license assigned in Microsoft Entra ID/O365. Once a license is assigned, Microsoft automatically creates the users as a base user into MS Teams. Base users must be synced in to Automate before they can be managed, for example, to assign policies and enable services for these users. Default Model Instance Filter for CsOnlineUsers '''''''''''''''''''''''''''''''''''''''''''''''''' Automate v21.4-PB5 ships with a new default model instance filter, ``CsOnlineUserAutoFilter__[tenant_name]``, which is designed for Microsoft Teams syncs from ``msteamsonline/CsOnlineUser``. This model instance filter is set up at the same hierarchy as the tenant, and is used to sync in only those ``CsOnlineUsers`` with a ``UserPrincipalName`` (UPN) that matches the UPN of Msol users. The ``CsOnlineUserAutoFilter__[tenant_name]`` model instance filter specifies the UPN values of all Msol user instances in the cache to allow only those CsOnlineUsers with a UPN corresponding to a matching Msol user in the cache to be present in Automate. By default, the ``CsOnlineUserAutoFilter__[tenant_name]`` model instance filter is updated by events or by the sync workflow, whenever a Msol user is created, updated, deleted, or purged from the cache: .. tabularcolumns:: |p{5cm}|p{10cm}| +------------------------------------+----------------------------------------------------------+ | When ... | Then ... | +====================================+==========================================================+ | Msol user is created | The UPN is added to the model instance filter. | +------------------------------------+----------------------------------------------------------+ | Msol user is updated | The model instance filter is updated if the UPN changes. | +------------------------------------+----------------------------------------------------------+ | Msol user is deleted | The UPN is removed from the model instance filter. | +------------------------------------+----------------------------------------------------------+ | Msol user is purged from the cache | The UPN is removed from the model instance filter. | +------------------------------------+----------------------------------------------------------+ .. note:: When running the default model instance filter the first time, ``CsOnlineUser`` cache instances are purged or added (as applicable) so that only those ``CsOnlineUsers`` with UPNs matching an Msol user in the cache are present in the system. This will remove unused data, streamline the cache, and improve usability. Automate 21.4-PB5 ships with a tool that allows an admin user to reinitialize the model instance filter if it appears to be incorrect. You can also add additional filters to the ``CsOnlineUserAutoFilter__[tenant_name]`` model instance filter. For example, you could add "country equals US" to this filter so that the result syncs in all ``CsOnlineUser`` records that have a UPN matching an ``MsolUser`` in the cache, *AND* have a country that is "US". Default Data Syncs ''''''''''''''''''''' Automate syncs in all 0365 and Teams data via the default data syncs that are created when a tenant is added. No Model Type List (MTL) is applied to the default syncs, which means that Automate syncs in all supported data for the tenants, including all ``CSOnlineUsers``, that is, both enabled and disabled users (``accountEnabled=true`` and ``accountEnabled=false``). Since user accounts remain in MS Teams even when licensing to MS Teams is removed from the user or the user is removed from Microsoft Entra ID/O365, syncing in all users in the default sync allows Automate to identify any numbers currently assigned to disabled subscribers. This ensures that the number inventory is accurate, and prevents assignment of duplicate numbers. In addition, the Microsoft Entra ID (M365) data syncs have *Quick Import* enabled, which improves the performance of the sync. To limit the sync to enabled users only (``accountEnabled=true``), you can apply a model instance filter to future syncs; that is, set up a sync with a model instance filter by using ``accountEnabled`` equals ``true`` as a condition. You may also consider menu filters and other configuration to separate disabled accounts from enabled accounts. As with the Microsoft Entra ID sync, you may consider filtering for setting up the MS Teams sync in your environment. As the MS Teams integration currently uses PowerShell, there are additional considerations for filtering and setup to ensure the syncs are optimized to be efficient, and performance is acceptable. PowerShell Filtering Guidelines for CsOnlineUser and MsolUser Syncs ...................................................................... PowerShell supports filtering with the ``equals`` condition on some fields. It is recommended that you plan your filters and conditions based on these fields due to the filtering that happens on the Microsoft end. This will significantly improve the PowerShell response time and limits the amount of data being passed between the systems for processing. Syncs using these conditions are faster and consume less bandwidth. The table describes the fields that PowerShell supports for filtering, for ``CsOnlineUser`` and ``MsolUser``: ================================================= =============================================== ``CsOnlineUser`` filtering * City * Country * UsageLocation * Department * FeatureTypes * AccountEnabled * UserPrincipalName ``MsolUser`` filtering * IsLicensed * City * Country * Licenses.SkuId * UserPrincipalName ================================================= =============================================== .. note:: There are an additional 15 read-only extension attributes to filter on for Microsoft subscribers (when MS Exchange is installed). The values for these fields are customizable via your Active Directory server only (not in Automate), and are synced in with MS Entra ID via the MS Graph API. The values for these attributes can be modified in Automate via the following device models, provided the MS Entra ID Active Directory server is not synced with an On Premise Active Directory server: ``device/msexchangeonline/UserMailbox`` and ``device/msexchangeonline/SharedMailbox`` You can find more information about these On Premises Extension Attributes in the following topic: * .. raw:: html View and Update a Microsoft Subscriber .. raw:: latex View and Update a Microsoft Subscriber in the Core Feature Guide If your requirements can't be met via the fields above, you can filter on other fields on the ``CsOnlineUser`` and ``MsolUser`` objects. This filtering is applied to the results provided by PowerShell back to the system using the PowerShell filter. This results in the additional non-PowerShell criteria being treated as an ``AND`` criteria with the PowerShell filter. The filter is configured via the same model instance filter mechanism used for all other sync filtering. Automate takes the model instance filter and breaks it down to use it in the following way: * The model instance filter is checked for any criteria that uses the fields supported for PowerShell filtering. It builds the PowerShell command and issues it using those fields as a filter. * The condition is ``equals`` regardless of the condition set in the model instance filter for that field (PowerShell only supports the ``equals`` condition). * The result is compared against the Automate cache for users that have been added, updated, or deleted. * With this resulting list of users, any further criteria from the model instance filter is applied to the returned data. * Conditions for these fields are applied as configured in the model instance filter * This results in an ``AND`` condition between the PowerShell filter criteria and the "regular" model instance filter criteria. Although combining criteria in a model instance filter allows for less complexity and more predictable results, it is recommended that you avoid doing this, if possible. If you need to mix criteria, it is recommended that you test this carefully for expected results. For example, consider a filter with: * Filter on country (can happen in PowerShell filtering) and telephoneNumber (not in PowerShell filtering): * The first list of users from the device would be filtered based on country (PowerShell Filter) * Code does the compare to see the resulting set of users that are added/updated/deleted * telephoneNumber filter is then applied. * The result of the model instance filter is users where country and telephone number match .. note:: It is possible to filter by ``UserPrincipalName`` (UPN) in the model instance filter. Due to size limitations, this is implemented slightly differently in terms of capturing the list of ``UserPrincipalNames`` to filter. Some common user cases and scenarios taken individually or in combination. The criteria needs to be ``equals``, and multiple values can be provided - either as ``AND`` or as ``OR``, depending on the setup in the model instance filter. These cases can all be handled via PowerShell filtering so are the preferred use cases: ========================================== ======================================================================== Filter users by geography - Country Pull in users from a subset of countries in the environment Filter users by geography - UsageLocation Similar to country, to pull in users in certain UsageLocations Filter users by geography - City Possible but less efficient as still requires pulling all the users into Automate, then filtering by city. Thus, best if used in conjunction with another PowerShell filter field. Filter users by department Pull in users assigned to certain departments ========================================== ======================================================================== The system includes a few sample MIFs for these use cases. The samples can be cloned and adjusted to meet specific needs. Example model instance filters included in Automate: ==================================================== ============================================================== ``MSTeamsOnline-CsOnlineUser-MultipleCountry`` A simple model instance filter that provides an example of how to filter on users matching `United Kingdom` OR `United States` ``MSTeamsOnline-CsOnlineUser-CountryMultipleCity`` A model instance filter that includes an example of how to filter on users matching a country and multiple cities in the country. ==================================================== ============================================================== Allowlist and Denylist Considerations ...................................... Automate ships with allowlist entries for Microsoft Entra ID and MS Teams users to help optimize syncs for these applications. .. note:: You can find out more about allowlist and denylist data sync settings in the Core Feature Guide. You can modify the default allowlists and denylists for your scenario and environment, for example, to meet your flow through provisioning requirements. The predefined allowlist and denylist entries that ship with the system are typically suitable for most cases, and will account for the fact that sync workflows for a user may not be triggered even though a change for the user was pulled in during the sync. VOSS may update these default lists from time to time during upgrades if future features/functionality require it. .. image:: /src/images/admin-menubar-new.png Delete Protection for Syncs .............................. While there are a number of considerations for the setup of Microsoft syncs with Automate, it is also important to consider delete protection. In this case, the relevant macros for Microsoft Entra ID and MS Teams syncs are the following: * ``PULL_SYNC_DELETE_THRESHOLD_MSTeamsOnline`` * ``PULL_SYNC_DELETE_THRESHOLD_MSGraph`` These macros help to prevent mass deletes above a configured threshold defined in the macro (default 10). You can adjust the values for these macros initially, if required (depends on churn in your environment) or adjust them later as needed, based on the sync messages. .. note:: For more information on sync delete thresholds, see: .. raw:: html Pull Sync Delete Threshold .. raw:: latex the Pull Sync Delete Threshold topic in the Advanced Configuration Guide. .. _sync-teams-groups-large-tenants: Syncing Teams and Groups on Large Microsoft Tenants ..................................................... Microsoft Teams users often create ad-hoc Teams with channels, settings, and members. Microsoft tenants in large-scale deployments may thus contain hundreds of thousands of Groups as a Group is created in Microsoft Entra ID each time a Team is created. Additionally, Teams may contain many channels and members. Previously, syncing in all data for a large Microsoft tenant resulted in inefficient and time-consuming syncs that included a large volume of unnecessary details, including Team members and channels. To reduce the time is takes to sync in Teams and Groups, Automate syncs in only high-level details for the list view for Teams and Groups on the tenant, via the ``device/msgraph/Group`` model. Additional details for the Team or Group is fetched as a live update on demand (when an admin clicks on a Team or Group to view its details). .. note:: In Automate 24.1, separate models that existed for Teams (``relation/MicrosoftTeams``) and Groups (``device/msgraph/Groups``) are merged into one model, ``device/msgraph/Group``. Microsoft doesn't differentiate between Teams and Groups. In Automate, the **Teams** list view contains an *Is Teams* flag (green check) to indicate that the entity you're viewing is a Team and not a Group. .. image:: /src/images/ms-groups-list-view.png .. rubric:: Related Topics * .. raw:: html Teams .. raw:: latex Teams in the Core Feature Guide * .. raw:: html Groups .. raw:: latex Groups in the Core Feature Guide