Added option ignorerole to gam update groups|cigroups <GroupEntity> sync [<GroupRole>|ignorerole] ... <UserTypeEntity>

docs/Cloud-Identity-Groups-Membership.md

@@ -178,14 +178,18 @@ testgroup@domain.com,testuser4@domain.com,MEMBER,Remove Failed,Does not exist
 
 ## Synchronize members in a group
 A synchronize operation gets the current membership for a group and does adds and deletes as necessary to make it match `<UserTypeEntity>`.
+This is done by specific role except for a special case where role is ignored.
 ```
-gam update cigroups <GroupEntity> sync [<GroupRole>]
+gam update cigroups <GroupEntity> sync [<GroupRole>|ignorerole]
         [usersonly|groupsonly] [addonly|removeonly]
         [notsuspended|suspended] [notarchived|archived]
         [expire|expires <Time>] [preview] [actioncsv]
         <UserTypeEntity>
 ```
-If `<GroupRole>` is not specified, `member` is assumed.
+If `ignorerole` is specified, GAM removes members regardless of role and adds new members with role MEMBER.
+This is a special purpose option, use with caution and ensure that `<UserTypeEntity>` specifies the full desired membership list of all roles.
+
+If neither `<GroupRole>` nor `ignorerole` is specified, `member` is assumed.
 
 When `<UserTypeEntity>` specifies a group or groups:
 * `usersonly` - Only the user members from the specified groups are added/deleted

docs/GamUpdates.md

@@ -10,6 +10,12 @@ Add the `-s` option to the end of the above commands to suppress creating the `g
 
 See [Downloads](https://github.com/taers232c/GAMADV-XTD3/wiki/Downloads) for Windows or other options, including manual installation.
 
+### 6.63.04
+
+Added option `ignorerole` to `gam update groups|cigroups <GroupEntity> sync [<GroupRole>|ignorerole] ... <UserTypeEntity>` that causes GAM
+to remove members regardless of role and add new members with role MEMBER. This is a special purpose option, use with caution
+and ensure that `<UserTypeEntity>` specifies the full desired membership list of all roles.
+
 ### 6.63.03
 
 Added option `externalusersallowed <Boolean>` to `gam <UserTypeEntity> create chatspace`

docs/Groups-Membership.md

@@ -213,8 +213,9 @@ testgroup@domain.com,testuser4@domain.com,MEMBER,Remove Failed,Does not exist
 
 ## Synchronize members in a group
 A synchronize operation gets the current membership for a group and does adds and deletes as necessary to make it match `<UserTypeEntity>`.
+This is done by specific role except for a special case where role is ignored.
 ```
-gam update group|groups <GroupEntity> sync [<GroupRole>]
+gam update group|groups <GroupEntity> sync [<GroupRole>|ignorerole]
         [usersonly|groupsonly] [addonly|removeonly]
         [notsuspended|suspended] [notarchived|archived]
         [remove_domain_nostatus_members]
@@ -223,7 +224,10 @@ gam update group|groups <GroupEntity> sync [<GroupRole>]
         (additionalmembers [<GroupRole>] <EmailAddressEntity>)*
         <UserItem>|<UserTypeEntity>
 ```
-If `<GroupRole>` is not specified, `member` is assumed.
+If `ignorerole` is specified, GAM removes members regardless of role and adds new members with role MEMBER.
+This is a special purpose option, use with caution and ensure that `<UserTypeEntity>` specifies the full desired membership list of all roles.
+
+If neither `<GroupRole>` nor `ignorerole` is specified, `member` is assumed.
 
 When `<UserTypeEntity>` specifies a group or groups:
 * `usersonly` - Only the user members from the specified groups are added/deleted

docs/How-to-Upgrade-from-Standard-GAM.md

@@ -330,7 +330,7 @@ writes the credentials into the file oauth2.txt.
 admin@server:/Users/admin/bin/gamadv-xtd3$ rm -f /Users/admin/GAMConfig/oauth2.txt
 admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam version
 WARNING: Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: /Users/admin/GAMConfig/oauth2.txt, Not Found
-GAMADV-XTD3 6.63.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.63.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.10.8 64-bit final
 MacOS High Sierra 10.13.6 x86_64
@@ -972,7 +972,7 @@ writes the credentials into the file oauth2.txt.
 C:\GAMADV-XTD3>del C:\GAMConfig\oauth2.txt
 C:\GAMADV-XTD3>gam version
 WARNING: Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: C:\GAMConfig\oauth2.txt, Not Found
-GAMADV-XTD3 6.63.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.63.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.11.4 64-bit final
 Windows-10-10.0.17134 AMD64

docs/Version-and-Help.md

@@ -3,7 +3,7 @@
 Print the current version of Gam with details
 ```
 gam version
-GAMADV-XTD3 6.63.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.63.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.11.4 64-bit final
 MacOS Monterey 12.6.6 x86_64
@@ -15,7 +15,7 @@ Time: 2023-06-02T21:10:00-07:00
 Print the current version of Gam with details and time offset information
 ```
 gam version timeoffset
-GAMADV-XTD3 6.63.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.63.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.11.4 64-bit final
 MacOS Monterey 12.6.6 x86_64
@@ -27,7 +27,7 @@ Your system time differs from www.googleapis.com by less than 1 second
 Print the current version of Gam with extended details and SSL information
 ```
 gam version extended
-GAMADV-XTD3 6.63.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.63.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.11.4 64-bit final
 MacOS Monterey 12.6.6 x86_64
@@ -64,7 +64,7 @@ MacOS High Sierra 10.13.6 x86_64
 Path: /Users/Admin/bin/gamadv-xtd3
 Version Check:
   Current: 5.35.08
-   Latest: 6.63.03
+   Latest: 6.63.04
 echo $?
 1
 ```
@@ -72,7 +72,7 @@ echo $?
 Print the current version number without details
 ```
 gam version simple
-6.63.03
+6.63.04
 ```
 In Linux/MacOS you can do:
 ```
@@ -82,7 +82,7 @@ echo $VER
 Print the current version of Gam and address of this Wiki
 ```
 gam help
-GAM 6.63.03 - https://github.com/taers232c/GAMADV-XTD3
+GAM 6.63.04 - https://github.com/taers232c/GAMADV-XTD3
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.11.4 64-bit final
 MacOS Monterey 12.6.6 x86_64

docs/_Sidebar.md

@@ -17,6 +17,7 @@ Installation
 Configuration
 * [Authorization](Authorization)
 * [GAM Configuration](gam.cfg)
+  * [Multiple Customers and Domains](https://github.com/taers232c/GAMADV-XTD3/wiki/gam.cfg#multiple-customers-and-domains)
 * [Running GAMADV-XTD3 securely on a Google Compute Engine](Running-GAMADV-XTD3-securely-on-a-Google-Compute-Engine)
 * [Using GAMADV-XTD3 with a delegated admin service account](Using-GAMADV-XTD3-with-a-delegated-admin-service-account)
 * [Using GAMADV-XTD3 with a YubiKey](Using-GAMADV-XTD3-with-a-YubiKey)

src/GamCommands.txt

@@ -3442,7 +3442,7 @@ gam update group|groups <GroupEntity> delete|remove [<GroupRole>]
         [notsuspended|suspended] [notarchived|archived]
         [preview] [actioncsv]
         <UserItem>|<UserTypeEntity>
-gam update group|groups <GroupEntity> sync [<GroupRole>]
+gam update group|groups <GroupEntity> sync [<GroupRole>|ignorerole]
         [usersonly|groupsonly] [addonly|removeonly]
         [notsuspended|suspended] [notarchived|archived]
         [removedomainnostatusmembers]
@@ -3649,7 +3649,7 @@ gam update cigroups <GroupEntity> delete|remove [<GroupRole>]
         [notsuspended|suspended] [notarchived|archived]
         [preview] [actioncsv]
         <UserTypeEntity>
-gam update cigroups <GroupEntity> sync [<GroupRole>]
+gam update cigroups <GroupEntity> sync [<GroupRole>|ignorerole]
         [usersonly|groupsonly] [addonly|removeonly]
         [notsuspended|suspended] [notarchived|archived]
         [expire|expires <Time>] [preview] [actioncsv]

src/GamUpdate.txt

@@ -2,6 +2,12 @@
 
 Merged GAM-Team version
 
+6.63.04
+
+Added option `ignorerole` to `gam update groups|cigroups <GroupEntity> sync [<GroupRole>|ignorerole] ... <UserTypeEntity>` that causes GAM
+to remove members regardless of role and add new members with role MEMBER. This is a special purpose option, use with caution
+and ensure that `<UserTypeEntity>` specifies the full desired membership list of all roles.
+
 6.63.03
 
 Added option `externalusersallowed <Boolean>` to `gam <UserTypeEntity> create chatspace`

src/gam/__init__.py

@@ -120,7 +120,6 @@ import google.oauth2.service_account
 import google_auth_oauthlib.flow
 import google_auth_httplib2
 import httplib2
-import urllib3.exceptions
 
 httplib2.RETRIES = 5
 
@@ -8952,10 +8951,10 @@ def doCheckConnection():
     ip = 'unknown'
     try:
       ip = socket.getaddrinfo(host, None)[0][-1][0] # works with ipv6
-    except socket.gaierror as err:
+    except socket.gaierror as e:
-      dns_err = f'{not_okay}\n   DNS failure: {err}\n'
+      dns_err = f'{not_okay}\n   DNS failure: {str(e)}\n'
     except Exception as e:
-      dns_err = f'{not_okay}\n   Unknown DNS failure: {err}\n'
+      dns_err = f'{not_okay}\n   Unknown DNS failure: {str(e)}\n'
     check_line = f'Checking {host} ({ip}) ({try_count}/{host_count})...'
     writeStdout(f'{check_line:<100}')
     flushStdout()
@@ -29123,7 +29122,7 @@ GROUP_PREVIEW_TITLES = ['group', 'email', 'role', 'action', 'message']
 #	[notsuspended|suspended] [notarchived|archived]
 #	[preview] [actioncsv]
 #	<UserTypeEntity>
-# gam update groups <GroupEntity> sync [<GroupRole>]
+# gam update groups <GroupEntity> sync [<GroupRole>|ignorerole]
 #	[usersonly|groupsonly] [addonly|removeonly]
 #	[notsuspended|suspended] [notarchived|archived]
 #	[removedomainnostatusmembers]
@@ -29159,8 +29158,11 @@ def doUpdateGroups():
     entityActionNotPerformedWarning([entityType, group, Ent.ROLE, role], Msg.INVALID_ROLE.format(','.join(sorted(GROUP_ROLES_MAP))), i, count)
     return (None, None)
 
-  def _getRoleGroupMemberType(defaultRole=Ent.ROLE_MEMBER):
+  def _getRoleGroupMemberType(defaultRole=Ent.ROLE_MEMBER, allowIgnoreRole=False):
-    role = getChoice(GROUP_ROLES_MAP, defaultChoice=defaultRole, mapChoice=True)
+    if not allowIgnoreRole or not checkArgumentPresent(['ignorerole']):
+      role = getChoice(GROUP_ROLES_MAP, defaultChoice=defaultRole, mapChoice=True)
+    else:
+      role = Ent.ROLE_ALL
     groupMemberType = getChoice({'usersonly': Ent.TYPE_USER, 'groupsonly': Ent.TYPE_GROUP}, defaultChoice='ALL', mapChoice=True)
     return (role, groupMemberType)
 
@@ -29702,7 +29704,8 @@ def doUpdateGroups():
                                                            checkForCustomerId=True) for member in removeMembers],
                                  role)
   elif CL_subCommand == 'sync':
-    baseRole, groupMemberType = _getRoleGroupMemberType()
+    baseRole, groupMemberType = _getRoleGroupMemberType(allowIgnoreRole=True)
+    ignoreRole = baseRole == Ent.ROLE_ALL
     syncOperation = getSyncOperation()
     isSuspended, isArchived = _getOptionalIsSuspendedIsArchived()
     removeDomainNoStatusMembers = checkArgumentPresent('removedomainnostatusmembers')
@@ -29780,7 +29783,8 @@ def doUpdateGroups():
         result = callGAPIpages(cd.members(), 'list', 'members',
                                pageMessage=getPageMessageForWhom(),
                                throwReasons=GAPI.MEMBERS_THROW_REASONS, retryReasons=GAPI.MEMBERS_RETRY_REASONS,
-                               groupKey=group, roles=None if Ent.ROLE_MEMBER in rolesSet else memberRoles,
+                               groupKey=group,
+                               roles=None if Ent.ROLE_MEMBER in rolesSet or ignoreRole else memberRoles,
                                fields='nextPageToken,members(email,id,type,status,role)',
                                maxResults=GC.Values[GC.MEMBER_MAX_RESULTS])
       except (GAPI.groupNotFound, GAPI.domainNotFound, GAPI.domainCannotUseApis, GAPI.invalid, GAPI.forbidden):
@@ -29791,7 +29795,7 @@ def doUpdateGroups():
         currentMembersMaps[role] = {}
         domainNoStatusMembersSets[role] = set()
       for member in result:
-        role = member.get('role', Ent.ROLE_MEMBER)
+        role = member.get('role', Ent.ROLE_MEMBER) if not ignoreRole else Ent.ROLE_ALL
         email, memberStatus = _getMemberEmailStatus(member)
         if groupMemberType in ('ALL', member['type']) and role in rolesSet:
           if not removeDomainNoStatusMembers or memberStatus != 'NONE':
@@ -29810,11 +29814,11 @@ def doUpdateGroups():
                                    [currentMembersMaps[role].get(emailAddress, emailAddress) for emailAddress in currentMembersSets[role]-syncMembersSets[role]],
                                    role)
       if syncOperation != 'removeonly':
-        for role in [Ent.ROLE_OWNER, Ent.ROLE_MANAGER, Ent.ROLE_MEMBER]:
+        for role in [Ent.ROLE_OWNER, Ent.ROLE_MANAGER, Ent.ROLE_MEMBER, Ent.ROLE_ALL]:
           if role in rolesSet:
             _batchAddGroupMembers(group, i, count,
                                   [syncMembersMaps[role].get(emailAddress, emailAddress) for emailAddress in syncMembersSets[role]-currentMembersSets[role]],
-                                  role, delivery_settings)
+                                  role if role != Ent.ROLE_ALL else Ent.ROLE_MEMBER, delivery_settings)
   elif CL_subCommand == 'update':
     baseRole, groupMemberType = _getRoleGroupMemberType(defaultRole=None)
     isSuspended, isArchived = _getOptionalIsSuspendedIsArchived()
@@ -31762,7 +31766,7 @@ def doCreateCIGroup():
 #	[notsuspended|suspended] [notarchived|archived]
 #	[preview] [actioncsv]
 #	<UserTypeEntity>
-# gam update cigroups <GroupEntity> sync [<GroupRole>]
+# gam update cigroups <GroupEntity> sync [<GroupRole>|ignorerole]
 #	[usersonly|groupsonly] [addonly|removeonly]
 #	[notsuspended|suspended] [notarchived|archived]
 #	[expire|expires <Time>] [preview] [actioncsv]
@@ -31798,8 +31802,11 @@ def doUpdateCIGroups():
     entityActionNotPerformedWarning([entityType, group, Ent.ROLE, role], Msg.INVALID_ROLE.format(','.join(sorted(GROUP_ROLES_MAP))), i, count)
     return (None, None)
 
-  def _getRoleGroupMemberType(defaultRole=Ent.ROLE_MEMBER):
+  def _getRoleGroupMemberType(defaultRole=Ent.ROLE_MEMBER, allowIgnoreRole=False):
-    role = getChoice(GROUP_ROLES_MAP, defaultChoice=defaultRole, mapChoice=True)
+    if not allowIgnoreRole or not checkArgumentPresent(['ignorerole']):
+      role = getChoice(GROUP_ROLES_MAP, defaultChoice=defaultRole, mapChoice=True)
+    else:
+      role = Ent.ROLE_ALL
     groupMemberType = getChoice({'usersonly': Ent.TYPE_USER, 'groupsonly': Ent.TYPE_GROUP}, defaultChoice='ALL', mapChoice=True)
     return (role, groupMemberType)
 
@@ -32136,7 +32143,8 @@ def doUpdateCIGroups():
             _showFailure(group, memberEmail, role, str(e), j, jcount)
         Ind.Decrement()
   elif CL_subCommand == 'sync':
-    baseRole, groupMemberType = _getRoleGroupMemberType()
+    baseRole, groupMemberType = _getRoleGroupMemberType(allowIgnoreRole=True)
+    ignoreRole = baseRole == Ent.ROLE_ALL
     syncOperation = getSyncOperation()
     isSuspended, isArchived = _getOptionalIsSuspendedIsArchived()
     expireTime = _getExpireTime(baseRole)
@@ -32213,25 +32221,25 @@ def doUpdateCIGroups():
         currentMembersMaps[role] = {}
       for member in result:
         getCIGroupMemberRoleFixType(member)
-        role = member['role']
+        role = member['role'] if not ignoreRole else Ent.ROLE_ALL
         email = member.get(CIGROUP_MEMBERKEY, {}).get('id', '')
         if groupMemberType in ('ALL', member['type']) and role in rolesSet:
           cleanAddress = _cleanConsumerAddress(email, currentMembersMaps[role])
           currentMembersSets[role].add(cleanAddress)
           currentMembersNames[cleanAddress] = member['name']
       del result
-      if syncOperation != 'removeonly':
-        for role in [Ent.ROLE_OWNER, Ent.ROLE_MANAGER, Ent.ROLE_MEMBER]:
-          if role in rolesSet:
-            _batchAddGroupMembers(parent, i, count,
-                                  [syncMembersMaps[role].get(emailAddress, emailAddress) for emailAddress in syncMembersSets[role]-currentMembersSets[role]],
-                                  role, expireTime)
       if syncOperation != 'addonly':
         for role in rolesSet:
           _batchRemoveGroupMembers(parent, i, count,
                                    [{'name': currentMembersNames[emailAddress],
                                      'email': currentMembersMaps[role].get(emailAddress, emailAddress)} for emailAddress in currentMembersSets[role]-syncMembersSets[role]],
                                    role)
+      if syncOperation != 'removeonly':
+        for role in [Ent.ROLE_OWNER, Ent.ROLE_MANAGER, Ent.ROLE_MEMBER, Ent.ROLE_ALL]:
+          if role in rolesSet:
+            _batchAddGroupMembers(parent, i, count,
+                                  [syncMembersMaps[role].get(emailAddress, emailAddress) for emailAddress in syncMembersSets[role]-currentMembersSets[role]],
+                                  role if role != Ent.ROLE_ALL else Ent.ROLE_MEMBER, expireTime)
   elif CL_subCommand == 'update':
     baseRole, groupMemberType = _getRoleGroupMemberType()
     isSuspended, isArchived = _getOptionalIsSuspendedIsArchived()
@@ -55108,7 +55116,8 @@ def copyDriveFile(users):
           try:
             result = callGAPI(drive.files(), 'copy',
                               bailOnInternalError=True,
-                              throwReasons=GAPI.DRIVE_COPY_THROW_REASONS+[GAPI.INTERNAL_ERROR, GAPI.TEAMDRIVES_SHORTCUT_FILE_NOT_SUPPORTED],
+                              throwReasons=GAPI.DRIVE_COPY_THROW_REASONS+[GAPI.INTERNAL_ERROR, GAPI.INSUFFICIENT_PARENT_PERMISSIONS,
+                                                                          GAPI.TEAMDRIVES_SHORTCUT_FILE_NOT_SUPPORTED],
                               fileId=childId, body=child, fields='id,name', supportsAllDrives=True)
             if not csvPF:
               entityModifierItemValueListActionPerformed(kvList, Act.MODIFIER_TO,
@@ -55139,7 +55148,8 @@ def copyDriveFile(users):
                                copyMoveOptions, False,
                                'copyFileInheritedPermissions',
                                'copyFileNonInheritedPermissions')
-          except (GAPI.fileNotFound, GAPI.forbidden, GAPI.internalError, GAPI.insufficientFilePermissions, GAPI.unknownError,
+          except (GAPI.fileNotFound, GAPI.forbidden, GAPI.internalError, GAPI.insufficientFilePermissions,
+                  GAPI.insufficientParentPermissions, GAPI.unknownError,
                   GAPI.invalid, GAPI.cannotCopyFile, GAPI.badRequest, GAPI.responsePreparationFailure, GAPI.fileNeverWritable, GAPI.fieldNotWritable,
                   GAPI.teamDrivesSharingRestrictionNotAllowed, GAPI.rateLimitExceeded, GAPI.userRateLimitExceeded,
                   GAPI.internalError, GAPI.teamDrivesShortcutFileNotSupported) as e:
@@ -55361,7 +55371,7 @@ def copyDriveFile(users):
           source.update(copyBody)
           result = callGAPI(drive.files(), 'copy',
                             bailOnInternalError=True,
-                            throwReasons=GAPI.DRIVE_COPY_THROW_REASONS+[GAPI.INTERNAL_ERROR],
+                            throwReasons=GAPI.DRIVE_COPY_THROW_REASONS+[GAPI.INTERNAL_ERROR, GAPI.INSUFFICIENT_PARENT_PERMISSIONS],
                             fileId=fileId,
                             ignoreDefaultVisibility=copyParameters[DFA_IGNORE_DEFAULT_VISIBILITY],
                             keepRevisionForever=copyParameters[DFA_KEEP_REVISION_FOREVER],
@@ -55395,7 +55405,8 @@ def copyDriveFile(users):
                              copyMoveOptions, False,
                              'copyFileInheritedPermissions',
                              'copyFileNonInheritedPermissions')
-      except (GAPI.fileNotFound, GAPI.forbidden, GAPI.internalError, GAPI.insufficientFilePermissions, GAPI.unknownError,
+      except (GAPI.fileNotFound, GAPI.forbidden, GAPI.internalError, GAPI.insufficientFilePermissions,
+              GAPI.insufficientParentPermissions, GAPI.unknownError,
               GAPI.invalid, GAPI.badRequest, GAPI.cannotCopyFile, GAPI.responsePreparationFailure, GAPI.fileNeverWritable, GAPI.fieldNotWritable,
               GAPI.teamDrivesSharingRestrictionNotAllowed, GAPI.rateLimitExceeded, GAPI.userRateLimitExceeded) as e:
         entityActionFailedWarning([Ent.USER, user, Ent.DRIVE_FILE_OR_FOLDER_ID, fileId], str(e), j, jcount)