Improve gam print group-members

docs/GamUpdates.md

@@ -11,6 +11,24 @@ Add the `-s` option to the end of the above commands to suppress creating the `g
 
 See [Downloads](https://github.com/taers232c/GAMADV-XTD3/wiki/Downloads) for Windows or other options, including manual installation
 
+### 6.67.16
+
+By default, `gam print group-members membernames` displays `Unknown` for members whose names can not be determined.
+Added option `unknownname <String>` that let's you specify an alternative value.
+
+Further improved performance of `gam print group-members membernames cachememberinfo`.
+
+### 6.67.15
+
+Update `gam print group-members membernames` to handle the following error:
+```
+ERROR: 400: failedPrecondition - Precondition check failed.
+```
+
+Added option `cachememberinfo [Boolean]` to `gam print group-members` that causes GAM to cache member info
+so that only one API call is made to get information for each user/group. This consumes
+more memory but dramatically reduces the number of API calls.
+
 ### 6.67.14
 
 Updated reseller commands to handle the following error:

docs/Groups-Membership.md

@@ -583,6 +583,7 @@ gam print group-members [todrive <ToDriveAttribute>*]
         [userfields <UserFieldNameList>]
         [(recursive [noduplicates])|includederivedmembership] [nogroupemail]
         [peoplelookup|(peoplelookupuser <EmailAddress>)]
+        [unknownname <String>] [cachememberinfo [Boolean]]
         [formatjson [quotechar <Character>]]
 ```
 By default, the group membership of all groups in the account are displayed, these options allow selection of subsets of groups:
@@ -646,11 +647,17 @@ these options specify which fields to display:
     * `delivery|deliverysettings` - Specify this field to get delivery information; an additional API call per member is required
 * `userfields <UserFieldNameList>` - For members that are users, display these user fields; an additional API call per member is required
 
+The additional API calls can be reduced with the `cachememberinfo` option; a single API call is made for each user/group
+and the data is cached to eliminate to need to repeat the API call; this consumes more memory but dramatically reduces the number of API calls.
+
 If member names are requested, names are not available for users not in the domain; you can request that GAM use the People API to retrieve
 names for these users. Names are not retrieved in all cases and success is dependent on what user is used to perform the retrievals.
 * `peoplelookup` - Use the administrator named in oauth2.txt to perform the retrievals
 * `peoplelookupuser <EmailAddress>` - Use `<EmailAddress>` to perform the retrievals
 
+By default, when `membernames` is specified, GAM displays `Unknown` for members whose names can not be determined.
+Use `unknownname <String>` to specify an alternative value.
+
 By default, the group email address is always shown, you can suppress it with the `nogroupemail` option.
 
 By default, members that are groups are displayed as a single entry of type GROUP; this option recursively expands group members to display their user members.

docs/How-to-Upgrade-from-Standard-GAM.md

@@ -334,7 +334,7 @@ writes the credentials into the file oauth2.txt.
 admin@server:/Users/admin/bin/gamadv-xtd3$ rm -f /Users/admin/GAMConfig/oauth2.txt
 admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam version
 WARNING: Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: /Users/admin/GAMConfig/oauth2.txt, Not Found
-GAMADV-XTD3 6.67.14 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.67.16 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.1 64-bit final
 MacOS Sonoma 14.2.1 x86_64
@@ -1002,7 +1002,7 @@ writes the credentials into the file oauth2.txt.
 C:\GAMADV-XTD3>del C:\GAMConfig\oauth2.txt
 C:\GAMADV-XTD3>gam version
 WARNING: Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: C:\GAMConfig\oauth2.txt, Not Found
-GAMADV-XTD3 6.67.14 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.67.16 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.1 64-bit final
 Windows-10-10.0.17134 AMD64

docs/Version-and-Help.md

@@ -4,7 +4,7 @@
 Print the current version of Gam with details
 ```
 gam version
-GAMADV-XTD3 6.67.14 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.67.16 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.1 64-bit final
 MacOS Sonoma 14.2.1 x86_64
@@ -16,7 +16,7 @@ Time: 2023-06-02T21:10:00-07:00
 Print the current version of Gam with details and time offset information
 ```
 gam version timeoffset
-GAMADV-XTD3 6.67.14 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.67.16 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.1 64-bit final
 MacOS Sonoma 14.2.1 x86_64
@@ -28,7 +28,7 @@ Your system time differs from www.googleapis.com by less than 1 second
 Print the current version of Gam with extended details and SSL information
 ```
 gam version extended
-GAMADV-XTD3 6.67.14 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.67.16 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.1 64-bit final
 MacOS Sonoma 14.2.1 x86_64
@@ -65,7 +65,7 @@ MacOS High Sierra 10.13.6 x86_64
 Path: /Users/Admin/bin/gamadv-xtd3
 Version Check:
   Current: 5.35.08
-   Latest: 6.67.14
+   Latest: 6.67.16
 echo $?
 1
 ```
@@ -73,7 +73,7 @@ echo $?
 Print the current version number without details
 ```
 gam version simple
-6.67.14
+6.67.16
 ```
 In Linux/MacOS you can do:
 ```
@@ -83,7 +83,7 @@ echo $VER
 Print the current version of Gam and address of this Wiki
 ```
 gam help
-GAM 6.67.14 - https://github.com/taers232c/GAMADV-XTD3
+GAM 6.67.16 - https://github.com/taers232c/GAMADV-XTD3
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.1 64-bit final
 MacOS Sonoma 14.2.1 x86_64

src/GamCommands.txt

@@ -3652,6 +3652,7 @@ gam print group-members [todrive <ToDriveAttribute>*]
         [userfields <UserFieldNameList>]
         [(recursive [noduplicates])|includederivedmembership] [nogroupemail]
         [peoplelookup|(peoplelookupuser <EmailAddress>)]
+        [unknownname <String>] [cachememberinfo [Boolean]]
         [formatjson [quotechar <Character>]]
 gam show group-members
         [([domain|domains <DomainNameEntity>] ([member|showownedby <EmailItem>]|[(query <QueryGroup>)|(queries <QueryGroupList>)]))|

src/GamUpdate.txt

@@ -2,6 +2,24 @@
 
 Merged GAM-Team version
 
+6.67.16
+
+By default, `gam print group-members membernames` displays `Unknown` for members whose names can not be determined.
+Added option `unknownname <String>` that let's you specify an alternative value.
+
+Further improved performance of `gam print group-members membernames cachememberinfo`.
+
+6.67.15
+
+Update `gam print group-members membernames` to handle the following error:
+```
+ERROR: 400: failedPrecondition - Precondition check failed.
+```
+
+Added option `cachememberinfo [Boolean]` to `gam print group-members` that causes GAM to cache member info
+so that only one API call is made to get information for each user/group. This consumes
+more memory but dramatically reduces the number of API calls.
+
 6.67.14
 
 Updated reseller commands to handle the following error:

src/gam/__init__.py

@@ -32034,6 +32034,7 @@ GROUPMEMBERS_DEFAULT_FIELDS = ['group', 'type', 'role', 'id', 'status', 'email']
 #	[userfields <UserFieldNameList>]
 #	[(recursive [noduplicates])|includederivedmembership] [nogroupemail]
 #	[peoplelookup|(peoplelookupuser <EmailAddress>)]
+#	[unknownname <String>] [cachememberinfo [Boolean]]
 #	[formatjson [quotechar <Character>]]
 def doPrintGroupMembers():
   def getNameFromPeople(memberId):
@@ -32049,12 +32050,11 @@ def doPrintGroupMembers():
               return name['displayName']
     except (GAPI.notFound, GAPI.serviceNotAvailable, GAPI.forbidden):
       pass
-    return ''
+    return unknownName
 
   cd = buildGAPIObject(API.DIRECTORY)
   ci = None
   people = None
-  peopleNames = {}
   memberOptions = initMemberOptions()
   groupColumn = True
   customerKey = GC.Values[GC.CUSTOMER_ID]
@@ -32071,6 +32071,10 @@ def doPrintGroupMembers():
   typesSet = set()
   matchPatterns = {}
   showDeliverySettings = False
+  cacheMemberInfo = False
+  memberInfo = {}
+  memberNames = {}
+  unknownName = UNKNOWN
   while Cmd.ArgumentsRemaining():
     myarg = getArgument()
     if myarg == 'todrive':
@@ -32127,6 +32131,10 @@ def doPrintGroupMembers():
       _, people = buildGAPIServiceObject(API.PEOPLE, getEmailAddress())
       if not people:
         return
+    elif myarg == 'unknownname':
+      unknownName = getString(Cmd.OB_STRING)
+    elif myarg == 'cachememberinfo':
+      cacheMemberInfo = getBoolean()
     else:
       FJQC.GetFormatJSONQuoteChar(myarg, False)
   if not typesSet:
@@ -32201,17 +32209,24 @@ def doPrintGroupMembers():
       memberType = member.get('type')
       if userFieldsList:
         if memberOptions[MEMBEROPTION_MEMBERNAMES]:
-          row['name'] = UNKNOWN
+          row['name'] = unknownName
         if memberType == Ent.TYPE_USER:
-          try:
+            if not cacheMemberInfo or memberId not in memberNames:
               mbinfo = callGAPI(cd.users(), 'get',
-                              throwReasons=GAPI.USER_GET_THROW_REASONS+[GAPI.SERVICE_NOT_AVAILABLE],
+                                throwReasons=GAPI.USER_GET_THROW_REASONS+[GAPI.SERVICE_NOT_AVAILABLE, GAPI.FAILED_PRECONDITION],
                                 retryReasons=GAPI.SERVICE_NOT_AVAILABLE_RETRY_REASONS,
                                 userKey=memberId, fields=userFields)
               if memberOptions[MEMBEROPTION_MEMBERNAMES]:
                 row['name'] = mbinfo['name'].pop('fullName')
                 if not mbinfo['name']:
                   mbinfo.pop('name')
+              if cacheMemberInfo:
+                memberNames[memberId] = row['name']
+                if mbinfo:
+                  memberInfo[memberId] = mbinfo
+            else:
+              row['name'] = memberNames[memberId]
+              mbinfo = memberInfo.get(memberId, {})
             if not FJQC.formatJSON:
               csvPF.WriteRowTitles(flattenJSON(mbinfo, flattened=row))
             else:
@@ -32224,32 +32239,47 @@ def doPrintGroupMembers():
               csvPF.WriteRowNoFilter(fjrow)
             continue
           except GAPI.userNotFound:
-            if memberOptions[MEMBEROPTION_MEMBERNAMES] and people:
+            if memberOptions[MEMBEROPTION_MEMBERNAMES]:
-              if memberId not in peopleNames:
+              if people:
-                peopleNames[memberId] = getNameFromPeople(memberId)
+                if memberId not in memberNames:
-              if peopleNames[memberId]:
+                  memberNames[memberId] = getNameFromPeople(memberId)
-                row['name'] = peopleNames[memberId]
+              else:
+                memberNames[memberId] = unknownName
+              row['name'] = memberNames[memberId]
           except (GAPI.domainNotFound, GAPI.domainCannotUseApis, GAPI.forbidden,
-                  GAPI.badRequest, GAPI.backendError, GAPI.systemError, GAPI.serviceNotAvailable):
+                  GAPI.badRequest, GAPI.backendError, GAPI.systemError, GAPI.serviceNotAvailable, GAPI.failedPrecondition):
-            pass
+            if memberOptions[MEMBEROPTION_MEMBERNAMES] and cacheMemberInfo:
+              memberNames[memberId] = unknownName
         elif memberType == Ent.TYPE_GROUP:
           if memberOptions[MEMBEROPTION_MEMBERNAMES]:
             try:
+              if not cacheMemberInfo or memberId not in memberNames:
                 row['name'] = callGAPI(cd.groups(), 'get',
-                                     throwReasons=GAPI.GROUP_GET_THROW_REASONS+[GAPI.SERVICE_NOT_AVAILABLE],
+                                       throwReasons=GAPI.GROUP_GET_THROW_REASONS+[GAPI.SERVICE_NOT_AVAILABLE, GAPI.FAILED_PRECONDITION],
                                        retryReasons=GAPI.GROUP_GET_RETRY_REASONS,
                                        groupKey=memberId, fields='name')['name']
+                if cacheMemberInfo:
+                  memberNames[memberId] = row['name']
+              else:
+                row['name'] = memberNames[memberId]
             except (GAPI.groupNotFound, GAPI.domainNotFound, GAPI.domainCannotUseApis, GAPI.forbidden, GAPI.badRequest,
-                    GAPI.invalid, GAPI.systemError, GAPI.serviceNotAvailable):
+                    GAPI.invalid, GAPI.systemError, GAPI.serviceNotAvailable, GAPI.failedPrecondition):
-              pass
+              if memberOptions[MEMBEROPTION_MEMBERNAMES] and cacheMemberInfo:
+                memberNames[memberId] = unknownName
         elif memberType == Ent.TYPE_CUSTOMER:
           if memberOptions[MEMBEROPTION_MEMBERNAMES]:
             try:
+              if not cacheMemberInfo or memberId not in memberNames:
                 row['name'] = callGAPI(cd.customers(), 'get',
                                        throwReasons=[GAPI.BAD_REQUEST, GAPI.INVALID_INPUT, GAPI.RESOURCE_NOT_FOUND, GAPI.FORBIDDEN],
                                        customerKey=memberId, fields='customerDomain')['customerDomain']
+                if cacheMemberInfo:
+                  memberNames[memberId] = row['name']
+              else:
+                row['name'] = memberNames[memberId]
             except (GAPI.badRequest, GAPI.invalidInput, GAPI.resourceNotFound, GAPI.forbidden):
-              pass
+              if memberOptions[MEMBEROPTION_MEMBERNAMES] and cacheMemberInfo:
+                memberNames[memberId] = unknownName
       if not FJQC.formatJSON:
         csvPF.WriteRow(row)
       else:
@@ -33239,6 +33269,7 @@ def doInfoCIGroups():
   rolesSet = set()
   typesSet = set()
   memberOptions = initMemberOptions()
+  cachedGroupMembers = {}
   while Cmd.ArgumentsRemaining():
     myarg = getArgument()
     if myarg == 'quick':
@@ -33339,7 +33370,6 @@ def doInfoCIGroups():
         printKeyValueList([Msg.TOTAL_ITEMS_IN_ENTITY.format(Ent.Plural(entityType), Ent.Singular(Ent.CLOUD_IDENTITY_GROUP)), len(members)])
         Ind.Decrement()
       elif showMemberTree:
-        cachedGroupMembers = {}
         Ind.Increment()
         printEntity([Ent.MEMBERSHIP_TREE, ''])
         Ind.Increment()