sa email address cleanup

docs/Bulk-Processing.md

@@ -8,6 +8,7 @@
 - [CSV files](#csv-files)
 - [CSV files with redirection and select](#csv-files-with-redirection-and-select)
 - [Automatic batch processing](#automatic-batch-processing)
+- [Process Google Sheet commands and save results](#process-google-sheet-commands-and-save-results)
 
 ## Introduction
 Batch and CSV file processing can improve performance by executing Gam commands in parallel.
@@ -156,3 +157,18 @@ If you want to select a `gam.cfg` section for the command, you must select and s
 ```
 gam select <Section> save config auto_batch_min 1 redirect csv ./filelistperms.csv multiprocess group sales@domain.com print filelist fields id,name,mimetype,basicpermissions
 ```
+
+## Process Google Sheet commands and save results
+You want to process data from a Google Sheet tab and save the results to another tab in the same sheet.
+Make a Google sheet with two tabs: Commands, Results; get the File ID and the two tab IDs.
+Put your command data in the Commands tab.
+
+Run your command, write the results to Results.txt
+```
+gam redirect stdout ./Results.txt multiprocess redirect stderr stdout csv gsheet user@domain.com <FileID> id:<CommandsTabID> gam ... Command
+```
+
+Upload Results.txt to the Results tab of the sheet.
+```
+gam user user@domain.com update drivefile <FileID> localfile Results.txt retainname gsheet id:<ResultsTabID>
+```

docs/GamUpdates.md

@@ -10,6 +10,17 @@ Add the `-s` option to the end of the above commands to suppress creating the `g
 
 See [Downloads](https://github.com/taers232c/GAMADV-XTD3/wiki/Downloads) for Windows or other options, including manual installation
 
+### 6.77.03
+
+Thanks to jay, added the following Colab License SKUs:
+```
+1010500001 - Colab Pro
+1010500002 - Colab Pro+
+```
+
+Thanks to Jay, updated `gam print|show admins` to properly display addresses
+of service accounts with admin role assignments.
+
 ### 6.77.02
 
 Cleaned up problems with some of the new Chat API asadmin commands.

docs/How-to-Upgrade-from-Standard-GAM.md

@@ -335,7 +335,7 @@ writes the credentials into the file oauth2.txt.
 admin@server:/Users/admin/bin/gamadv-xtd3$ rm -f /Users/admin/GAMConfig/oauth2.txt
 admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam version
 WARNING: Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: /Users/admin/GAMConfig/oauth2.txt, Not Found
-GAMADV-XTD3 6.77.02 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.77.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.4 64-bit final
 MacOS Sonoma 14.5 x86_64
@@ -1009,7 +1009,7 @@ writes the credentials into the file oauth2.txt.
 C:\GAMADV-XTD3>del C:\GAMConfig\oauth2.txt
 C:\GAMADV-XTD3>gam version
 WARNING: Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: C:\GAMConfig\oauth2.txt, Not Found
-GAMADV-XTD3 6.77.02 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.77.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.4 64-bit final
 Windows-10-10.0.17134 AMD64

docs/Licenses.md

@@ -24,6 +24,7 @@
 | Cloud Identity Free | 101001 |
 | Cloud Identity Premium | 101005 |
 | Cloud Search | 101035 |
+| Colab | 101050 |
 | Education Endpoint Management | 101049 |
 | Gemini | 101047 |
 | Google Chrome Device Management | Google-Chrome-Device-Management |
@@ -49,6 +50,8 @@
 | Cloud Identity Free | 1010010001 | cloudidentity |
 | Cloud Identity Premium | 1010050001 | cloudidentitypremium |
 | Cloud Search | 1010350001 | cloudsearch |
+| Colab Pro | 1010500001 | colabpro |
+| Colab Pro+ | 1010500002 | colabpro+ | colabproplus |
 | Endpoint Education Upgrade | 1010490001 | eeu |
 | G Suite Basic | Google-Apps-For-Business | gsuitebasic |
 | G Suite Business | Google-Apps-Unlimited | gsuitebusiness |
@@ -123,6 +126,7 @@
         101043 |
         101047 |
         101049 |
+        101050 |
         Google-Apps |
         Google-Chrome-Device-Management |
         Google-Drive-storage |
@@ -151,6 +155,8 @@
         cloudidentity | identity | 1010010001 | Cloud Identity |
         cloudidentitypremium | identitypremium | 1010050001 | Cloud Identity Premium |
         cloudsearch | 1010350001 | Cloud Search |
+        colabpro | 1010500001 | Colab Pro |
+        colabpro+ | colabproplus | 1010500002 | Colab Pro+ |
         eeu | 1010490001 | SKU Endpoint Education Upgrade |
         geminibiz | 1010470003 | Gemini Business |
         geminiedu | 1010470004 | Gemini Education |

docs/Reseller.md

@@ -70,6 +70,8 @@ Thanks to Duncan Isaksen-Loxton for a script to help manage multiple domains.
         cloudidentity | identity | 1010010001 | Cloud Identity |
         cloudidentitypremium | identitypremium | 1010050001 | Cloud Identity Premium |
         cloudsearch | 1010350001 | Cloud Search |
+        colabpro | 1010500001 | Colab Pro |
+        colabpro+ | colabproplus | 1010500002 | Colab Pro+ |
         eeu | 1010490001 | SKU Endpoint Education Upgrade |
         geminibiz | 1010470003 | Gemini Business |
         geminiedu | 1010470004 | Gemini Education |

docs/Users-Chat.md

@@ -360,6 +360,8 @@ By default, Gam displays the information as an indented list of keys and values.
 
 ### Display information about all chat spaces, asadmin
 For query and orderby information, see: https://developers.google.com/workspace/chat/api/reference/rest/v1/spaces/search
+
+Only spaces of `<ChatSpaceType>` `space` are displayed; spaces of `<ChatSpaceType>` `groupchat` and `directmessage` are not displayed.
 ```
 gam <UserItem> show chatspaces asadmin
         [query <String>] [querytime<String> <Time>]

docs/Users-Drive-Files-Manage.md

@@ -519,6 +519,7 @@ You can update a specific sheet within a Google spreadsheet or add a new sheet t
 * `addsheet <String>` - Specify a sheet name to be added to the Google Sheets file
 * `charset <Charset>` - Specify the character set of the local file; if not specified, the value of `charset` from `gam.cfg` will be used
 * `columndelimiter <Character>` - Columns are separated by `<Character>`; if not specified, the value of `csv_input_column_delimiter` from `gam.cfg` will be used
+
 If you want the Google spreadsheet to retain its name, specify: `retainname localfile LocalFile.csv`.
 
 By default, the user, file name, updated file name and id values are displayed on stdout.

docs/Version-and-Help.md

@@ -3,7 +3,7 @@
 Print the current version of Gam with details
 ```
 gam version
-GAMADV-XTD3 6.77.02 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.77.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.4 64-bit final
 MacOS Sonoma 14.5 x86_64
@@ -15,7 +15,7 @@ Time: 2023-06-02T21:10:00-07:00
 Print the current version of Gam with details and time offset information
 ```
 gam version timeoffset
-GAMADV-XTD3 6.77.02 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.77.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.4 64-bit final
 MacOS Sonoma 14.5 x86_64
@@ -27,7 +27,7 @@ Your system time differs from www.googleapis.com by less than 1 second
 Print the current version of Gam with extended details and SSL information
 ```
 gam version extended
-GAMADV-XTD3 6.77.02 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
+GAMADV-XTD3 6.77.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.4 64-bit final
 MacOS Sonoma 14.5 x86_64
@@ -64,7 +64,7 @@ MacOS High Sierra 10.13.6 x86_64
 Path: /Users/Admin/bin/gamadv-xtd3
 Version Check:
   Current: 5.35.08
-   Latest: 6.77.02
+   Latest: 6.77.03
 echo $?
 1
 ```
@@ -72,7 +72,7 @@ echo $?
 Print the current version number without details
 ```
 gam version simple
-6.77.02
+6.77.03
 ```
 In Linux/MacOS you can do:
 ```
@@ -82,7 +82,7 @@ echo $VER
 Print the current version of Gam and address of this Wiki
 ```
 gam help
-GAM 6.77.02 - https://github.com/taers232c/GAMADV-XTD3
+GAM 6.77.03 - https://github.com/taers232c/GAMADV-XTD3
 Ross Scroggs <ross.scroggs@gmail.com>
 Python 3.12.4 64-bit final
 MacOS Sonoma 14.5 x86_64

src/GamCommands.txt

@@ -244,6 +244,7 @@ If an item contains spaces, it should be surrounded by ".
         101043 |
         101047 |
         101049 |
+        101050 |
         Google-Apps |
         Google-Chrome-Device-Management |
         Google-Drive-storage |
@@ -270,6 +271,8 @@ If an item contains spaces, it should be surrounded by ".
         cloudidentity | identity | 1010010001 | Cloud Identity |
         cloudidentitypremium | identitypremium | 1010050001 | Cloud Identity Premium |
         cloudsearch | 1010350001 | Cloud Search |
+        colabpro | 1010500001 | Colab Pro |
+        colabpro+ | colabproplus | 1010500002 | Colab Pro+ |
         eeu | 1010490001 | SKU Endpoint Education Upgrade |
         geminibiz | 1010470003 | Gemini Business |
         geminiedu | 1010470004 | Gemini Education |

src/GamUpdate.txt

@@ -2,6 +2,17 @@
 
 Merged GAM-Team version
 
+6.77.03
+
+Thanks to jay, added the following Colab License SKUs:
+```
+1010500001 - Colab Pro
+1010500002 - Colab Pro+
+```
+
+Thanks to Jay, updated `gam print|show admins` to properly display addresses
+of service accounts with admin role assignments.
+
 6.77.02
 
 Cleaned up problems with some of the new Chat API asadmin commands.

src/gam/__init__.py

@@ -5485,8 +5485,7 @@ def buildGAPIObject(api, credentials=None):
       API_Scopes = set(API.VAULT_SCOPES) if api == API.VAULT else set()
     GM.Globals[GM.CURRENT_CLIENT_API] = api
     GM.Globals[GM.CURRENT_CLIENT_API_SCOPES] = API_Scopes.intersection(GM.Globals[GM.CREDENTIALS_SCOPES])
-    scopeless_apis = {API.OAUTH2, API.CHROMEVERSIONHISTORY, API.SERVICEACCOUNTLOOKUP}
+    if api not in API.SCOPELESS_APIS and not GM.Globals[GM.CURRENT_CLIENT_API_SCOPES]:
-    if api not in scopeless_apis and not GM.Globals[GM.CURRENT_CLIENT_API_SCOPES]:
       systemErrorExit(NO_SCOPES_FOR_API_RC, Msg.NO_SCOPES_FOR_API.format(API.getAPIName(api)))
     if not GC.Values[GC.DOMAIN]:
       GC.Values[GC.DOMAIN] = GM.Globals[GM.DECODED_ID_TOKEN].get('hd', 'UNKNOWN').lower()
@@ -5604,7 +5603,6 @@ def getSitesObject(entityType=Ent.DOMAIN, entityName=None, i=0, count=0):
     sitesObject.debug = True
   return (userEmail, sitesObject)
 
-
 def getUserEmailFromID(uid, cd):
   try:
     result = callGAPI(cd.users(), 'get',
@@ -5624,6 +5622,26 @@ def getGroupEmailFromID(uid, cd):
   except (GAPI.groupNotFound, GAPI.domainNotFound, GAPI.domainCannotUseApis, GAPI.forbidden, GAPI.badRequest):
     return None
 
+def getServiceAccountEmailFromID(account_id, sal=None):
+  if sal is None:
+    sal = buildGAPIObject('serviceaccountlookup')
+  try:
+    certs = callGAPI(sal.serviceaccounts(), 'lookup',
+                     throwReasons = [GAPI.BAD_REQUEST, GAPI.RESOURCE_NOT_FOUND,  GAPI.INVALID_ARGUMENT],
+                     account=account_id)
+  except (GAPI.badRequest, GAPI.resourceNotFound, GAPI.invalidArgument):
+    return None
+  sa_cn_rx = r'CN=(.+)\.(.+).iam\.gservice.*'
+  sa_emails = []
+  for _, raw_cert in certs.items():
+    cert = x509.load_pem_x509_certificate(raw_cert.encode(), default_backend())
+    mg = re.match(sa_cn_rx, cert.issuer.rfc4514_string())
+    if mg:
+      sa_email = f'{mg.group(1)}@{mg.group(2)}.iam.gserviceaccount.com'
+      if sa_email not in sa_emails:
+        sa_emails.append(sa_email)
+  return GC.Values[GC.CSV_OUTPUT_FIELD_DELIMITER].join(sa_emails)
+
 # Convert UID to email address and type
 def convertUIDtoEmailAddressWithType(emailAddressOrUID, cd=None, sal=None, emailTypes=None,
                                      checkForCustomerId=False, ciGroupsAPI=False, aliasAllowed=True):
@@ -5677,35 +5695,11 @@ def convertUIDtoEmailAddressWithType(emailAddressOrUID, cd=None, sal=None, email
     except (GAPI.badRequest, GAPI.resourceNotFound, GAPI.forbidden):
       pass
   if 'serviceaccount' in emailTypes:
-    if sal is None:
-      sal = buildGAPIObject(API.SERVICEACCOUNTLOOKUP)
     uid = getServiceAccountEmailFromID(normalizedEmailAddressOrUID, sal)
     if uid:
       return (uid, 'serviceaccount')
   return (normalizedEmailAddressOrUID, 'unknown')
 
-def getServiceAccountEmailFromID(account_id, sal=None):
-  if sal is None:
-    sal = buildGAPIObject('serviceaccountlookup')
-  throwReasons = [GAPI.BAD_REQUEST,
-                  GAPI.RESOURCE_NOT_FOUND,
-                  GAPI.INVALID_ARGUMENT]
-  try:
-    certs = callGAPI(sal.serviceaccounts(),
-                     'lookup',
-                     account=account_id,
-                     throwReasons=throwReasons)
-  except (GAPI.badRequest, GAPI.resourceNotFound, GAPI.invalidArgument):
-    return
-  sa_cn_rx = r'CN=.*\.gserviceaccount\.com$'
-  sa_emails = []
-  for kid, raw_cert in certs.items():
-    cert = x509.load_pem_x509_certificate(raw_cert.encode(), default_backend())
-    subject = cert.issuer.rfc4514_string()
-    if re.match(sa_cn_rx, subject):
-      sa_emails.append(subject[3:])
-  return ' or '.join(sa_emails)
-
 # Convert UID to email address
 def convertUIDtoEmailAddress(emailAddressOrUID, cd=None, emailTypes=None,
                              checkForCustomerId=False, ciGroupsAPI=False, aliasAllowed=True):
@@ -16287,10 +16281,8 @@ def doInfoAdminRole():
   fields = ','.join(set(fieldsList))
   try:
     role = callGAPI(cd.roles(), 'get',
-                    throwReasons=[GAPI.BAD_REQUEST,
+                    throwReasons=[GAPI.NOT_FOUND, GAPI.FORBIDDEN, GAPI.FAILED_PRECONDITION,
-                                  GAPI.CUSTOMER_NOT_FOUND,
+                                  GAPI.BAD_REQUEST, GAPI.CUSTOMER_NOT_FOUND],
-                                  GAPI.FORBIDDEN]+[GAPI.NOT_FOUND,
-                                                   GAPI.FAILED_PRECONDITION],
                     customer=GC.Values[GC.CUSTOMER_ID], roleId=roleId, fields=fields)
     role.setdefault('isSuperAdminRole', False)
     role.setdefault('isSystemRole', False)
@@ -16434,8 +16426,15 @@ def doDeleteAdmin():
   except (GAPI.badRequest, GAPI.customerNotFound):
     accessErrorExit(cd)
 
+ASSIGNEE_EMAILTYPE_TOFIELD_MAP = {
+  'user': 'assignedToUser',
+  'group': 'assignedToGroup',
+  'serviceaccount': 'assignedToServiceAccount',
+  }
 PRINT_ADMIN_FIELDS = ['roleAssignmentId', 'roleId', 'assignedTo', 'scopeType', 'orgUnitId']
-PRINT_ADMIN_TITLES = ['roleAssignmentId', 'roleId', 'role', 'assignedTo', 'assignedToUser', 'assignedToGroup', 'scopeType', 'orgUnitId', 'orgUnit']
+PRINT_ADMIN_TITLES = ['roleAssignmentId', 'roleId', 'role',
+                      'assignedTo', 'assignedToUser', 'assignedToGroup', 'assignedToServiceAccount',
+                      'scopeType', 'orgUnitId', 'orgUnit']
 
 # gam print admins [todrive <ToDriveAttribute>*]
 #	[user|group <EmailAddress>|<UniqueID>] [role <RoleItem>] [condition]
@@ -16449,10 +16448,8 @@ def doPrintShowAdmins():
       if roleId not in rolePrivileges:
         try:
           rolePrivileges[roleId] = callGAPI(cd.roles(), 'get',
-                                            throwReasons=[GAPI.BAD_REQUEST,
+                                            throwReasons=[GAPI.NOT_FOUND, GAPI.FORBIDDEN, GAPI.FAILED_PRECONDITION,
-                                                          GAPI.CUSTOMER_NOT_FOUND,
+                                                          GAPI.BAD_REQUEST, GAPI.CUSTOMER_NOT_FOUND],
-                                                          GAPI.FORBIDDEN]+[GAPI.NOT_FOUND,
-                                                                           GAPI.FAILED_PRECONDITION],
                                             customer=GC.Values[GC.CUSTOMER_ID],
                                             roleId=roleId,
                                             fields='rolePrivileges')
@@ -16468,26 +16465,13 @@ def doPrintShowAdmins():
     assignedTo = admin['assignedTo']
     if assignedTo not in assignedToIdEmailMap:
       assigneeType = admin.get('assigneeType')
-      if assigneeType == 'user':
+      assignedToField = ASSIGNEE_EMAILTYPE_TOFIELD_MAP.get(assigneeType, None)
-        assignedToField = 'assignedToUser'
-      elif assigneeType == 'group':
-        assignedToField = 'assignedToGroup'
-      elif assigneeType == 'serviceaccount':
-        assignedToField = 'assignedToServiceAccount'
-      else:
-        assignedToField = None
-      emailTypes = ['user', 'group', 'serviceaccount']
       assigneeEmail, assigneeType = convertUIDtoEmailAddressWithType(f'uid:{assignedTo}',
                                                                      cd,
                                                                      sal,
-                                                                     emailTypes=emailTypes)
+                                                                     emailTypes=list(ASSIGNEE_EMAILTYPE_TOFIELD_MAP.keys()))
-      if not assignedToField and assigneeType in ['user', 'group', 'serviceaccount']:
+      if not assignedToField and assigneeType in ASSIGNEE_EMAILTYPE_TOFIELD_MAP:
-        if assigneeType == 'user':
+        assignedToField = ASSIGNEE_EMAILTYPE_TOFIELD_MAP[assigneeType]
-          assignedToField = 'assignedToUser'
-        elif assigneeType == 'group':
-          assignedToField = 'assignedToGroup'
-        elif assigneeType == 'serviceaccount':
-          assignedToField = 'assignedToServiceAccount'
       assignedToIdEmailMap[assignedTo] = {'assignedToField': assignedToField, 'assigneeEmail': assigneeEmail}
     assignedToField = assignedToIdEmailMap[assignedTo]['assignedToField']
     if assignedToField:

src/gam/gamlib/glapi.py

@@ -121,6 +121,12 @@ JWT_APIS = {
   ORGPOLICY: [CLOUD_PLATFORM_SCOPE],
   }
 #
+SCOPELESS_APIS = {
+  CHROMEVERSIONHISTORY,
+  OAUTH2,
+  SERVICEACCOUNTLOOKUP,
+  }
+#
 APIS_NEEDING_ACCESS_TOKEN = {
   CBCM: ['https://www.googleapis.com/auth/admin.directory.device.chromebrowsers']
   }

src/gam/gamlib/glcfg.py

@@ -292,7 +292,7 @@ TODRIVE_USER = 'todrive_user'
 TRUNCATE_CLIENT_ID  = 'truncate_client_id'
 # Update CrOS org unit with orgUnitId
 UPDATE_CROS_OU_WITH_ID = 'update_cros_ou_with_id'
-# Use chat asadmin where possible
+# Use admin access for chat where possible
 USE_CHAT_ADMIN_ACCESS = 'use_chat_admin_access'
 # Use course owner for course access
 USE_COURSE_OWNER_ACCESS = 'use_course_owner_access'