# CSV Output Filtering - [Python Regular Expressions](Python-Regular-Expressions) Search function - [Definitions](#definitions) - [Quoting rules](#quoting-rules) - [Column header filtering](#column-header-filtering) - [Column row filtering](#column-row-filtering) - [Field names](#field-names) - [Inclusive filters](#inclusive-filters) - [Exclusive filters](#exclusive-filters) - [Matches](#matches) - [Column row limiting](#column-row-limiting) - [Saving filters in gam.cfg](#saving-filters-in-gamcfg) There are seven values in `gam.cfg` that can be used to filter the output from `gam print` commands. * `csv_output_header_filter` - A list of `` used to select specific column headers to include * `csv_output_header_drop_filter` - A list of `` used to select specific column headers to exclude * `csv_output_header_force` - A list of used to specify the exact column headers to include * `csv_output_header_order` - A list of used to specify the column header order; any headers in the file but not in the list will appear after the headers in the list. * `csv_output_row_filter` - A list or JSON dictionary used to include specific rows based on column values * `csv_output_row_drop_filter` - A list or JSON dictionary used to exclude specific rows based on column values * `csv_output_row_limit` - A limit on the number of rows written The original implementation required that row filters be expressed in JSON notation; these are almost impossible to enter correctly in Windows; on Mac OS or Linux, it's easy. You can now enter the row filters as lists on all platforms. ## Definitions [Data Selectors](Collections-of-items) ``` ::= | | ``` ``` ::= -- | (+|-)(d|w|y) | never| today ::= --T::[.](Z|(+|-(:))) | (+|-)(m|h|d|w|y) | never| now|today ::= <|<=|>=|>|=|!= ::= See: https://docs.python.org/3/library/re.html> ::= ::= ::= > :: = ::= "(,)*" ::= [(any|all):]boolean:| [(any|all):]count| [(any|all):]countrange!=/| [(any|all):]countrange=/| [(any|all):]data:| [(any|all):]date| [(any|all):]daterange!=/| [(any|all):]daterange=/| [(any|all):]length| [(any|all):]lengthrange!=/| [(any|all):]lengthrange=/| [(any|all):]notdata: [(any|all):]notregex:| [(any|all):]notregexcs:| [(any|all):]regex:| [(any|all):]regexcs:| [(any|all):]text| [(any|all):]textrange!=/| [(any|all):]textrange=/| [(any|all):]time| [(any|all):]timeofdayrange!=:/:| [(any|all):]timeofdayrange=:/:| [(any|all):]timerange!=/| [(any|all):]timerange=/| ::= "':'(,':')*" ::= '{"": ""(,"": "")*}' | "{\"\": \"\"(,\"\": \"\")*}" ``` ## Quoting rules Name:value form. ``` ::= "':'(,':')*" ``` * ``, even if it has one element, should be enclosed in `"`. * Each `:` pair should be enclosed in `'`. * If `` contains a `:` or a space, it should be enclosed in `\"`. * If `` or `` in `` contain a space, it should be enclosed in `\"`. * If `` or `` in `` contain a `\` to escape a special character or enter a special sequence, enter `\\\` on Linux and Mac OS, `\\` on Windows, Examples: ``` csv_output_row_filter "'\"accounts:used_quota_in_mb\":count>15000'" csv_output_row_filter "'email:data:\"csvfile gsheet:email user@domain.com FileID Sheet1\"'" Linux and Mac OS csv_output_row_filter "'phones.\\\d+.value:regex:(?:^\\$510\\$ )|(?:^510[- ])\\\d{3}-\\\d{4}'" Windows csv_output_row_filter "'phones.\\d+.value:regex:(?:^\$510\$ )|(?:^510[- ])\\d{3}-\\d{4}'" ``` JSON form. ``` ::= '{"": ""(,"": "")*}' | "{\"\": \"\"(,\"\": \"\")*}" ``` * The first form can be used on Linux and Mac OS; it can not be used on Windows. * The second form can be used on Linux, Mac OS and Windows. * If `` contains a `:`, no additional quoting is required Example: ``` csv_output_row_filter '{"accounts:used_quota_in_mb": "count>=150"}' csv_output_row_filter "{\"accounts:used_quota_in_mb\": \"count>=150\"}" ``` ## Column header filtering Gam gives you the ability to select fields(column headers) in its print commands, but there may be cases where you get more columns than is desirable. * `csv_output_header_filter` - Used to select the column headers to include in the output * `csv_output_header_drop_filter` - Used to select the column headers to exclude from the output Typically, you would use the option that involves typing the fewest column names but both options can be used. When both options are used, `csv_output_header_drop_filter` is processed first, then `csv_output_header_filter`. Field names are specified by regular expressions; at its simplest, you specify a complete field name. Field names are matched in a case insensitive manner. ``` gam config csv_output_header_filter ... gam config csv_output_header_drop_filter ... ``` ### Example you want a list of user email addresses and full names; you do not need the given or family names. No filtering. ``` gam print users name primaryEmail,name.givenName,name.familyName,name.fullName testuser1@domain.com,Test,User1,Test User1 testuser2@domain.com,Test,User2,Test User2 ... ``` With inclusion filtering. ``` gam config csv_output_header_filter "primaryEmail,name.fullName" print users name primaryEmail,name.fullName testuser1@domain.com,Test User1 testuser2@domain.com,Test User2 ... ``` With exclusion filtering. ``` gam config csv_output_header_drop_filter "name.givenName,name.familyName" print users name primaryEmail,name.fullName testuser1@domain.com,Test User1 testuser2@domain.com,Test User2 ... ``` ## Column row filtering Row filtering includes/excludes rows based on column values. ### Field names Field names are specified by regular expressions; at its simplest, you specify a complete field name. Field names are matched in a case insensitive manner. If the field name doesn't contain any of the following regular expression characters `^$*+|$[{(`, it will be surrounded with `^$` so that it doesn't match any subfields that begin with the field name as a prefix. The following filter will match the count field and not the subfields. ``` config csv_output_row_filter "'externalIds:countrange=1/10'" primaryEmail,externalIds,externalIds.0.type,externalIds.0.value,externalIds.1.type,externalIds.1.value,... ``` ### Inclusive filters You can include rows generated by gam print commands based on column values. You specify a list of fields (headers) and the values they must have. `csv_output_row_filter` is used to specify the fields and values. Each field name/expression can appear only once in the list. ``` gam config csv_output_row_filter ... gam config csv_output_row_filter ... ``` You optionally specify whether all or any value filters must match for the row to be included in the output. * `csv_output_row_filter_mode allmatch` - All value filters must match for the row to be included in the output; this is the default * `csv_output_row_filter_mode anymatch` - Any value filter must match for the row to be included in the output ``` gam config csv_output_row_filter_mode anymatch csv_output_row_filter ... gam config csv_output_row_filter_mode anymatch csv_output_row_filter ... ``` ### Exclusive filters You can exclude rows generated by gam print commands based on column values. You specify a list of fields (headers) and the values they must not have. `csv_output_row_drop_filter` is used to specify the fields and values. Each field name/expression can appear only once in the list. ``` gam config csv_output_row_drop_filter ... gam config csv_output_row_drop_filter ... ``` You optionally specify whether all or any value filters must match for the row to be excluded from the output. * `csv_output_row_drop_filter_mode allmatch` - If all value filters match, the row is excluded from the output * `csv_output_row_drop_filter_mode anymatch` - If any value filter matches, the row is excluded from the output; this is the default ``` gam config csv_output_row_drop_filter_mode allmatch csv_output_row_drop_filter ... gam config csv_output_row_drop_filter_mode allmatch csv_output_row_drop_filter ... ``` ### Matches A filter matches if the field has the desired value. lf you specify a regular expression for a field name that matches several columns, the filter matches if any of the columns has a match. In the case of `notregex|notregexcs|notdata`, the filter matches if none (not any) of the columns has a match. `` allows specifying that the filter will match only if all of the columns have a match. In the case of `notregex|notregexcs|notdata`, the filter matches if some (not all) of the columns have a match. If neither `any` or `all` is explicitly specified, `any` is the default. These are the row value filter types: * `boolean:` - Used on fields with Boolean values; a blank field is considered False * `count` - Used on fields with numbers; a blank field will not match * `countrange=/` - Used on fields with numbers; a blank field will not match * The field value must be `>=` the left `` and `<=` the right `` * `countrange!=/` - Used on fields with numbers; a blank field will not match * The field value must be `<` the left `` or `>` the right `` * `data:` - Used on fields with text; field value must match some value in ``; case sensitive * `date` - Used on fields with dates or times; only the date portion of a time field is compared; a blank field will not match * `daterange=/` - Used on fields with dates or times; only the date portion of a time field is compared; a blank field will not match * The field value must be `>=` the left `` and `<=` the right `` * `daterange!=/` - Used on fields with dates or times; only the date portion of a time field is compared; a blank field will not match * The field value must be `<` the left `` or `>` the right `` * `length` - Used on fields with strings; non string fields will not match * `lengthrange=/` - Used on fields with strings; non string fields will not match * The field length must be `>=` the left `` and `<=` the right `` * `lengthrange!=/` - Used on fields with strings; non string fields will not match * The field length must be `<` the left `` or `>` the right `` * `notdata:` - Used on fields with text; field value must not match any value in ``; case sensitive * `notregex:` - Used on fields with text; field value must not match ``; case insensitive * `notregexcs:` - Used on fields with text; field value must not match ``; case sensitive * `regex:` - Used on fields with text; field value must match ``; case insensitive * `regexcs:` - Used on fields with text; field value must match ``; case sensitive * `text` - Used on fields with text * `textrange=/` - Used on fields with strings * The field value must be `>=` the left `` and `<=` the right `` * `textrange!=/` - Used on fields with strings * The field value must be `<` the left `` or `>` the right `` * `time` - Used on fields with times; a blank field will not match * `timeofdayrange=:/:` - Used on fields with times; a blank field will not match * The field value must be `>=` the left `:` and `<=` the right `:` * `timeofdayrange!=:/:` - Used on fields with times; a blank field will not match * The field value must be `<` the left `:` or `>` the right `:` * `timerange=/` - Used on fields with times; a blank field will not match * The field value must be `>=` the left `` and `<=` the right `` * `timerange!=/` - Used on fields with times; a blank field will not match * The field value must be `<` the left `` or `>` the right `` ### Examples You want a list of groups with 100 or more direct members. ``` gam config csv_output_row_filter "'directMembersCount:count>100'" print groups fields directmemberscount ``` You want a list of users created in the last 30 days. ``` gam config csv_output_row_filter "'creationTime:date>=-30d'" print users fields creationtime ``` You want a list of users in the OU /Test that are consuming more than 15GB of storage. Special quoting is required because the field name contains a colon. ``` gam config csv_output_row_filter "'\"accounts:used_quota_in_mb\":count>15000'" report users select ou /Test fields accounts:used_quota_in_mb ``` You want the names of users directly in the OU /Test, you do not want users in any sub-OUs of /Test. * The Google API will only supply users in an OU and sub-OUs, GAM has to filter out the users in the sub-OU. ``` gam config csv_output_row_filter "'orgUnitPath:regex:^/Test$'" print users query "orgUnitPath=/Test" fields name,ou ``` You want the names of female users directly in the OU /Test, you do not want users in any sub-OUs of /Test. * The Google API will only supply users in an OU and sub-OUs, GAM has to filter out the users in the sub-OU. ``` gam config csv_output_row_filter "'orgUnitPath:regex:^/Test$','gender:regex:female'" print users query "orgUnitPath=/Test" fields name,ou,gender ``` You want a list of groups not created by an administrator. ``` gam config csv_output_row_filter "'adminCreated:boolean:false'" print groups fields admincreated ``` You want a list of users with phone numbers in the area code 510; the number can be in the format `(510) ddd-dddd` or `510-ddd-dddd` or `510 ddd-dddd`. ``` gam config csv_output_header_filter "primaryEmail,name.fullName,phones.*value" csv_output_row_filter "'"'phones.\\\d+.value:regex:(?:^\\$510\\$ )|(?:^510[- ])\\\d{3}-\\\d{4}'"'" print users name phones primaryEmail,name.fullName,phones.0.value testuser1@domain.com,Test User1,(510) 555-1212 testuser2@domain.com,Test User2,510-555-1212 testuser3@domain.com,Test User3,510 555-1212 ``` You want a list of users not in the organization cost center "Tech Support". ``` gam config csv_output_header_filter "primaryEmail,name.fullName,orgUnitPath,organizations.*costCenter" csv_output_row_filter 'organizations.*costCenter:notregex:"Tech Support"' print users fields name,ou,organizations gam config csv_output_header_filter "primaryEmail,name.fullName,orgUnitPath,organizations.*costCenter" csv_output_row_drop_filter 'organizations.*costCenter:regex:"Tech Support"' print users fields name,ou,organizations primaryEmail,name.fullName,orgUnitPath,organizations.0.costCenter testuser1@domain.com,Test User1,/Test,Sales testuser2@domain.com,Test User2,/Test,Development ``` You want a list of recurring events with at least one external guest. ``` gam config csv_output_row_filter "'^attendees$:count>1','recurrence:count>=1','attendees.*email:all:notregex:(^$)|(.+@domain.com)'" csv_output_row_drop_filter "'attendees.*email:regex:.+@resource.calendar.google.com'" redirect csv ./externalrecurringEvents.csv calendar print events ``` ## Column row limiting You can limit the number of rows written to a CSV file. When single processing, the limit is on the total number of rows written to the file. When multiprocessing, the limit is on the number of rows written to the file by each subprocess. ### Examples Display the 10 files with the largest quotaBytesUsed values for a single user. ``` gam config csv_output_row_limit 10 redirect csv ./BigQuotaFiles.csv user user@domain.com print filelist fields id,name,quotabytesused orderby quotabytesused descending ``` Display the 10 files with the largest quotaBytesUsed values for all users ``` gam config csv_output_row_limit 10 auto_batch_min 1 redirect csv ./BigQuotaFiles.csv multiprocess all users print filelist fields id,name,quotabytesused orderby quotabytesused descending ``` ## Saving filters in gam.cfg If you define a value for `csv_output_header_filter`, `csv_output_header_drop_filter`, `csv_output_header_force`, `csv_output_header_order`, `csv_output_row_filter`, `csv_output_row_drop_filter` or `csv_output_row_limit` in the `[DEFAULT]` section of `gam.cfg`, it will apply to every `gam print` command which is probably not desirable. You can store them in `gam.cfg` in named sections. ``` [Filter510] csv_output_header_filter = primaryEmail,name.fullName,phones.*value csv_output_row_filter = 'phones.\\\d+.value:regex:(?:^\\$510\\$ )|(?:^510[- ])\\\d{3}-\\\d{4}' $ gam selectfilter Filter510 print users name phone primaryEmail,name.fullName,phones.0.value testuser1@domain.com,Test User1,(510) 555-1212 testuser2@domain.com,Test User2,510-555-1212 testuser3@domain.com,Test User3,510 555-1212 ``` If you have multiple customers or domains in separate sections of gam.cfg, you use `select` to choose the customer/domain and `selectfilter` to choose a filter. ``` [foo] domain = foo.com customer_id = C111111111 config_dir = foo [goo] domain = goo.com customer_id = C222222222 config_dir = goo [Filter510] csv_output_header_filter = primaryEmail,name.fullName,phones.*value csv_output_row_filter = 'phones.\\\d+.value:regex:(?:^\\$510\\$ )|(?:^510[- ])\\\d{3}-\\\d{4}' $ gam select foo selectfilter Filter510 print users name phone primaryEmail,name.fullName,phones.0.value testuser1@foo.com,Test User1,(510) 555-1212 testuser2@foo.com,Test User2,510-555-1212 testuser3@foo.com,Test User2,510 555-1212 ```