Added csv_output_header_order variable to gam.cfg
Some checks are pending
Build and test GAM / build (Win64, build, 8, VC-WIN64A, windows-2022) (push) Waiting to run
Build and test GAM / build (aarch64, build, 3, linux-aarch64, [self-hosted linux arm64]) (push) Waiting to run
Build and test GAM / build (aarch64, build, 5, linux-aarch64, [self-hosted linux arm64], yes) (push) Waiting to run
Build and test GAM / build (aarch64, build, 7, darwin64-arm64, macos-14) (push) Waiting to run
Build and test GAM / build (x86_64, build, 1, linux-x86_64, ubuntu-22.04) (push) Waiting to run
Build and test GAM / build (x86_64, build, 2, linux-x86_64, ubuntu-24.04) (push) Waiting to run
Build and test GAM / build (x86_64, build, 4, linux-x86_64, ubuntu-22.04, yes) (push) Waiting to run
Build and test GAM / build (x86_64, build, 6, darwin64-x86_64, macos-13) (push) Waiting to run
Build and test GAM / build (x86_64, test, 10, ubuntu-24.04, 3.10) (push) Waiting to run
Build and test GAM / build (x86_64, test, 11, ubuntu-24.04, 3.11) (push) Waiting to run
Build and test GAM / build (x86_64, test, 12, ubuntu-24.04, 3.12) (push) Waiting to run
Build and test GAM / build (x86_64, test, 9, ubuntu-24.04, 3.9) (push) Waiting to run
Build and test GAM / merge (push) Blocked by required conditions
Build and test GAM / publish (push) Blocked by required conditions
CodeQL / Analyze (python) (push) Waiting to run
Check for Google Root CA Updates / check-apis (push) Waiting to run

This commit is contained in:
Ross Scroggs
2024-10-18 09:51:57 -07:00
parent 932fe5db02
commit eb33b6521b
11 changed files with 208 additions and 136 deletions

View File

@@ -1,4 +1,4 @@
!# CSV Output Filtering
# CSV Output Filtering
- [Python Regular Expressions](Python-Regular-Expressions) Search function
- [Definitions](#definitions)
- [Quoting rules](#quoting-rules)
@@ -11,9 +11,11 @@
- [Column row limiting](#column-row-limiting)
- [Saving filters in gam.cfg](#saving-filters-in-gamcfg)
There are five values in `gam.cfg` that can be used to filter the output from `gam print` commands.
There are seven values in `gam.cfg` that can be used to filter the output from `gam print` commands.
* `csv_output_header_filter` - A list of `<RegularExpressions>` used to select specific column headers to include
* `csv_output_header_drop_filter` - A list of `<RegularExpressions>` used to select specific column headers to exclude
* `csv_output_header_force` - A list of <Strings> used to specify the exact column headers to include
* `csv_output_header_order` - A list of <Strings> used to specify the column header order; any headers in the file but not in the list will appear after the headers in the list.
* `csv_output_row_filter` - A list or JSON dictionary used to include specific rows based on column values
* `csv_output_row_drop_filter` - A list or JSON dictionary used to exclude specific rows based on column values
* `csv_output_row_limit` - A limit on the number of rows written
@@ -334,7 +336,7 @@ gam config csv_output_row_limit 10 auto_batch_min 1 redirect csv ./BigQuotaFiles
```
## Saving filters in gam.cfg
If you define a value for `csv_output_header_filter`, `csv_output_header_drop_filter`, `csv_output_row_filter`, `csv_output_row_drop_filter` or `csv_output_row_limit` in the `[DEFAULT]` section of `gam.cfg`,
If you define a value for `csv_output_header_filter`, `csv_output_header_drop_filter`, `csv_output_header_force`, `csv_output_header_order`, `csv_output_row_filter`, `csv_output_row_drop_filter` or `csv_output_row_limit` in the `[DEFAULT]` section of `gam.cfg`,
it will apply to every `gam print` command which is probably not desirable. You can store them in `gam.cfg` in named sections.
```
[Filter510]

View File

@@ -12,7 +12,12 @@ See [Downloads-Installs-GAM7](https://github.com/GAM-team/GAM/wiki/Downloads-Ins
### 7.00.21
uprev version to test MSI installs
Added `csv_output_header_order` variable to `gam.cfg` that is a list of `<Strings>`
that are used to specify the order of column headers in the CSV file written by a gam print command.
Any headers in the file but not in the list will appear after the headers in the list.
This might be used when the CSV file data is to be processed by another program
that requires that the headers be in a particular order.
### 7.00.20

View File

@@ -51,6 +51,7 @@ The only `<VariableNames>` recognized in this `<Section>` are:
* `csv_output_header_filter`
* `csv_output_header_drop_filter`
* `csv_output_header_force`
* `csv_output_header_order`
* `csv_output_row_filter`
* `csv_output_row_filter_mode`
* `csv_output_row_drop_filter`

View File

@@ -811,7 +811,7 @@ User: testuser@domain.com, Drive Files/Folders: 261, Size: 13822521
```
Print file counts for a user.
```
$ gam user testuser@domain,com print filecounts showsize
$ gam user testuser@domain.com print filecounts showsize
Getting all Drive Files/Folders that match query ('me' in owners) for testuser@domain.com
Got 261 Drive Files/Folders that matched query ('me' in owners) for testuser@domain.com...
User,Total,Size,application/octet-stream,application/pdf,application/vnd.google-apps.document,application/vnd.google-apps.drawing,application/vnd.google-apps.drive-sdk.423565144751,application/vnd.google-apps.folder,application/vnd.google-apps.form,application/vnd.google-apps.jam,application/vnd.google-apps.presentation,application/vnd.google-apps.shortcut,application/vnd.google-apps.site,application/vnd.google-apps.spreadsheet,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.wordprocessingml.template,application/x-gzip,application/zip,image/jpeg,image/vnd.adobe.photoshop,text/csv,text/plain,text/rtf,text/x-sh
@@ -825,6 +825,14 @@ Got 261 Drive Files/Folders that matched query ('me' in owners) for testuser@dom
User,Total,Size,application/octet-stream,application/octet-stream-size,application/pdf,application/pdf-size,application/vnd.google-apps.document,application/vnd.google-apps.document-size,application/vnd.google-apps.drawing,application/vnd.google-apps.drawing-size,application/vnd.google-apps.drive-sdk.423565144751,application/vnd.google-apps.drive-sdk.423565144751-size,application/vnd.google-apps.folder,application/vnd.google-apps.folder-size,application/vnd.google-apps.form,application/vnd.google-apps.form-size,application/vnd.google-apps.jam,application/vnd.google-apps.jam-size,application/vnd.google-apps.presentation,application/vnd.google-apps.presentation-size,application/vnd.google-apps.shortcut,application/vnd.google-apps.shortcut-size,application/vnd.google-apps.site,application/vnd.google-apps.site-size,application/vnd.google-apps.spreadsheet,application/vnd.google-apps.spreadsheet-size,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet-size,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.wordprocessingml.document-size,application/vnd.openxmlformats-officedocument.wordprocessingml.template,application/vnd.openxmlformats-officedocument.wordprocessingml.template-size,application/x-gzip,application/x-gzip-size,application/zip,application/zip-size,image/jpeg,image/jpeg-size,image/vnd.adobe.photoshop,image/vnd.adobe.photoshop-size,text/csv,text/csv-size,text/plain,text/plain-size,text/rtf,text/rtf-size,text/x-sh,text/x-sh-size
testuser@domain.com,261,13822521,8,17,1,9879,98,52858,2,2048,1,0,68,0,3,0,1,1024,1,0,14,0,1,0,24,11264,1,8157,3,34407,1,25906,4,2768,2,765,8,16498,1,13613198,2,397,13,41461,3,1738,1,136
```
Print file counts for a Shared Drive
```
$ gam user testuser@domain.com print filecounts select <SharedDriveID> showsize
Getting all Drive Files/Folders for testuser@domain.com
Got 261 Drive Files/Folders for testuser@domain.com...
User,id,name,Total,Size,Item cap,application/octet-stream,application/pdf,application/vnd.google-apps.document,application/vnd.google-apps.drawing,application/vnd.google-apps.drive-sdk.423565144751,application/vnd.google-apps.folder,application/vnd.google-apps.form,application/vnd.google-apps.jam,application/vnd.google-apps.presentation,application/vnd.google-apps.shortcut,application/vnd.google-apps.site,application/vnd.google-apps.spreadsheet,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.wordprocessingml.template,application/x-gzip,application/zip,image/jpeg,image/vnd.adobe.photoshop,text/csv,text/plain,text/rtf,text/x-sh
testuser@domain.com,0AMzwfhFBpwLHUkWXYZ,Shared Drive Name,261,13822521,3.45%,8,1,98,2,1,68,3,1,1,14,1,24,1,3,1,4,2,8,1,2,13,3,1
```
Get file count summaries by OU; top level selector is ou, sub level selectors are ou_and_children
```
gam redirect csv ./TopLevelOUs.csv print ous showparent toplevelonly parentselector ou childselector ou_and_children fields orgunitpath

View File

@@ -2,6 +2,7 @@
- [API documentation](#api-documentation)
- [Query documentation](Users-Drive-Query)
- [Python Regular Expressions](Python-Regular-Expressions) Sub function
- [Folders with Limited Access Beta](#folders-with-limited-access-beta)
- [Permission Matches](Permission-Matches)
- [Definitions](#definitions)
- [Create files](#create-files)
@@ -31,6 +32,15 @@
* https://support.google.com/a/users/answer/7338880
* https://developers.google.com/docs/api/reference/rest
## Folders with Limited Access Beta
If you are enrolled in the Beta and want to access the `inheritedpermissionsdisabled` field,
you must turn on Drive API v3 beta.
```
gam config drive_v3_beta true user user@domain.com update drivefile <FolderID> inheritedpermissionsdisabled true
```
## Definitions
* [`<DriveFileEntity>`](Drive-File-Selection)
* [`<UserTypeEntity>`](Collections-of-Users)

View File

@@ -214,6 +214,12 @@ csv_output_header_force
A list of <Strings> used to specify the exact column headers
for inclusion in the CSV file written by a gam print command
Default: ''
csv_output_header_order
A list of <Strings> used to specify the order of column headers
for inclusion in the CSV file written by a gam print command
Any headers in the file but not in the list will appear after
the headers in the list
Default: ''
csv_output_line_terminator
Allowed values: cr, lf, crlf
Designates character(s) used to terminate the lines of a CSV file.
@@ -305,6 +311,9 @@ drive_max_results
how many should be retrieved in each API call
Default: 1000
Range: 1 - 1000
drive_v3_beta
Enable/disable use of Drive API v3 beta for Limited Folder Access testing
Default: False
drive_v3_native_names
Enable/disable use of Drive API v3 native column names
in all gam print/show commands related to Google Drive

View File

@@ -1,10 +1,16 @@
7.00.21
uprev version to test MSI installs
Added `csv_output_header_order` variable to `gam.cfg` that is a list of `<Strings>`
that are used to specify the order of column headers in the CSV file written by a gam print command.
Any headers in the file but not in the list will appear after the headers in the list.
This might be used when the CSV file data is to be processed by another program
that requires that the headers be in a particular order.
7.00.20
Fix Windows MSI installer issues on version upgrade. If you are having issues upgrading from a version older than 7.00.20 to this version or newer you may need to do a one time uninstall of GAM7 and then reinstall the new version. No configuration files will be lost during the uninstall / reinstall.
Fix Windows MSI installer issues on version upgrade. If you are having issues upgrading from a version older than 7.00.20 to this version or newer you may need to do a one time uninstall of GAM7 and then reinstall the new version.
No configuration files will be lost during the uninstall / reinstall.
7.00.19

View File

@@ -30,6 +30,8 @@ upgrade_only=false
gamversion="latest"
adminuser=""
regularuser=""
gam_x86_64_glibc_vers="2.35"
gam_arm64_glibc_vers="2.35"
strip_gam="--strip-components 0"
while getopts "hd:a:o:b:lp:u:r:v:s" OPTION
@@ -88,6 +90,7 @@ version_gt()
# MacOS < 10.13 doesn't support sort -V
echo "" | sort -V > /dev/null 2>&1
vsort_failed=$?
echo "Check:${2}"
if [ "${1}" = "${2}" ]; then
true
elif (( $vsort_failed != 0 )); then
@@ -97,6 +100,81 @@ else
fi
}
case $gamos in
[lL]inux)
gamos="linux"
if [ "$osversion" == "" ]; then
this_glibc_ver=$(ldd --version | awk '/ldd/{print $NF}')
else
this_glibc_ver=$osversion
fi
echo "This Linux distribution uses glibc $this_glibc_ver"
case $gamarch in
x86_64)
useglibc="legacy"
for gam_glibc_ver in $gam_x86_64_glibc_vers; do
if version_gt $this_glibc_ver $gam_glibc_ver; then
useglibc="glibc$gam_glibc_ver"
echo_green "Using GAM compiled against $useglibc"
break
fi
done
gamfile="linux-x86_64-$useglibc.tar.xz";;
arm|arm64|aarch64)
useglibc=""
for gam_glibc_ver in $gam_arm64_glibc_vers; do
if version_gt $this_glibc_ver $gam_glibc_ver; then
useglibc="glibc$gam_glibc_ver"
echo_green "Using GAM compiled against $useglibc"
break
fi
done
if [ "$useglibc" == "" ]; then
echo_red "Sorry, you need to be running at least glibc $useglibc to run GAM"
exit
fi
gamfile="linux-aarch64-$useglibc.tar.xz";;
*)
echo_red "ERROR: this installer currently only supports x86_64 and arm64 Linux. Looks like you're running on $gamarch. Exiting."
exit
esac
;;
[Mm]ac[Oo][sS]|[Dd]arwin)
gamos="macos"
fullversion=$(sw_vers -productVersion)
osversion=${fullversion:0:2}
case $gamarch in
x86_64)
gamfile="macos-x86_64.tar.xz"
minimum_version=13
;;
arm|arm64|aarch64)
gamfile="macos-aarch64.tar.xz"
minimum_version=14
;;
*)
echo_red "ERROR: this installer currently only supports x86_64 and arm64 MacOS. Looks like you're running on $gamarch. Exiting."
exit
;;
esac
if [[ "$osversion" -ge "$minimum_version" ]]; then
echo_green "You are running MacOS ${fullversion}, good. Using GAM with ${gamfile}."
else
echo_red "Sorry, you are running MacOS ${fullversion} but GAM on ${gamarch} requires MacOS ${minimum_version}. Exiting."
exit
fi
;;
MINGW64_NT*)
gamos="windows"
echo "You are running Windows"
gamfile="-windows-x86_64.zip"
;;
*)
echo_red "Sorry, this installer currently only supports Linux and MacOS. Looks like you're running on $gamos. Exiting."
exit
;;
esac
if [ "$gamversion" == "latest" ]; then
release_url="https://api.github.com/repos/GAM-team/GAM/releases/latest"
elif [ "$gamversion" == "prerelease" -o "$gamversion" == "draft" ]; then
@@ -107,19 +185,13 @@ fi
if [ -z ${GHCLIENT+x} ]; then
check_type="unauthenticated"
curl_opts=( )
echo_yellow "Checking GitHub URL $release_url for $gamversion GAM release ($check_type)..."
release_json=$(curl -s "$release_url" 2>&1 /dev/null)
else
check_type="authenticated"
curl_opts=( "$GHCLIENT" )
echo_yellow "Checking GitHub URL $release_url for $gamversion GAM release ($check_type)..."
release_json=$(curl -s "$GHCLIENT" "$release_url" 2>&1 /dev/null)
fi
echo_yellow "Checking GitHub URL $release_url for $gamversion GAM release ($check_type)..."
release_json=$(curl \
--silent \
"${curl_opts[@]}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"$release_url" \
2>&1 /dev/null)
echo_yellow "Getting file and download URL..."
# Python is sadly the nearest to universal way to safely handle JSON with Bash
@@ -143,9 +215,11 @@ if type(release) is list:
break
try:
for asset in release['assets']:
print(asset[attrib])
#else:
# print('ERROR: Attribute: {0} for version {1} not found'.format(attrib, gamversion))
if asset[attrib].endswith('$gamfile'):
print(asset[attrib])
break
else:
print('ERROR: Attribute: {0} for $gamfile version {1} not found'.format(attrib, gamversion))
except KeyError:
print('ERROR: assets value not found in JSON value of:\n\n%s' % release)"
@@ -171,119 +245,41 @@ if (( $rc != 0 )); then
echo_red "ERROR: No version of python installed."
exit
fi
download_urls=$(echo "$release_json" | $pycmd -c "$pycode" browser_download_url "$gamversion")
if [[ ${download_urls:0:5} = "ERROR" ]]; then
echo_red "${download_urls}"
browser_download_url=$(echo "$release_json" | $pycmd -c "$pycode" browser_download_url "$gamversion")
if [[ ${browser_download_url:0:5} = "ERROR" ]]; then
echo_red "${browser_download_url}"
exit
fi
name=$(echo "$release_json" | $pycmd -c "$pycode" name "$gamversion")
if [[ ${name:0:5} = "ERROR" ]]; then
echo_red "${name}"
exit
fi
case $gamos in
[lL]inux)
gamos="linux"
download_urls=$(echo -e "$download_urls" | grep "\-linux-")
if [ "$osversion" == "" ]; then
this_glibc_ver=$(ldd --version | awk '/ldd/{print $NF}')
else
this_glibc_ver=$osversion
fi
echo "This Linux distribution uses glibc $this_glibc_ver"
case $gamarch in
x86_64)
download_urls=$(echo -e "$download_urls" | grep "\-x86_64-")
gam_x86_64_glibc_vers=$(echo -e "$download_urls" | \
grep --only-matching 'glibc[0-9\.]*\.tar\.xz$' \
| cut -c 6-9 )
useglibc="legacy"
for gam_glibc_ver in $gam_x86_64_glibc_vers; do
if version_gt $this_glibc_ver $gam_glibc_ver; then
useglibc="glibc$gam_glibc_ver"
echo_green "Using GAM compiled against $useglibc"
break
fi
done
download_url=$(echo -e "$download_urls" | grep "$useglibc")
;;
arm|arm64|aarch64)
download_urls=$(echo -e "$download_urls" | grep "\-aarch64-")
gam_arm64_glibc_vers=$(echo -e "$download_urls" | \
grep --only-matching 'glibc[0-9\.]*\.tar\.xz$' \
| cut -c 6-9 )
useglibc="legacy"
for gam_glibc_ver in $gam_arm64_glibc_vers; do
if version_gt $this_glibc_ver $gam_glibc_ver; then
useglibc="glibc$gam_glibc_ver"
echo_green "Using GAM compiled against $useglibc"
break
fi
done
download_url=$(echo -e "$download_urls" | grep "$useglibc")
;;
*)
echo_red "ERROR: this installer currently only supports x86_64 and arm64 Linux. Looks like you're running on $gamarch. Exiting."
exit
esac
;;
[Mm]ac[Oo][sS]|[Dd]arwin)
gamos="macos"
fullversion=$(sw_vers -productVersion)
# override osversion only if it wasn't set by cli arguments
osversion=${osversion:-${fullversion:0:2}}
download_urls=$(echo -e "$download_urls" | grep "\-macos-")
case $gamarch in
x86_64)
download_url=$(echo -e "$download_urls" | grep "\-x86_64")
minimum_version=13
;;
arm|arm64|aarch64)
download_url=$(echo -e "$download_urls" | grep "\-aarch64")
minimum_version=14
;;
*)
echo_red "ERROR: this installer currently only supports x86_64 and arm64 MacOS. Looks like you're running on ${gamarch}. Exiting."
exit
;;
esac
if [[ "$osversion" -ge "$minimum_version" ]]; then
echo_green "You are running MacOS ${fullversion}, good. Using GAM with ${download_url}."
else
echo_red "Sorry, you are running MacOS ${fullversion} but GAM on ${gamarch} requires MacOS ${minimum_version}. Exiting."
exit
fi
;;
MINGW64_NT*)
gamos="windows"
echo "You are running Windows"
download_url=$(echo -e "$download_urls" | grep "\-windows-" | grep ".zip")
;;
*)
echo_red "Sorry, this installer currently only supports Linux and MacOS. Looks like you're running on ${gamos}. Exiting."
exit
;;
esac
# Temp dir for archive
#temp_archive_dir=$(mktemp -d)
temp_archive_dir=$(mktemp -d 2>/dev/null || mktemp -d -t 'mytmpdir')
# Clean up after ourselves even if we are killed with CTRL-C
trap "rm -rf $temp_archive_dir" EXIT
# hack to grab the end of the URL which should be the filename.
name=$(echo -e "$download_url" | rev | cut -f1 -d "/" | rev)
echo_yellow "Downloading ${download_url} to $temp_archive_dir ($check_type)..."
echo_yellow "Downloading file $name from $browser_download_url to $temp_archive_dir ($check_type)..."
# Save archive to temp w/o losing our path
(cd "$temp_archive_dir" && curl -O -L -s "${curl_opts[@]}" "$download_url")
if [ -z ${GHCLIENT+x} ]; then
(cd "$temp_archive_dir" && curl -O -L $browser_download_url)
else
(cd "$temp_archive_dir" && curl -O -L $GHCLIENT $browser_download_url)
fi
mkdir -p "$target_dir"
echo_yellow "Extracting archive to $target_dir"
if [[ "$name" =~ tar.xz|tar.gz|tar ]]; then
if [[ "${name}" == *.tar.xz ]]; then
tar $strip_gam -xf "$temp_archive_dir"/"$name" -C "$target_dir"
elif [[ "${name}" == *.tar ]]; then
tar $strip_gam -xf "$temp_archive_dir"/"$name" -C "$target_dir"
elif [[ "$name" == *.zip ]]; then
unzip -o "${temp_archive_dir}/${name}" -d "${target_dir}"
else
echo "I don't know what to do with files like ${name}. Giving up."
exit 1
unzip "${temp_archive_dir}/${name}" -d "${target_dir}"
fi
rc=$?
if (( $rc != 0 )); then

View File

@@ -4008,7 +4008,7 @@ def SetGlobalVariables():
GC.Values[itemName] = _getCfgPassword(sectionName, itemName)
elif varType == GC.TYPE_STRING:
GC.Values[itemName] = _getCfgString(sectionName, itemName)
elif varType in {GC.TYPE_STRINGLIST, GC.TYPE_HEADERFORCE}:
elif varType in {GC.TYPE_STRINGLIST, GC.TYPE_HEADERFORCE, GC.TYPE_HEADERORDER}:
GC.Values[itemName] = _getCfgStringList(sectionName, itemName)
elif varType == GC.TYPE_FILE:
GC.Values[itemName] = _getCfgFile(sectionName, itemName)
@@ -4031,6 +4031,7 @@ def SetGlobalVariables():
else:
GC.Values[GC.CSV_OUTPUT_HEADER_FILTER] = _getCfgHeaderFilter(outputFilterSectionName, GC.CSV_OUTPUT_HEADER_FILTER)
GC.Values[GC.CSV_OUTPUT_HEADER_DROP_FILTER] = _getCfgHeaderFilter(outputFilterSectionName, GC.CSV_OUTPUT_HEADER_DROP_FILTER)
GC.Values[GC.CSV_OUTPUT_HEADER_ORDER] = _getCfgStringList(outputFilterSectionName, GC.CSV_OUTPUT_HEADER_ORDER)
GC.Values[GC.CSV_OUTPUT_ROW_FILTER] = _getCfgRowFilter(outputFilterSectionName, GC.CSV_OUTPUT_ROW_FILTER)
GC.Values[GC.CSV_OUTPUT_ROW_FILTER_MODE] = _getCfgChoice(outputFilterSectionName, GC.CSV_OUTPUT_ROW_FILTER_MODE)
GC.Values[GC.CSV_OUTPUT_ROW_DROP_FILTER] = _getCfgRowFilter(outputFilterSectionName, GC.CSV_OUTPUT_ROW_DROP_FILTER)
@@ -4151,7 +4152,7 @@ def SetGlobalVariables():
if GM.Globals[GM.PID] == 0:
for itemName, itemEntry in sorted(iter(GC.VAR_INFO.items())):
varType = itemEntry[GC.VAR_TYPE]
if varType in {GC.TYPE_HEADERFILTER, GC.TYPE_HEADERFORCE, GC.TYPE_ROWFILTER}:
if varType in {GC.TYPE_HEADERFILTER, GC.TYPE_HEADERFORCE, GC.TYPE_HEADERORDER, GC.TYPE_ROWFILTER}:
GM.Globals[GM.PARSER].set(sectionName, itemName, '')
elif (varType == GC.TYPE_INTEGER) and itemName in {GC.CSV_INPUT_ROW_LIMIT, GC.CSV_OUTPUT_ROW_LIMIT}:
GM.Globals[GM.PARSER].set(sectionName, itemName, '0')
@@ -4164,6 +4165,8 @@ def SetGlobalVariables():
GC.Values[GC.CSV_OUTPUT_HEADER_DROP_FILTER] = GM.Globals[GM.CSV_OUTPUT_HEADER_DROP_FILTER][:]
if not GC.Values[GC.CSV_OUTPUT_HEADER_FORCE]:
GC.Values[GC.CSV_OUTPUT_HEADER_FORCE] = GM.Globals[GM.CSV_OUTPUT_HEADER_FORCE][:]
if not GC.Values[GC.CSV_OUTPUT_HEADER_ORDER]:
GC.Values[GC.CSV_OUTPUT_HEADER_ORDER] = GM.Globals[GM.CSV_OUTPUT_HEADER_ORDER][:]
if not GC.Values[GC.CSV_OUTPUT_ROW_FILTER]:
GC.Values[GC.CSV_OUTPUT_ROW_FILTER] = GM.Globals[GM.CSV_OUTPUT_ROW_FILTER][:]
GC.Values[GC.CSV_OUTPUT_ROW_FILTER_MODE] = GM.Globals[GM.CSV_OUTPUT_ROW_FILTER_MODE]
@@ -7731,6 +7734,7 @@ class CSVPrintFile():
if not self.headerForce and titles is not None:
self.SetTitles(titles)
self.SetJSONTitles(titles)
self.SetHeaderOrder(GC.Values[GC.CSV_OUTPUT_HEADER_ORDER])
if GM.Globals.get(GM.CSV_OUTPUT_COLUMN_DELIMITER) is None:
GM.Globals[GM.CSV_OUTPUT_COLUMN_DELIMITER] = GC.Values.get(GC.CSV_OUTPUT_COLUMN_DELIMITER, ',')
self.SetColumnDelimiter(GM.Globals[GM.CSV_OUTPUT_COLUMN_DELIMITER])
@@ -7740,10 +7744,12 @@ class CSVPrintFile():
GM.Globals[GM.CSV_OUTPUT_NO_ESCAPE_CHAR] = GC.Values.get(GC.CSV_OUTPUT_NO_ESCAPE_CHAR, False)
self.SetNoEscapeChar(GM.Globals[GM.CSV_OUTPUT_NO_ESCAPE_CHAR])
self.SetQuoteChar(GM.Globals[GM.CSV_OUTPUT_QUOTE_CHAR])
if GM.Globals.get(GM.CSV_OUTPUT_SORT_HEADERS) is None:
# if GM.Globals.get(GM.CSV_OUTPUT_SORT_HEADERS) is None:
if not GM.Globals.get(GM.CSV_OUTPUT_SORT_HEADERS):
GM.Globals[GM.CSV_OUTPUT_SORT_HEADERS] = GC.Values.get(GC.CSV_OUTPUT_SORT_HEADERS, [])
self.SetSortHeaders(GM.Globals[GM.CSV_OUTPUT_SORT_HEADERS])
if GM.Globals.get(GM.CSV_OUTPUT_TIMESTAMP_COLUMN) is None:
# if GM.Globals.get(GM.CSV_OUTPUT_TIMESTAMP_COLUMN) is None:
if not GM.Globals.get(GM.CSV_OUTPUT_TIMESTAMP_COLUMN):
GM.Globals[GM.CSV_OUTPUT_TIMESTAMP_COLUMN] = GC.Values.get(GC.CSV_OUTPUT_TIMESTAMP_COLUMN, '')
self.SetTimestampColumn(GM.Globals[GM.CSV_OUTPUT_TIMESTAMP_COLUMN])
self.SetFormatJSON(False)
@@ -8431,6 +8437,15 @@ class CSVPrintFile():
self.SetTitles(headerForce)
self.SetJSONTitles(headerForce)
def SetHeaderOrder(self, headerOrder):
self.headerOrder = headerOrder
def orderHeaders(self, titlesList):
for title in self.headerOrder:
if title in titlesList:
titlesList.remove(title)
return self.headerOrder+titlesList
@staticmethod
def HeaderFilterMatch(filters, title):
for filterStr in filters:
@@ -8870,14 +8885,21 @@ class CSVPrintFile():
self.FixNodataTitles()
if self.mapDrive3Titles:
self. MapDrive3TitlesToDrive2()
else:
self.titlesList = self.headerForce
if self.timestampColumn:
self.AddTitle(self.timestampColumn)
if self.headerOrder:
self.titlesList = self.orderHeaders(self.titlesList)
titlesList = self.titlesList
else:
if self.fixPaths:
self.FixPathsTitles(self.JSONtitlesList)
if not self.rows and self.nodataFields is not None:
self.FixNodataTitles()
if not self.headerForce:
if self.fixPaths:
self.FixPathsTitles(self.JSONtitlesList)
if not self.rows and self.nodataFields is not None:
self.FixNodataTitles()
else:
self.JSONtitlesList = self.headerForce
if self.timestampColumn:
for i, v in enumerate(self.JSONtitlesList):
if v.startswith('JSON'):
@@ -8886,6 +8908,8 @@ class CSVPrintFile():
break
else:
self.AddJSONTitle(self.timestampColumn)
if self.headerOrder:
self.JSONtitlesList = self.orderHeaders(self.JSONtitlesList)
titlesList = self.JSONtitlesList
normalizeSortHeaders()
if (not self.todrive) or self.todrive['localcopy']:
@@ -9623,7 +9647,7 @@ def ProcessGAMCommandMulti(pid, numItems, logCmd, mpQueueCSVFile, mpQueueStdout,
csvColumnDelimiter, csvNoEscapeChar, csvQuoteChar,
csvSortHeaders, csvTimestampColumn,
csvHeaderFilter, csvHeaderDropFilter,
csvHeaderForce,
csvHeaderForce, csvHeaderOrder,
csvRowFilter, csvRowFilterMode, csvRowDropFilter, csvRowDropFilterMode,
csvRowLimit,
showGettings, showGettingsGotNL,
@@ -9647,13 +9671,14 @@ def ProcessGAMCommandMulti(pid, numItems, logCmd, mpQueueCSVFile, mpQueueStdout,
GM.Globals[GM.CSV_OUTPUT_HEADER_DROP_FILTER] = csvHeaderDropFilter[:]
GM.Globals[GM.CSV_OUTPUT_HEADER_FILTER] = csvHeaderFilter[:]
GM.Globals[GM.CSV_OUTPUT_HEADER_FORCE] = csvHeaderForce[:]
GM.Globals[GM.CSV_OUTPUT_HEADER_ORDER] = csvHeaderOrder[:]
GM.Globals[GM.CSV_OUTPUT_QUOTE_CHAR] = csvQuoteChar
GM.Globals[GM.CSV_OUTPUT_ROW_DROP_FILTER] = csvRowDropFilter[:]
GM.Globals[GM.CSV_OUTPUT_ROW_DROP_FILTER_MODE] = csvRowDropFilterMode
GM.Globals[GM.CSV_OUTPUT_ROW_FILTER] = csvRowFilter[:]
GM.Globals[GM.CSV_OUTPUT_ROW_FILTER_MODE] = csvRowFilterMode
GM.Globals[GM.CSV_OUTPUT_ROW_LIMIT] = csvRowLimit
GM.Globals[GM.CSV_OUTPUT_SORT_HEADERS] = csvSortHeaders
GM.Globals[GM.CSV_OUTPUT_SORT_HEADERS] = csvSortHeaders[:]
GM.Globals[GM.CSV_OUTPUT_TIMESTAMP_COLUMN] = csvTimestampColumn
GM.Globals[GM.CSV_TODRIVE] = todrive.copy()
GM.Globals[GM.DEBUG_LEVEL] = debugLevel
@@ -9661,9 +9686,9 @@ def ProcessGAMCommandMulti(pid, numItems, logCmd, mpQueueCSVFile, mpQueueStdout,
GM.Globals[GM.OUTPUT_TIMEFORMAT] = output_timeformat
GM.Globals[GM.NUM_BATCH_ITEMS] = numItems
GM.Globals[GM.PID] = pid
GM.Globals[GM.PRINT_AGU_DOMAINS] = printAguDomains
GM.Globals[GM.PRINT_CROS_OUS] = printCrosOUs
GM.Globals[GM.PRINT_CROS_OUS_AND_CHILDREN] = printCrosOUsAndChildren
GM.Globals[GM.PRINT_AGU_DOMAINS] = printAguDomains[:]
GM.Globals[GM.PRINT_CROS_OUS] = printCrosOUs[:]
GM.Globals[GM.PRINT_CROS_OUS_AND_CHILDREN] = printCrosOUsAndChildren[:]
GM.Globals[GM.SAVED_STDOUT] = None
GM.Globals[GM.SHOW_GETTINGS] = showGettings
GM.Globals[GM.SHOW_GETTINGS_GOT_NL] = showGettingsGotNL
@@ -9870,6 +9895,7 @@ def MultiprocessGAMCommands(items, showCmds):
GC.Values[GC.CSV_OUTPUT_HEADER_FILTER],
GC.Values[GC.CSV_OUTPUT_HEADER_DROP_FILTER],
GC.Values[GC.CSV_OUTPUT_HEADER_FORCE],
GC.Values[GC.CSV_OUTPUT_HEADER_ORDER],
GC.Values[GC.CSV_OUTPUT_ROW_FILTER],
GC.Values[GC.CSV_OUTPUT_ROW_FILTER_MODE],
GC.Values[GC.CSV_OUTPUT_ROW_DROP_FILTER],
@@ -25780,7 +25806,7 @@ CHAT_UPDATE_SPACE_PERMISSIONS_MAP = {
'managewebhooks': 'manageWebhooks',
'replymessages': 'replyMessages',
}
# gam <UserTypeEntity> update chatspace <ChatSpace>
# [restricted|(audience <String>)]|
# ([displayname <String>]
@@ -57236,7 +57262,7 @@ def createDriveFile(users):
continue
result = callGAPI(drive.files(), 'create',
throwReasons=GAPI.DRIVE_USER_THROW_REASONS+[GAPI.FORBIDDEN, GAPI.INSUFFICIENT_PERMISSIONS, GAPI.INSUFFICIENT_PARENT_PERMISSIONS,
GAPI.INVALID, GAPI.BAD_REQUEST, GAPI.CANNOT_ADD_PARENT,
GAPI.PERMISSION_DENIED, GAPI.INVALID, GAPI.BAD_REQUEST, GAPI.CANNOT_ADD_PARENT,
GAPI.FILE_NOT_FOUND, GAPI.UNKNOWN_ERROR, GAPI.INTERNAL_ERROR,
GAPI.STORAGE_QUOTA_EXCEEDED, GAPI.TEAMDRIVES_SHARING_RESTRICTION_NOT_ALLOWED,
GAPI.TEAMDRIVE_FILE_LIMIT_EXCEEDED, GAPI.TEAMDRIVE_HIERARCHY_TOO_DEEP,
@@ -57275,7 +57301,7 @@ def createDriveFile(users):
row.update(addCSVData)
csvPF.WriteRow(row)
except (GAPI.forbidden, GAPI.insufficientFilePermissions, GAPI.insufficientParentPermissions,
GAPI.invalidQuery, GAPI.invalid, GAPI.badRequest, GAPI.cannotAddParent,
GAPI.invalidQuery, GAPI.permissionDenied, GAPI.invalid, GAPI.badRequest, GAPI.cannotAddParent,
GAPI.fileNotFound, GAPI.unknownError, GAPI.storageQuotaExceeded, GAPI.teamDrivesSharingRestrictionNotAllowed,
GAPI.teamDriveFileLimitExceeded, GAPI.teamDriveHierarchyTooDeep,
GAPI.uploadTooLarge, GAPI.teamDrivesShortcutFileNotSupported) as e:

View File

@@ -117,6 +117,8 @@ CSV_OUTPUT_HEADER_FILTER = 'csv_output_header_filter'
CSV_OUTPUT_HEADER_DROP_FILTER = 'csv_output_header_drop_filter'
# Force output column headers
CSV_OUTPUT_HEADER_FORCE = 'csv_output_header_force'
# Orde output column headers
CSV_OUTPUT_HEADER_ORDER = 'csv_output_header_order'
# Line terminator in CSV output file
CSV_OUTPUT_LINE_TERMINATOR = 'csv_output_line_terminator'
# Quote character in CSV output file
@@ -309,7 +311,8 @@ CSV_INPUT_ROW_FILTER_ITEMS = {CSV_INPUT_ROW_FILTER, CSV_INPUT_ROW_FILTER_MODE,
CSV_INPUT_ROW_DROP_FILTER, CSV_INPUT_ROW_DROP_FILTER_MODE,
CSV_INPUT_ROW_LIMIT}
CSV_OUTPUT_ROW_FILTER_ITEMS = {CSV_OUTPUT_HEADER_FILTER, CSV_OUTPUT_HEADER_DROP_FILTER, CSV_OUTPUT_HEADER_FORCE,
CSV_OUTPUT_ROW_FILTER_ITEMS = {CSV_OUTPUT_HEADER_FILTER, CSV_OUTPUT_HEADER_DROP_FILTER,
CSV_OUTPUT_HEADER_FORCE, CSV_OUTPUT_HEADER_ORDER,
CSV_OUTPUT_ROW_FILTER, CSV_OUTPUT_ROW_FILTER_MODE,
CSV_OUTPUT_ROW_DROP_FILTER, CSV_OUTPUT_ROW_DROP_FILTER_MODE,
CSV_OUTPUT_ROW_LIMIT}
@@ -351,6 +354,7 @@ Defaults = {
CSV_OUTPUT_HEADER_FILTER: '',
CSV_OUTPUT_HEADER_DROP_FILTER: '',
CSV_OUTPUT_HEADER_FORCE: '',
CSV_OUTPUT_HEADER_ORDER: '',
CSV_OUTPUT_LINE_TERMINATOR: 'lf',
CSV_OUTPUT_QUOTE_CHAR: '\'"\'',
CSV_OUTPUT_ROW_FILTER: '',
@@ -460,6 +464,7 @@ TYPE_FILE = 'file'
TYPE_FLOAT = 'floa'
TYPE_HEADERFILTER = 'heaf'
TYPE_HEADERFORCE = 'hefo'
TYPE_HEADERORDER = 'heor'
TYPE_INTEGER = 'inte'
TYPE_LANGUAGE = 'lang'
TYPE_LOCALE = 'locl'
@@ -514,6 +519,7 @@ VAR_INFO = {
CSV_OUTPUT_HEADER_FILTER: {VAR_TYPE: TYPE_HEADERFILTER},
CSV_OUTPUT_HEADER_DROP_FILTER: {VAR_TYPE: TYPE_HEADERFILTER},
CSV_OUTPUT_HEADER_FORCE: {VAR_TYPE: TYPE_HEADERFORCE},
CSV_OUTPUT_HEADER_ORDER: {VAR_TYPE: TYPE_HEADERORDER},
CSV_OUTPUT_LINE_TERMINATOR: {VAR_TYPE: TYPE_CHOICE, VAR_CHOICES: {'cr': '\r', 'lf': '\n', 'crlf': '\r\n'}},
CSV_OUTPUT_QUOTE_CHAR: {VAR_TYPE: TYPE_CHARACTER},
CSV_OUTPUT_ROW_FILTER: {VAR_TYPE: TYPE_ROWFILTER},

View File

@@ -65,6 +65,8 @@ CSV_OUTPUT_HEADER_DROP_FILTER = 'cohd'
CSV_OUTPUT_HEADER_FILTER = 'cohf'
# Force output column headers
CSV_OUTPUT_HEADER_FORCE = 'cofh'
# Order output column headers
CSV_OUTPUT_HEADER_ORDER = 'coho'
# No escape character in CSV output file
CSV_OUTPUT_NO_ESCAPE_CHAR = 'cone'
# Quote character in CSV output file
@@ -80,7 +82,7 @@ CSV_OUTPUT_ROW_FILTER_MODE = 'corm'
# Limit number of output rows
CSV_OUTPUT_ROW_LIMIT = 'corl'
# Add timestamp column to CSV output file
CSV_OUTPUT_TIMESTAMP_COLUMN = 'csv_output_timestamp_column'
CSV_OUTPUT_TIMESTAMP_COLUMN = 'cotc'
# Output sort headers
CSV_OUTPUT_SORT_HEADERS = 'cosh'
# CSV todrive options
@@ -235,6 +237,7 @@ Globals = {
CSV_OUTPUT_HEADER_DROP_FILTER: [],
CSV_OUTPUT_HEADER_FILTER: [],
CSV_OUTPUT_HEADER_FORCE: [],
CSV_OUTPUT_HEADER_ORDER: [],
CSV_OUTPUT_NO_ESCAPE_CHAR: None,
CSV_OUTPUT_QUOTE_CHAR: None,
CSV_OUTPUT_ROW_DROP_FILTER: [],