deepblue/GoogleDriveManagement
1
0
Fork 0
mirror of https://github.com/GAM-team/GAM.git synced 2025-07-05 12:13:34 +00:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Added option skiprows <Integer> to gam csv|loop

Browse Source
This commit is contained in:
Ross Scroggs
2024-02-09 10:00:33 -08:00
parent 2d091c8ca0
commit adbee45073
7 changed files with 69 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/Bulk-Processing.md
View File

@ -72,13 +72,13 @@ gam redirect stdout ./NewStudents.out redirect stderr ./NewStudents.err tbatch N
gam csv <FileName>|-|(gsheet <UserGoogleSheet>)|(gdoc <UserGoogleDoc>) [charset <Charset>] [warnifnodata] gam csv <FileName>|-|(gsheet <UserGoogleSheet>)|(gdoc <UserGoogleDoc>) [charset <Charset>] [warnifnodata]
[columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>] [columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>]
(matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]] (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]]
[maxrows <Integer>] [skiprows <Integer>] [maxrows <Integer>]
gam <GAMArgumentList> gam <GAMArgumentList>
gam loop <FileName>|-|(gsheet <UserGoogleSheet>)|(gdoc <UserGoogleDoc>) [charset <Charset>] [warnifnodata] gam loop <FileName>|-|(gsheet <UserGoogleSheet>)|(gdoc <UserGoogleDoc>) [charset <Charset>] [warnifnodata]
[columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>] [columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>]
(matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]] (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]]
[maxrows <Integer>] [skiprows <Integer>] [maxrows <Integer>]
gam <GAMArgumentList> gam <GAMArgumentList>
``` ```
* `gam csv` - Use parallel processing * `gam csv` - Use parallel processing
@ -93,7 +93,10 @@ gam loop <FileName>|-|(gsheet <UserGoogleSheet>)|(gdoc <UserGoogleDoc>) [charset
* `fields <FieldNameList>` - The column headings of a CSV file that does not contain column headings. * `fields <FieldNameList>` - The column headings of a CSV file that does not contain column headings.
* `(matchfield|skipfield <FieldName> <RegularExpression>)*` - The criteria to select rows from the CSV file; can be used multiple times; if not specified, all rows are selected * `(matchfield|skipfield <FieldName> <RegularExpression>)*` - The criteria to select rows from the CSV file; can be used multiple times; if not specified, all rows are selected
* `showcmds` - Write `timestamp,command number/number of commands,command` to stderr when each command starts; write `timestamp, command number/numberof commands,complete` to stderr when command completes * `showcmds` - Write `timestamp,command number/number of commands,command` to stderr when each command starts; write `timestamp, command number/numberof commands,complete` to stderr when command completes
* `maxrows <Integer>` - Limit the number of filtered rows processed from the CSV file/Google Sheet. * `skiprows <Integer>` - Skip filtered rows from the CSV file/Google Sheet.
* `skiprows 0` - All rows are processed, this is the default
* `skiprows N` - The first N filtered rows are skipped
* `maxrows <Integer>` - Limit the number of filtered rows processed from the CSV file/Google Sheet after any skipped rows.
* `maxrows 0` - All rows are processed, this is the default * `maxrows 0` - All rows are processed, this is the default
* `maxrows N` - N filtered rows are processed * `maxrows N` - N filtered rows are processed

6
docs/GamUpdates.md
View File

@ -10,6 +10,12 @@ Add the `-s` option to the end of the above commands to suppress creating the `g
See [Downloads](https://github.com/taers232c/GAMADV-XTD3/wiki/Downloads) for Windows or other options, including manual installation See [Downloads](https://github.com/taers232c/GAMADV-XTD3/wiki/Downloads) for Windows or other options, including manual installation
### 6.68.04
Added option `skiprows <Integer>` to `gam csv|loop` that causes GAM to skip processing the first `<Integer>` filtered rows.
* See: https://github.com/taers232c/GAMADV-XTD3/wiki/Bulk-Processing#csv-files
### 6.68.03 ### 6.68.03
Fixed bug in `gam <UserTypeEntity> create drivefileacl` that caused a trap. Fixed bug in `gam <UserTypeEntity> create drivefileacl` that caused a trap.

12
docs/How-to-Upgrade-from-Standard-GAM.md
View File

@ -334,12 +334,12 @@ writes the credentials into the file oauth2.txt.
admin@server:/Users/admin/bin/gamadv-xtd3$ rm -f /Users/admin/GAMConfig/oauth2.txt admin@server:/Users/admin/bin/gamadv-xtd3$ rm -f /Users/admin/GAMConfig/oauth2.txt
admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam version admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam version
WARNING: Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: /Users/admin/GAMConfig/oauth2.txt, Not Found WARNING: Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: /Users/admin/GAMConfig/oauth2.txt, Not Found
GAMADV-XTD3 6.68.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource GAMADV-XTD3 6.68.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
Ross Scroggs <ross.scroggs@gmail.com> Ross Scroggs <ross.scroggs@gmail.com>
Python 3.12.1 64-bit final Python 3.12.2 64-bit final
MacOS Sonoma 14.2.1 x86_64 MacOS Sonoma 14.2.1 x86_64
Path: /Users/admin/bin/gamadv-xtd3 Path: /Users/admin/bin/gamadv-xtd3
Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain.com Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain: domain.com
admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam oauth create admin@server:/Users/admin/bin/gamadv-xtd3$ ./gam oauth create
@ -1002,12 +1002,12 @@ writes the credentials into the file oauth2.txt.
C:\GAMADV-XTD3>del C:\GAMConfig\oauth2.txt C:\GAMADV-XTD3>del C:\GAMConfig\oauth2.txt
C:\GAMADV-XTD3>gam version C:\GAMADV-XTD3>gam version
WARNING: Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: C:\GAMConfig\oauth2.txt, Not Found WARNING: Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, Item: oauth2_txt, Value: C:\GAMConfig\oauth2.txt, Not Found
GAMADV-XTD3 6.68.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource GAMADV-XTD3 6.68.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
Ross Scroggs <ross.scroggs@gmail.com> Ross Scroggs <ross.scroggs@gmail.com>
Python 3.12.1 64-bit final Python 3.12.2 64-bit final
Windows-10-10.0.17134 AMD64 Windows-10-10.0.17134 AMD64
Path: C:\GAMADV-XTD3 Path: C:\GAMADV-XTD3
Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, customer_id: my_customer, domain.com Config File: C:\GAMConfig\gam.cfg, Section: DEFAULT, customer_id: my_customer, domain: domain.com
C:\GAMADV-XTD3>gam oauth create C:\GAMADV-XTD3>gam oauth create

30
docs/Version-and-Help.md
View File

@ -3,41 +3,41 @@
Print the current version of Gam with details Print the current version of Gam with details
``` ```
gam version gam version
GAMADV-XTD3 6.68.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource GAMADV-XTD3 6.68.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
Ross Scroggs <ross.scroggs@gmail.com> Ross Scroggs <ross.scroggs@gmail.com>
Python 3.12.1 64-bit final Python 3.12.2 64-bit final
MacOS Sonoma 14.2.1 x86_64 MacOS Sonoma 14.2.1 x86_64
Path: /Users/Admin/bin/gamadv-xtd3 Path: /Users/Admin/bin/gamadv-xtd3
Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain.com Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain: domain.com
Time: 2023-06-02T21:10:00-07:00 Time: 2023-06-02T21:10:00-07:00
``` ```
Print the current version of Gam with details and time offset information Print the current version of Gam with details and time offset information
``` ```
gam version timeoffset gam version timeoffset
GAMADV-XTD3 6.68.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource GAMADV-XTD3 6.68.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
Ross Scroggs <ross.scroggs@gmail.com> Ross Scroggs <ross.scroggs@gmail.com>
Python 3.12.1 64-bit final Python 3.12.2 64-bit final
MacOS Sonoma 14.2.1 x86_64 MacOS Sonoma 14.2.1 x86_64
Path: /Users/Admin/bin/gamadv-xtd3 Path: /Users/Admin/bin/gamadv-xtd3
Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain.com Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain: domain.com
Your system time differs from www.googleapis.com by less than 1 second Your system time differs from www.googleapis.com by less than 1 second
``` ```
Print the current version of Gam with extended details and SSL information Print the current version of Gam with extended details and SSL information
``` ```
gam version extended gam version extended
GAMADV-XTD3 6.68.03 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource GAMADV-XTD3 6.68.04 - https://github.com/taers232c/GAMADV-XTD3 - pythonsource
Ross Scroggs <ross.scroggs@gmail.com> Ross Scroggs <ross.scroggs@gmail.com>
Python 3.12.1 64-bit final Python 3.12.2 64-bit final
MacOS Sonoma 14.2.1 x86_64 MacOS Sonoma 14.2.1 x86_64
Path: /Users/Admin/bin/gamadv-xtd3 Path: /Users/Admin/bin/gamadv-xtd3
Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain.com Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain: domain.com
Time: 2023-06-02T21:10:00-07:00 Time: 2023-06-02T21:10:00-07:00
Your system time differs from admin.googleapis.com by less than 1 second Your system time differs from admin.googleapis.com by less than 1 second
OpenSSL 3.1.1 30 May 2023 OpenSSL 3.1.1 30 May 2023
cryptography 41.0.1 cryptography 41.0.1
filelock 3.12.1 filelock 3.12.2
google-api-python-client 2.88.0 google-api-python-client 2.88.0
google-auth-httplib2 0.1.0 google-auth-httplib2 0.1.0
google-auth-oauthlib 1.0.0 google-auth-oauthlib 1.0.0
@ -64,7 +64,7 @@ MacOS High Sierra 10.13.6 x86_64
Path: /Users/Admin/bin/gamadv-xtd3 Path: /Users/Admin/bin/gamadv-xtd3
Version Check: Version Check:
Current: 5.35.08 Current: 5.35.08
Latest: 6.68.03 Latest: 6.68.04
echo $? echo $?
1 1
``` ```
@ -72,7 +72,7 @@ echo $?
Print the current version number without details Print the current version number without details
``` ```
gam version simple gam version simple
6.68.03 6.68.04
``` ```
In Linux/MacOS you can do: In Linux/MacOS you can do:
``` ```
@ -82,12 +82,12 @@ echo $VER
Print the current version of Gam and address of this Wiki Print the current version of Gam and address of this Wiki
``` ```
gam help gam help
GAM 6.68.03 - https://github.com/taers232c/GAMADV-XTD3 GAM 6.68.04 - https://github.com/taers232c/GAMADV-XTD3
Ross Scroggs <ross.scroggs@gmail.com> Ross Scroggs <ross.scroggs@gmail.com>
Python 3.12.1 64-bit final Python 3.12.2 64-bit final
MacOS Sonoma 14.2.1 x86_64 MacOS Sonoma 14.2.1 x86_64
Path: /Users/Admin/bin/gamadv-xtd3 Path: /Users/Admin/bin/gamadv-xtd3
Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain.com Config File: /Users/admin/GAMConfig/gam.cfg, Section: DEFAULT, customer_id: my_customer, domain: domain.com
Time: 2023-06-02T21:10:00-07:00 Time: 2023-06-02T21:10:00-07:00
Help: Syntax in file /Users/Admin/bin/gamadv-xtd3/GamCommands.txt Help: Syntax in file /Users/Admin/bin/gamadv-xtd3/GamCommands.txt
Help: Documentation is at https://github.com/taers232c/GAMADV-XTD3/wiki Help: Documentation is at https://github.com/taers232c/GAMADV-XTD3/wiki

4
src/GamCommands.txt
View File

@ -1288,13 +1288,13 @@ gam tbatch <BatchContent> [showcmds [<Boolean>]]
gam csv <CSVLoopContent> [warnifnodata] gam csv <CSVLoopContent> [warnifnodata]
[columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>] [columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>]
(matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]] (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]]
[maxrows <Integer>] [skiprows <Integer>] [maxrows <Integer>]
gam <GAMArgumentList> gam <GAMArgumentList>
gam loop <CSVLoopContent> [warnifnodata] gam loop <CSVLoopContent> [warnifnodata]
[columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>] [columndelimiter <Character>] [noescapechar <Boolean>] [quotechar <Character>] [fields <FieldNameList>]
(matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]] (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]]
[maxrows <Integer>] [skiprows <Integer>] [maxrows <Integer>]
gam <GAMArgumentList> gam <GAMArgumentList>
You can make substitutions in <GAMArgumentList> with values from the CSV file. You can make substitutions in <GAMArgumentList> with values from the CSV file.

6
src/GamUpdate.txt
View File

@ -2,6 +2,12 @@
Merged GAM-Team version Merged GAM-Team version
6.68.04
Added option `skiprows <Integer>` to `gam csv|loop` that causes GAM to skip processing the first `<Integer>` filtered rows.
* See: https://github.com/taers232c/GAMADV-XTD3/wiki/Bulk-Processing#csv-files
6.68.03 6.68.03
Fixed bug in `gam <UserTypeEntity> create drivefileacl` that caused a trap. Fixed bug in `gam <UserTypeEntity> create drivefileacl` that caused a trap.

33
src/gam/__init__.py
View File

@ -9859,6 +9859,12 @@ def _getShowCommands():
return getBoolean() return getBoolean()
return GC.Values[GC.SHOW_COMMANDS] return GC.Values[GC.SHOW_COMMANDS]
def _getSkipRows():
if checkArgumentPresent('skiprows'):
return getInteger(minVal=0)
# return GC.Values[GC.CSV_INPUT_ROW_SKIP]
return 0
def _getMaxRows(): def _getMaxRows():
if checkArgumentPresent('maxrows'): if checkArgumentPresent('maxrows'):
return getInteger(minVal=0) return getInteger(minVal=0)
@ -10041,7 +10047,7 @@ def processSubFields(GAM_argv, row, subFields):
# gam csv <CSVLoopContent> [warnifnodata] # gam csv <CSVLoopContent> [warnifnodata]
# [columndelimiter <Character>] [quotechar <Character>] [fields <FieldNameList>] # [columndelimiter <Character>] [quotechar <Character>] [fields <FieldNameList>]
# (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]] # (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]]
# [maxrows <Integer>] # [skiprows <Integer>] [maxrows <Integer>]
# gam <GAM argument list> # gam <GAM argument list>
def doCSV(testMode=False): def doCSV(testMode=False):
filename = getString(Cmd.OB_FILE_NAME) filename = getString(Cmd.OB_FILE_NAME)
@ -10051,6 +10057,7 @@ def doCSV(testMode=False):
f, csvFile, fieldnames = openCSVFileReader(filename) f, csvFile, fieldnames = openCSVFileReader(filename)
matchFields, skipFields = getMatchSkipFields(fieldnames) matchFields, skipFields = getMatchSkipFields(fieldnames)
showCmds = _getShowCommands() showCmds = _getShowCommands()
skipRows = _getSkipRows()
maxRows = _getMaxRows() maxRows = _getMaxRows()
checkArgumentPresent(Cmd.GAM_CMD, required=True) checkArgumentPresent(Cmd.GAM_CMD, required=True)
if not Cmd.ArgumentsRemaining(): if not Cmd.ArgumentsRemaining():
@ -10065,8 +10072,13 @@ def doCSV(testMode=False):
i = 0 i = 0
for row in csvFile: for row in csvFile:
if checkMatchSkipFields(row, fieldnames, matchFields, skipFields): if checkMatchSkipFields(row, fieldnames, matchFields, skipFields):
items.append(processSubFields(GAM_argv, row, subFields))
i += 1 i += 1
if skipRows:
if i <= skipRows:
continue
i = 1
skipRows = 0
items.append(processSubFields(GAM_argv, row, subFields))
if maxRows and i >= maxRows: if maxRows and i >= maxRows:
break break
closeFile(f) closeFile(f)
@ -10091,7 +10103,7 @@ def doCSVTest():
# gam loop <CSVLoopContent> [warnifnodata] # gam loop <CSVLoopContent> [warnifnodata]
# [columndelimiter <Character>] [quotechar <Character>] [fields <FieldNameList>] # [columndelimiter <Character>] [quotechar <Character>] [fields <FieldNameList>]
# (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]] # (matchfield|skipfield <FieldName> <RegularExpression>)* [showcmds [<Boolean>]]
# [maxrows <Integer>] # [skiprows <Integer>] [maxrows <Integer>]
# gam <GAM argument list> # gam <GAM argument list>
def doLoop(loopCmd): def doLoop(loopCmd):
filename = getString(Cmd.OB_FILE_NAME) filename = getString(Cmd.OB_FILE_NAME)
@ -10101,6 +10113,7 @@ def doLoop(loopCmd):
f, csvFile, fieldnames = openCSVFileReader(filename) f, csvFile, fieldnames = openCSVFileReader(filename)
matchFields, skipFields = getMatchSkipFields(fieldnames) matchFields, skipFields = getMatchSkipFields(fieldnames)
showCmds = _getShowCommands() showCmds = _getShowCommands()
skipRows = _getSkipRows()
maxRows = _getMaxRows() maxRows = _getMaxRows()
checkArgumentPresent(Cmd.GAM_CMD, required=True) checkArgumentPresent(Cmd.GAM_CMD, required=True)
if not Cmd.ArgumentsRemaining(): if not Cmd.ArgumentsRemaining():
@ -10136,9 +10149,14 @@ def doLoop(loopCmd):
i = 0 i = 0
for row in csvFile: for row in csvFile:
if checkMatchSkipFields(row, fieldnames, matchFields, skipFields): if checkMatchSkipFields(row, fieldnames, matchFields, skipFields):
i += 1
if skipRows:
if i <= skipRows:
continue
i = 1
skipRows = 0
item = processSubFields(GAM_argv, row, subFields) item = processSubFields(GAM_argv, row, subFields)
logCmd = Cmd.QuotedArgumentList(item) logCmd = Cmd.QuotedArgumentList(item)
i += 1
if i % 100 == 0: if i % 100 == 0:
batchWriteStderr(Msg.PROCESSING_ITEM_N.format(currentISOformatTimeStamp(), i)) batchWriteStderr(Msg.PROCESSING_ITEM_N.format(currentISOformatTimeStamp(), i))
sysRC = ProcessGAMCommand(item, processGamCfg=processGamCfg, inLoop=True) sysRC = ProcessGAMCommand(item, processGamCfg=processGamCfg, inLoop=True)
@ -10154,8 +10172,13 @@ def doLoop(loopCmd):
i = 0 i = 0
for row in csvFile: for row in csvFile:
if checkMatchSkipFields(row, fieldnames, matchFields, skipFields): if checkMatchSkipFields(row, fieldnames, matchFields, skipFields):
items.append(processSubFields(GAM_argv, row, subFields))
i += 1 i += 1
if skipRows:
if i <= skipRows:
continue
i = 1
skipRows = 0
items.append(processSubFields(GAM_argv, row, subFields))
if maxRows and i >= maxRows: if maxRows and i >= maxRows:
break break
closeFile(f) closeFile(f)