GAM 4.23

* Update calendar to allow access to user's secondary calendars

* Code cleanup

* Code cleanup

* Code cleanup

README.md

@@ -1,22 +1,19 @@
-GAM
-============================
 GAM is a command line tool for Google G Suite Administrators to manage domain and user settings quickly and easily.
-
-Downloads
----------
-You can download the current GAM release from the [GitHub Releases] page.
-
-Documentation
-------------------
+# Quick Start
+## Linux / MacOS
+Open a terminal and run:
+```
+bash <(curl -s -S -L https://git.io/install-gam)
+```
+this will download GAM, install it and start setup.
+## Windows
+Download the MSI Installer from the [GitHub Releases] page. Install the MSI and you'll be prompted to setup GAM.
+# Documentation
 The GAM documentation is hosted in the [GitHub Wiki]
-
-Mailing List / Discussion group
--------------------------------
+# Mailing List / Discussion group
 The GAM mailing list / discussion group is hosted on [Google Groups].  You can join the list and interact via email, or just post from the web itself.
-
-Author
-------
-GAM is maintained by <a href="mailto:jay0lee@gmail.com">Jay Lee</a>.
+# Author
+GAM is maintained by <a href="mailto:jay0lee@gmail.com">Jay Lee</a>. Please direct "how do I?" questions to [Google Groups].
 
 [GAM release]: https://git.io/gamreleases
 [GitHub Releases]: https://github.com/jay0lee/GAM/releases

src/GamCommands.txt

@@ -1,6 +1,8 @@
 This document describes the GAM command line syntax in  modified BNF, see https://en.wikipedia.org/wiki/Backus-Naur_Form
+Skip the History section and start reading at Introduction.
+
 Items on the command line are space separated, when an actual space character is required, it will be indicated by <Space>.
-If an item contains spaces, it should be surrounded by " or '.
+If an item contains spaces, it should be surrounded by ".
 
 [] optional item
 () group items
@@ -8,24 +10,47 @@ If an item contains spaces, it should be surrounded by " or '.
 + item may appear one or more times
 | separates alternative items
 
-# Primatives
+Primitives
 <Digit> ::= 0|1|2|3|4|5|6|7|8|9
 <Number> ::= <Digit>+
 <Hex> ::= <Digit>|a|b|c|d|e|f|A|B|C|D|E|F
 <Space> ::= an actual space character
-<String> ::= a string of characters, surrounded by " or ' if it contains spaces
+<String> ::= a string of characters, surrounded by " if it contains spaces
 <TrueValues> ::= true|on|yes|enabled|1
 <FalseValues>= false|off|no|disabled|0
 <DataTransferService> ::= googledrive|gdrive|drive|"drive and docs"
-<ProductID> ::= Google-Apps|Google-Coordinate|Google-Drive-storage|Google-Vault
-<SKUID> ::= apps|gafb|gafw|gams|gau|unlimited|d4w|dfw|coordinate|vault|vfe|
-	    drive-20gb|drive20gb|20gb|drive-50gb|drive50gb|50gb|drive-200gb|drive200gb|200gb|drive-400gb|drive400gb|400gb|
-	    drive-1tb|drive1tb|1tb|drive-2tb|drive2tb|2tb|drive-4tb|drive4tb|4tb|drive-8tb|drive8tb|8tb|drive-16tb|drive16tb|16tb
+<ProductID> ::=
+	Google-Apps|
+	Google-Chrome-Device-Management|
+	Google-Coordinate|
+	Google-Drive-storage|
+	Google-Vault
+<SKUID> ::=
+	free|standard|Google-Apps|
+	gafb|gafw|basic|gsuitebasic|Google-Apps-For-Business|
+	gafg|gsuitegovernment|gsuitegov|Google-Apps-For-Government|
+	gams|postini|gsuitegams|gsuitepostini|gsuitemessagesecurity|Google-Apps-For-Postini|
+	gal|lite|gsuitelite|Google-Apps-Lite|
+	gau|unlimited|gsuitebusiness|Google-Apps-Unlimited|
+	gae|enterprise|gsuiteenterprise|1010020020|
+	chrome|cdm|googlechromedevicemanagement|Google-Chrome-Device-Management|
+	coordinate|googlecoordinate|Google-Coordinate|
+	drive20gb|20gb|googledrivestorage20gb|Google-Drive-storage-20GB|
+	drive50gb|50gb|googledrivestorage50gb|Google-Drive-storage-50GB|
+	drive200gb|200gb|googledrivestorage200gb|Google-Drive-storage-200GB|
+	drive400gb|400gb|googledrivestorage400gb|Google-Drive-storage-400GB|
+	drive1tb|1tb|googledrivestorage1tb|Google-Drive-storage-1TB|
+	drive2tb|2tb|googledrivestorage2tb|Google-Drive-storage-2TB|
+	drive4tb|4tb|googledrivestorage4tb|Google-Drive-storage-4TB|
+	drive8tb|8tb|googledrivestorage8tb|Google-Drive-storage-8TB|
+	drive16tb|16tb|googledrivestorage16tb|Google-Drive-storage-16TB|
+	vault|googlevault|Google-Vault|
+	vfe|googlevaultformeremployee|Google-Vault-Former-Employee
 <Charset> ::= ascii|mbcs|utf-8|utf-8-sig|utf-16|<String>
 <FileFormat> ::= csv|html|txt|tsv|jpeg|jpg|png|svg|pdf|rtf|pptx|xlsx|docx|odt|ods|openoffice|ms|microsoft|micro$oft
 <Language> ::= ar|bn|bg|ca|zh-CN|zh-TW|hr|cs|da|nl|en|en-GB|et|fi|fr|de|el|gu|iw|is|in|it|ja|kn|ko|lv|lt|ms|ml|mr|no|or|fa|pl|pt-BR|pt-PT|ro|ru|sr|sk|sl|es|sv|tl|ta|te|th|tr|uk|ur|vi
 
-# Basic items built from primatives
+Basic items built from primitives
 <Boolean> ::= <TrueValues>|<FalseValues>
 <ByteCount> ::= <Number>[m|k|b]
 <CIDRnetmask> ::= <Number>.<Number>.<Number>.<Number>/<Number>
@@ -46,7 +71,7 @@ If an item contains spaces, it should be surrounded by " or '.
 <Tag> ::= <String>
 <UniqueID> ::= uid:<String>
 
-# Named items
+Named items
 <AccessToken> ::= <String>
 <ACLScope> ::= [user:]<EmailAddress>|group:<EmailAddress>|domain[:<DomainName>]|default
 <CalendarACLRole> ::= editor|freebusy|freebusyreader|owner|reader|writer
@@ -57,10 +82,10 @@ If an item contains spaces, it should be surrounded by " or '.
 <CourseID> ::= <Number>|d:<CourseAlias>
 <CourseParticipantType> ::= teacher|teachers|student|students
 <CrOSID> ::= <String>
-<CrOSItem> ::= <CrOSID>|(query:<QueryCrOS>)|(query <QueryCrOS>)
+<CrOSItem> ::= <CrOSID>|(query:<QueryCrOS>)
 <DestEmailAddress> ::= <EmailAddress>
 <DomainAlias> ::= <String>
-<DriveFileACLRole> :: =commenter|editor|owner|reader|writer
+<DriveFileACLRole> :: =commenter|editor|organizer|owner|reader|writer
 <DriveFileID> ::= <String>
 <DriveFileURL> :: = https://docs.google.com/a/<DomainName>/document/d/<DriveFileID>/<String>
 <DriveFileItem> ::= <DriveFileID>|<DriveFileURL>
@@ -99,9 +124,10 @@ If an item contains spaces, it should be surrounded by " or '.
 <PrintJobStatus> ::= done|error|held|in_progress|queued|submitted
 <PropertyKey> ::= <String>
 <PropertyValue> ::= <String>
+<QueryCalendar> ::= <String>
 <QueryContact> ::= <String> See: https://developers.google.com/google-apps/contacts/v3/reference#contacts-query-parameters-reference
 <QueryCrOS> ::= <String> See: https://support.google.com/chrome/a/answer/1698333?hl=en
-<QueryDriveFile> :: = <String> See: https://developers.google.com/drive/v2/web/search-parameters 
+<QueryDriveFile> :: = <String> See: https://developers.google.com/drive/v2/web/search-parameters
 <QueryGmail> ::= <String> See: https://support.google.com/mail/answer/7190
 <QueryMobile> ::= <String> See: https://support.google.com/a/answer/1408863?hl=en#search
 <QueryPrinter> ::= <String> See: https://developers.google.com/cloud-print/docs/appInterfaces#search
@@ -113,13 +139,34 @@ If an item contains spaces, it should be surrounded by " or '.
 <RoleAssignmentID> ::= <String>
 <SchemaName> ::= <String>
 <Section> ::= <String>
+<S/MIMEID> ::= <String>	
 <StudentItem> ::= <EmailAddress>|<UniqueID>|<String>
+<TeamDriveID> ::= <String>
 <Timezone> ::= <String>
 <Title> ::= <String>
 <URI> ::= <String>
 <URL> ::= <String>
 <UserItem> ::= <EmailAddress>|<UniqueID>|<String>
 
+<CourseFieldName> ::=
+	alternatelink|
+	coursegroupemail|
+	coursematerialsets|
+	coursestate|
+	creationtime|
+	description|
+	descriptionheading|
+	enrollmentcode|
+	guardiansenabled|
+	id|
+	name|
+	ownerid|
+	room|
+	section|
+	teacherfolder|
+	teachergroupemail|
+	updatetime
+<CourseFieldNameList> ::= "<CourseFieldName>(,<CourseFieldName>)*"
 
 <CrOSFieldName> ::=
 	activetimeranges|timeranges|
@@ -223,6 +270,7 @@ If an item contains spaces, it should be surrounded by " or '.
 	customreplyto|
 	defaultmessagedenynotificationtext|
 	description|
+	directmemberscount|
 	email|
 	id|
 	includeinglobaladdresslist|gal|
@@ -281,6 +329,8 @@ If an item contains spaces, it should be surrounded by " or '.
 	ipwhitelisted|
 	isdelegatedadmin|admin|isadmin|
 	ismailboxsetup|
+	is2svenforced|
+	is2svenrolled|
 	lastlogintime|
 	noneditablealiases|aliases|nicknames|
 	notes|note|
@@ -296,45 +346,45 @@ If an item contains spaces, it should be surrounded by " or '.
 <UserOrderByFieldName> ::=
 	familyname|lastname|givenname|firstname|email
 
-# Named Lists
-# Lists can be in the following formats
-# Items, separated by commas, without spaces or commas in the items themselves: item(,item)*
-# Items, separated by spaces, without spaces or commas in the items themselves: "item( item)*"
-# Items, separated by commas, with spaces or commas in the items themselves: "'it em'(,'it em')*"
-# Items, separated by spaces, with spaces or commas in the items themselves: "'it em'( 'it em')*"
+Named Lists
+Lists can be in the following formats
+Items, separated by commas, without spaces or commas in the items themselves: item(,item)*
+Items, separated by spaces, without spaces or commas in the items themselves: "item( item)*"
+Items, separated by commas, with spaces or commas in the items themselves: "'it em'(,'it,em')*"
+Items, separated by spaces, with spaces or commas in the items themselves: "'it em'( 'it,em')*"
 
-<ACLList> ::== '<ACLScope>(,<ACLScope>)*'
-<CalendarList> ::= '<CalendarItem>(,<CalendarItem>)*'
-<CourseAliasList> ::= '<CourseAlias>(,<CourseAlias>)*'
-<CourseIDList> ::= '<CourseID>(,<CourseID>)*'
-<CrOSFieldNameList> ::= '<CrOSFieldName>(,<CrOSFieldName>)*'
-<CrOSList> ::= '<CrOSID>(,<CrOSID>)*'
-<DriveFileList> ::= '<DriveFileItem>(,<DriveFileItem>)*'
-<EmailAddressList> ::= '<EmailAddress>(,<EmailAddress>)*'
-<EventIDList> ::= '<EventID>(,<EventID>)*'
-<FileFormatList> ::= '<FileFormat>(,<FileFormat)*'
-<FilterIDList> ::= '<FilterID>(,<FilterID>)*'
-<GroupFieldNameList> ::= '<GroupFieldName>(,<GroupFieldName>)*'
-<GroupList> ::= '<GroupItem>(,<GroupItem>)*'
-<GuardianStateList> ::= '<GuardianState>(,<GuardianState>)*'
-<LabelNameList> ::= '<LabelName>(,<LabelName)*'
-<MembersFieldNameList> ::= '<MembersFieldName>(,<MembersFieldName>)*'
-<MobileList> ::= '<MobileId>(,<MobileId>)*'
-<OrgUnitList> ::== '<OrgUnitPath>(,<OrgUnitPath>)*'
-<PrinterIDList> ::= '<PrinterID>(,<PrinterID>)*'
-<ProductIDList> ::= '(<ProductID>|SKUID>)(,<ProductID>|SKUID>)*'
-<PrintJobIDList> ::= '<PrintJobID>(,<PrintJobID>)*'
-<ResourceIDList> ::= '<ResourceID>(,<ResourceID>)*'
-<SKUIDList> ='<SKUID>(,<SKUID>)*'
-<SchemaNameList> ::= '<SchemaName>(,<SchemaName>)*'
-<UserFieldNameList> ::= '<UserFieldName>(,<UserFieldName>)*'
-<UserList> ::= '<UserItem>(,<UserItem>)*'
+<ACLList> ::== "<ACLScope>(,<ACLScope>)*"
+<CalendarList> ::= "<CalendarItem>(,<CalendarItem>)*"
+<CourseAliasList> ::= "<CourseAlias>(,<CourseAlias>)*"
+<CourseIDList> ::= "<CourseID>(,<CourseID>)*"
+<CrOSFieldNameList> ::= "<CrOSFieldName>(,<CrOSFieldName>)*"
+<CrOSList> ::= "<CrOSID>(,<CrOSID>)*"
+<DriveFileList> ::= "<DriveFileItem>(,<DriveFileItem>)*"
+<EmailAddressList> ::= "<EmailAddress>(,<EmailAddress>)*"
+<EventIDList> ::= "<EventID>(,<EventID>)*"
+<FileFormatList> ::= "<FileFormat>(,<FileFormat)*"
+<FilterIDList> ::= "<FilterID>(,<FilterID>)*"
+<GroupFieldNameList> ::= "<GroupFieldName>(,<GroupFieldName>)*"
+<GroupList> ::= "<GroupItem>(,<GroupItem>)*"
+<GuardianStateList> ::= "<GuardianState>(,<GuardianState>)*"
+<LabelNameList> ::= "<LabelName>(,<LabelName)*"
+<MembersFieldNameList> ::= "<MembersFieldName>(,<MembersFieldName>)*"
+<MobileList> ::= "<MobileId>(,<MobileId>)*"
+<OrgUnitList> ::== "<OrgUnitPath>(,<OrgUnitPath>)*"
+<PrinterIDList> ::= "<PrinterID>(,<PrinterID>)*"
+<ProductIDList> ::= "(<ProductID>|SKUID>)(,<ProductID>|SKUID>)*"
+<PrintJobIDList> ::= "<PrintJobID>(,<PrintJobID>)*"
+<ResourceIDList> ::= "<ResourceID>(,<ResourceID>)*"
+<SKUIDList> ="<SKUID>(,<SKUID>)*"
+<SchemaNameList> ::= "<SchemaName>(,<SchemaName>)*"
+<UserFieldNameList> ::= "<UserFieldName>(,<UserFieldName>)*"
+<UserList> ::= "<UserItem>(,<UserItem>)*"
 
-# Specify a collection of ChromeOS devices by directly specifying them
+Specify a collection of ChromeOS devices by directly specifying them
 <CrOSTypeEntity> ::=
 		(all cros)|
 		(cros <CrOSList>)|
-# Specify a collection of Users by directly specifying them or by specifiying items that will yield a list of users
+Specify a collection of Users by directly specifying them or by specifiying items that will yield a list of users
 <UserTypeEntity> ::=
 		(all users)|
 		(user <UserItem>)|
@@ -351,7 +401,7 @@ If an item contains spaces, it should be surrounded by " or '.
 		(query <QueryUser>)
 
 
-# Item attributes
+Item attributes
 <CalendarAttributes> ::=
 	(selected <Boolean>)|(hidden <Boolean>)|(summary <String>)|(colorindex|colorid <CalendarColorIndex>)|(backgroundcolor <ColorHex>)|(foregroundcolor <ColorHex>)|
 	(reminder clear|(email|sms|pop <Number>))|
@@ -430,7 +480,7 @@ If an item contains spaces, it should be surrounded by " or '.
 	(defaultdisplayname <String>)|(description <String>)|(displayname <String>)|(firmware <String>)|(gcpversion <String>)|
 	(istosaccepted <Boolean>)|(manufacturer <String>)|(model <String>)|(name <String>)|(ownerid <EmailAddress>)|(proxy <String>)|(public <Boolean>)|
 	(quotaenabled <Boolean>)|(status <Number>)|(type <String>)|(uuid <String>)|
-	(setupurl <URL>)|(supporturl <URL>)|(updateurl <URL>) 
+	(setupurl <URL>)|(supporturl <URL>)|(updateurl <URL>)
 
 <SchemaFieldDefinition> ::=
 	field <FieldName> (type bool|date|double|email|int64|phone|string) [multivalued|multivalue] [indexed] [restricted] [range <Number> <Number>] endfield
@@ -451,7 +501,7 @@ If an item contains spaces, it should be surrounded by " or '.
 	(im|ims clear|(type work|home|other|(custom <String>) protocol aim|gtalk|icq|jabber|msn|net_meeting|qq|skype|yahoo|(custom_protocol <String>) <String> [notprimary|primary]))|
 	(ipwhitelisted <Boolean>)|
 	(lastname|familyname <String>)|
-	(note|notes clear|([text_plain|text_html] <String>|(file <FileName>)))|
+	(note|notes clear|([text_plain|text_html] <String>|(file <FileName> [charset <CharSet>])))|
 	(organization|organizations clear|([type domain_only|school|unknown|work] [customtype <String>] [name <String>] [title <String>] [department <String>] [symbol <String>]
 		[costcenter <String>]  [location <String>] [description <String>] [domain <String>] notprimary|primary))|
 	(org|ou|orgunitpath <OrgUnitPath>)
@@ -461,27 +511,52 @@ If an item contains spaces, it should be surrounded by " or '.
 	(relation|relations clear|(spouse|child|mother|father|parent|brother|sister|friend|relative|domestic_partner|manager|assistant|referred_by|partner|<String> <String>))|
 	(suspended <Boolean>)|
 	(website|websites clear|(home_page|blog|profile|work|home|other|ftp|reservations|app_install_page|<String> <URL> [notprimary|primary]))|
-	(<SchemaName>.<FieldName> [multivalued|multivalue|value [type work|home|other|(custom <String>)]] <String>)
+	(<SchemaName>.<FieldName> [multivalued|multivalue|value|multinonempty [type work|home|other|(custom <String>)]] <String>)
 
-gam version [check]
+gam version [check] [simple]
 gam help
 
 gam batch <FileName>|- [charset <Charset>]
 gam csv <FileName>|- [charset <Charset>] gam <GAM argument list>
 
-# You can make substitutions in <GAMArgumentList> with values from the CSV file.
-# An argument containing exactly ~xxx is replaced by the value of field xxx from the CSV file
-# An argument containing instances of ~~xxx~~ has xxx replaced by the value of field xxx from the CSV file
+You can make substitutions in <GAMArgumentList> with values from the CSV file.
+An argument containing exactly ~xxx is replaced by the value of field xxx from the CSV file
+An argument containing instances of ~~xxx~~ has xxx replaced by the value of field xxx from the CSV file
 
-# Example: gam csv Users.csv gam update user '~primaryEmail' address type work unstructured '~~Street~~, ~~City~~, ~~State~~ ~~ZIP~~'
-# Each user (~primaryEmail, e.g. foo@bar.com) would have their work address updated
+Example: gam csv Users.csv gam update user "~primaryEmail" address type work unstructured "~~Street~~, ~~City~~, ~~State~~ ~~ZIP~~"
+Each user (~primaryEmail, e.g. foo@bar.com) would have their work address updated
 
-gam oauth|oauth2 create|request
+gam create project [<EmailAddress>]
+
+gam oauth|oauth2 create|request [<EmailAddress>]
 gam oauth|oauth2 delete|revoke
 gam oauth|oauth2 info|verify [<AccessToken>]
 
+gam <UserTypeEntity> check serviceaccount
+
 gam whatis <EmailItem>
 
+gam create resoldcustomer <CustomerDomain> (customer_auth_token <String>)
+	(email|alternate_email <EmailAddress>) (name|organization_name <String>) (contact|contact_name <String>) [phone|phone_number <String>]
+	[address|address1 <String>] [address2 <String>] [address3 <String>]
+	[locality|city <String>] [region|state <String>] [postal|postal_code <String>] [country|country_code <String>]
+gam update resoldcustomer <CustomerID> [customer_auth_token <String>]
+	[email|alternate_email <EmailAddress>] [name|organization_name <String>] [contact|contact_name <String>] [phone|phone_number <String>]
+	[address|address1 <String>] [address2 <String>] [address3 <String>]
+	[locality|city <String>] [region|state <String>] [postal|postal_code <String>] [country|country_code <String>]
+gam info resoldcustomer <CustomerID>
+
+gam create resoldsubscription <CustomerID> (sku <SKUID>)
+	 (plan annual_monthly_pay|annual_yearly_pay|flexible|trial) (seats <NumberOfSeats> <MaximumNumberOfSeats>)
+	 [customer_auth_token <String>] [deal <String>] [purchaseorderid <String>]
+gam update resoldsubscription <CustomerID> <SKUID>
+	activate|suspend|startpaidservice|
+	(renewal auto_renew_monthly_pay|auto_renew_yearly_pay|cancel|renew_current_users_monthly_pay|renew_current_users_yearly_pay|switch_to_pay_as_you_go)|
+	(seats <NumberOfSeats> [<MaximumNumberOfSeats>])|
+	(plan annual_monthly_pay|annual_yearly_pay|flexible|trial [deal <String>] [purchaseorderid <String>] [seats <NumberOfSeats> [<MaximumNumberOfSeats>]])
+gam delete resoldsubscription <CustomerID> <SKUID> cancel|downgrade|transfer_to_direct
+gam info resoldsubscriptions <CustomerID> [customer_auth_token <String>]
+
 gam report users|user [todrive]
 	[date <Date>] [(user all|<UserItem>)] [filter|filters <String>] [fields|parameters <String>]
 gam report customers|customer|domain [todrive]
@@ -505,7 +580,6 @@ gam delete domainalias|aliasdomain <DomainAlias>
 gam info domainalias|aliasdomain <DomainAlias>
 gam print domainaliases|aliasdomains [todrive]
 
-
 gam info customer
 gam update customer [adminsecondaryemail|alternateemail <EmailAddress>] [language <LanguageCode] [phone|phonenumber <String>]
 	[contact|contactname <String>] [name|organizationname <String>]
@@ -537,14 +611,15 @@ gam calendar <CalendarItem> del|delete <CalendarACLRole> <EmailAddress>|(domain
 gam calendar <CalendarItem> showacl
 
 gam calendar <CalendarItem> addevent <EventAttributes>+
+gam calendar <CalendarItem> deleteevent (id|eventid <EventID>)* (query|eventquery <QueryCalendar>)* [doit] [notifyattendees]
 gam calendar <CalendarItem> wipe
 
 gam update cros <CrOSItem> (<CrOSAttributes>+)|(action deprovision_same_model_replace|deprovision_different_model_replace|deprovision_retiring_device|disable|reenable [acknowledge_device_touch_requirement])
-gam info cros <CrOSItem> [nolists] [listlimit <Number>]
+gam info cros <CrOSItem> [nolists] [listlimit <Number>] [start <Date>] [end <Date>]
 	[basic|full|allfields] <CrOSFieldName>* [fields <CrOSFieldNameList>]
 
-gam print cros [todrive] [query <QueryCrOS>]
-	[orderby <CrOSOrderByFieldName> [ascending|descending]] [nolists] [listlimit <Number>]
+gam print cros [todrive] [query <QueryCrOS>] [limittoou <OrgUnitItem>]
+	[orderby <CrOSOrderByFieldName> [ascending|descending]] [nolists] [listlimit <Number>] [start <Date>] [end <Date>]
 	[basic|full|allfields] <CrOSFieldName>* [fields <CrOSFieldNameList>]
 gam <CrOSTypeEntity> print
 
@@ -558,11 +633,27 @@ Prints a header row and selected fields for specified CrOS devices.
 
 The basic argument yields these column headers: deviceId,annotatedAssetId,annotatedLocation,annotatedUser,lastSync,notes,serialNumber,status
 The full argument yields all column headers including two headers, recentUsers and activeTimeRanges,
-that repeat with two subvalues each, yielding a large number of columns that make the output hard to process.
+that repeat with two/four subvalues each, yielding a large number of columns that make the output hard to process.
 The nolists argument suppresses these two headers; if you want these headers in a more manageable form use the following arguments.
-The listlimit <Number> argument limits the number of repetitions to <Number>. If <Number> equals zero, there is no limit.
-If recentusers is specified as a field, each pair of values for recentUsers is put on a separate row with all of the other headers.
-If timeranges is specified as a field, each  pair of values for activeTimeRanges is put on a separate row with all of the other headers.
+If recentusers is specified, for each recent user, the columns recentUsers.email and recentUsers.type are output on a separate row
+with all of the other headers.
+If timeranges is specified, for each time range entry, the columns activeTimeRanges.date, activeTimeRange.activeTime,
+activeTimeRanges.duration and activeTimeRanges.minutes are output on a separate row with all of the other headers.
+The listlimit <Number> argument limits the number of repetitions to <Number>; if not specified or <Number> equals zero, there is no limit.
+The start <Date> and end <Date> arguments filter the time ranges.
+
+gam print crosactivity [todrive] [query <QueryCrOS>] [limittoou <OrgUnitItem>]
+	[recentusers] [timeranges] [both] [listlimit <Number>] [start <Date>] [end <Date>] [delimiter <String>]
+
+The basic column headers are: deviceId,annotatedAssetId,annotatedLocation,serialNumber,orgUnitPath.
+If recentusers is specified, all of the recent users email addresses, separated by the delimiter <String>,
+with header recentUsers.email, are output with the basic headers.
+If timeranges is specified, for each time range entry, activeTimeRanges.date and activeTimeRanges.duration and activeTimeRanges.minutes,
+are output on a separate row with the basic headers.
+The default is to include both recentusers and timeranges.
+The listlimit <Number> argument limits the number of recent users and time ranges to <Number>; if not specified or <Number> equals zero, there is no limit.
+The start <Date> and end <Date> arguments filter the time ranges.
+Delimiter defaults to comma.
 
 gam update mobile <MobileItem> <MobileAttributes>+
 gam delete mobile <MobileItem>
@@ -580,8 +671,8 @@ gam info group <GroupItem> [nousers] [noaliases] [groups]
 gam update group <GroupItem> clear [member] [manager] [owner]
 
 gam print groups [todrive] ([domain <DomainName>] [member <UserItem>])
-	[maxresults <Number>] [delimiter <String>]
-	[members] [managers] [owners] [settings] <GroupFieldName>* [fields <GroupFieldNameList>]
+	[maxresults <Number>] [allfields|([settings] <GroupFieldName>* [fields <GroupFieldNameList>])] [delimiter <String>]
+	[members|memberscount] [managers|managerscount] [owners|ownerscount]
 
 gam print group-members|groups-members [todrive] ([domain <DomainName>] [member <UserItem>])|[group <GroupItem>]
 	[membernames] [fields <MembersFieldNameList>]
@@ -605,7 +696,7 @@ gam info schema <SchemaName>
 gam show schema|schemas
 gam print schema|schemas
 
-gam create user <EmailAddress> <UserAttrubutes>*
+gam create user <EmailAddress> <UserAttributes>*
 gam update user <UserItem> <UserAttributes>* [clearschema <SchemaName>] [clearschema <SchemaName>.<FieldName>]
 gam delete user <UserItem>
 gam undelete user <UserItem> [org|ou <OrgUnitPath>]
@@ -614,7 +705,7 @@ gam info user [<UserItem>] [noaliases] [nogroups] [nolicenses|nolicences] [nosch
 gam print users [todrive] ([domain <DomainName>] [query <QueryUser>] [deleted_only|only_deleted])
 	[groups] [license|licenses|licence|licences] [emailpart|emailparts|username]
 	[orderby <UserOrderByFieldName> [ascending|descending]] [userview]
-	[basic|full|allfields | <UserFieldName>* | fields <UserFieldNameList>] [schemas|custom all|<SchemaNameList>]
+	[allfields|basic|full | ((<UserFieldName>* | fields <UserFieldNameList>) [schemas|custom all|<SchemaNameList>])]
 gam <UserTypeEntity> print
 
 Summary of printing:
@@ -631,7 +722,8 @@ gam create course id|alias <CourseAlias> [teacher <UserItem>] <CourseAttributes>
 gam update course <CourseID> <CourseAttributes>+
 gam delete course <CourseID>
 gam info course <CourseID>
-gam print courses [todrive] [teacher] [student] [alias|aliases] [delimiter <String>]
+gam print courses [todrive] [teacher <UserItem>] [student <UserItem>] [alias|aliases] [delimiter <String>]
+	[show all|students|teachers] [countsonly] [fields <CourseFieldNameList>] [skipfields <CourseFieldNameList>]
 
 gam course <CourseID> add alias <CourseAlias>
 gam course <CourseID> delete alias <CourseAlias>
@@ -681,7 +773,7 @@ gam <UserTypeEntity> delete|del backupcodes|backupcode|verificationcodes
 gam <UserTypeEntity> show backupcodes|backupcode|verificationcodes
 
 gam <UserTypeEntity> add calendar <CalendarItem> <CalendarAttributes>*
-gam <UserTypeEntity> update calendar <CalendarItem> <CalendarAttributes>+
+gam <UserTypeEntity> update calendar <CalendarItem>|primary <CalendarAttributes>+
 gam <UserTypeEntity> delete|del calendar <CalendarItem>
 gam <UserTypeEntity> show calendars
 gam <UserTypeEntity> info calendar <CalendarItem>|primary
@@ -709,9 +801,9 @@ gam <UserTypeEntity> delete|del emptydrivefolders
 gam <UserTypeEntity> empty drivetrash
 
 gam <UserTypeEntity> add drivefileacl <DriveFileID> anyone|(user <UserItem>)|(group <GroupItem>)|(domain <DomainName>)
-	(role <DriveFileACLRole>) [withlink] [sendmail] [emailmessage <String>]
+	(role <DriveFileACLRole>) [withlink|discoverable] [sendemail] [emailmessage <String>]
 gam <UserTypeEntity> update drivefileacl <DriveFileID> <PermissionID>
-	(role <DriveFileACLRole>) [withlink] [transferownership <Boolean>]
+	(role <DriveFileACLRole>) [withlink|discoverable] [removeexpiration]
 gam <UserTypeEntity> delete|del drivefileacl <DriveFileID> <PermissionID>
 gam <UserTypeEntity> show drivefileacl <DriveFileID>
 
@@ -719,9 +811,9 @@ gam <UserTypeEntity> delete|del alias|aliases
 
 gam <UserTypeEntity> delete|del group|groups
 
-gam <UserTypeEntity> add license <SKUID>
-gam <UserTypeEntity> update license <SKUID> [from] <SKUID>
-gam <UserTypeEntity> delete|del license <SKUID>
+gam <UserTypeEntity> add license <SKUID> [product|productid <ProductID>]
+gam <UserTypeEntity> update license <SKUID> [product|productid <ProductID>] [from] <SKUID>
+gam <UserTypeEntity> delete|del license <SKUID> [product|productid <ProductID>]
 
 gam <UserTypeEntity> update photo <FileNamePattern>
 gam <UserTypeEntity> delete|del photo
@@ -735,12 +827,10 @@ gam <UserTypeEntity> show tokens|token [clientid <ClientID>]
 gam <UserTypeEntity> print tokens|token [todrive] [clientid <ClientID>]
 gam print tokens|token [todrive] [clientid <ClientID>] [<UserTypeEntity>]
 
-gam <UserTypeEntity> update user <UserAttrubutes>
+gam <UserTypeEntity> update user <UserAttributes>
 
 gam <UserTypeEntity> deprovision|deprov
-#
-# Update user Gmail mailbox
-#
+
 gam <UserTypeEntity> [add] label|labels <Name> [messagelistvisibility hide|show] [labellistvisibility hide|show|showifunread]
 gam <UserTypeEntity> update labelsettings <LabelName> [name <Name>] [messagelistvisibility hide|show] [labellistvisibility hide|show|showifunread]
 gam <UserTypeEntity> update label|labels [search <RegularExpression>] [replace <LabelReplacement>] [merge]
@@ -751,9 +841,7 @@ gam <UserTypeEntity> delete messages query <QueryGmail> [doit] [max_to_delete|ma
 gam <UserTypeEntity> modify messages query <QueryGmail> [doit] [max_to_modify|max_to_process <Number>] (addlabel <LabelName>)* (removelabel <LabelName>)*
 gam <UserTypeEntity> trash messages query <QueryGmail> [doit] [max_to_trash|max_to_process <Number>]
 gam <UserTypeEntity> untrash messages query <QueryGmail> [doit] [max_to_untrash|max_to_process <Number>]
-#
-# Update user Gmail settings
-#
+
 gam <UserTypeEntity> show gmailprofile [todrive]
 gam <UserTypeEntity> show gplusprofile [todrive]
 
@@ -787,16 +875,28 @@ gam <UserTypeEntity> show imap|imap4
 gam <UserTypeEntity> pop|pop3 <Boolean> [for allmail|newmail|mailfromnowon|fromnowown] [action keep|leaveininbox|archive|delete|trash|markread]
 gam <UserTypeEntity> show pop|pop3
 
-gam <UserTypeEntity> [add] sendas <EmailAddress> <Name> [signature|sig <String>|(file <FileName> [charset <CharSet>]) (replace <Tag> <String>)*] [replyto <EmailAddress>] [default] [treatasalias <Boolean>]
-gam <UserTypeEntity> update sendas <EmailAddress> [name <Name>] [signature|sig <String>|(file <FileName> [charset <CharSet>]) (replace <Tag> <String>)*] [replyto <EmailAddress>] [default] [treatasalias <Boolean>]
+gam <UserTypeEntity> [add] sendas <EmailAddress> <Name> [signature|sig <String>|(file <FileName> [charset <CharSet>]) (replace <Tag> <String>)*] [html] [replyto <EmailAddress>] [default] [treatasalias <Boolean>]
+gam <UserTypeEntity> update sendas <EmailAddress> [name <Name>] [signature|sig <String>|(file <FileName> [charset <CharSet>]) (replace <Tag> <String>)*] [html] [replyto <EmailAddress>] [default] [treatasalias <Boolean>]
 gam <UserTypeEntity> delete sendas <EmailAddress>
 gam <UserTypeEntity> show sendas [format]
 gam <UserTypeEntity> info sendas <EmailAddress> [format]
 gam <UserTypeEntity> print sendas [todrive]
 
-gam <UserTypeEntity> signature|sig <String>|(file <FileName> [charset <Charset>]) (replace <Tag> <String>)* [name <String>] [replyto <EmailAddress>] [default] [treatasalias <Boolean>]
+gam <UserTypeEntity> add smime file <FileName> [password <Password>] [sendas|sendasemail <EmailAddress>] [default]
+gam <UserTypeEntity> update smime [id <S/MIMEID>] [sendas|sendasemail <EmailAddress>] [default]
+gam <UserTypeEntity> delete smime [id <S/MIMEID>] [sendas|sendasemail <EmailAddress>]
+gam <UserTypeEntity> show smime [primaryonly]
+gam <UserTypeEntity> print smime [todrive] [primaryonly]
+
+gam <UserTypeEntity> signature|sig <String>|(file <FileName> [charset <Charset>]) (replace <Tag> <String>)* [html] [name <String>] [replyto <EmailAddress>] [default] [treatasalias <Boolean>]
 gam <UserTypeEntity> show signature|sig [format]
 
+gam <UserTypeEntity> add teamdrive <Name>
+gam <UserTypeEntity> update teamdrive <TeamDriveID> [name <Name>]
+gam <UserTypeEntity> delete teamdrive <TeamDriveID>
+gam <UserTypeEntity> show teamdrives
+gam <UserTypeEntity> print teamdrives [todrive]
+
 gam <UserTypeEntity> vacation <FalseValues>
 gam <UserTypeEntity> vacation <TrueValues> subject <String> (message <String>)|(file <FileName> [charset <CharSet>]) (replace <Tag> <String>)* [html]
 	[contactsonly] [domainonly] [startdate <Date>] [enddate <Date>]

src/email-audit-v1.json

@@ -1,34 +0,0 @@
-{
- "kind": "discovery#restDescription",
- "discoveryVersion": "v1",
- "id": "email-audit:v1",
- "name": "email-audit",
- "version": "v1",
- "revision": "20130823",
- "title": "Email Audit API",
- "description": "Lets you perform Google Apps email audits",
- "ownerDomain": "google.com",
- "ownerName": "Google",
- "icons": {
-  "x16": "http://www.google.com/images/icons/product/search-16.gif",
-  "x32": "http://www.google.com/images/icons/product/search-32.gif"
- },
- "documentationLink": "https://developers.google.com/admin-sdk/email-audit",
- "protocol": "rest",
- "baseUrl": "https://apps-apis.google.com/",
- "rootUrl": "https://apps-apis.google.com/",
- "servicePath": "/a/feeds/compliance/audit/",
- "auth": {
-  "oauth2": {
-   "scopes": {
-    "https://apps-apis.google.com/a/feeds/compliance/audit/": {
-     "description": "Manage email audits"
-    }
-   }
-  }
- },
- "schemas": {
- },
- "resources": {
- }
-}

src/gam-install.sh

@@ -0,0 +1,310 @@
+#!/usr/bin/env bash
+
+usage()
+{
+cat << EOF
+GAM installation script.
+
+OPTIONS:
+   -h      show help.
+   -d      Directory where gam folder will be installed. Default is \$HOME/bin/
+   -a      Architecture to install (i386, x86_64, arm). Default is to detect your arch with "uname -m".
+   -o      OS we are running (linux, macos). Default is to detect your OS with "uname -s".
+   -l      Just upgrade GAM to latest version. Skips project creation and auth.
+   -p      Profile update (true, false). Should script add gam command to environment. Default is true.
+   -u      Admin user email address to use with GAM. Default is to prompt.
+   -r      Regular user email address. Used to test service account access to user data. Default is to prompt.
+   -v      Version to install (latest, prerelease, draft, 3.8, etc). Default is latest.
+EOF
+}
+
+target_dir="$HOME/bin"
+gamarch=$(uname -m)
+gamos=$(uname -s)
+update_profile=true
+upgrade_only=false
+gamversion="latest"
+adminuser=""
+regularuser=""
+while getopts "hd:a:o:lp:u:r:v:" OPTION
+do
+     case $OPTION in
+         h) usage; exit;;
+         d) target_dir="$OPTARG";;
+         a) gamarch="$OPTARG";;
+         o) gamos="$OPTARG";;
+         l) upgrade_only=true;;
+         p) update_profile="$OPTARG";;
+         u) adminuser="$OPTARG";;
+         r) regularuser="$OPTARG";;
+         v) gamversion="$OPTARG";;
+         ?) usage; exit;;
+     esac
+done
+
+update_profile() {
+	[ -f "$1" ] || return 1
+
+	grep -F "$alias_line" "$1" > /dev/null 2>&1
+	if [ $? -ne 0 ]; then
+                echo_yellow "Adding gam alias to profile file $1."
+		echo -e "\n$alias_line" >> "$1"
+        else
+          echo_yellow "gam alias already exists in profile file $1. Skipping add."
+	fi
+}
+
+echo_red()
+{
+echo -e "\x1B[1;31m$1"
+echo -e '\x1B[0m'
+}
+
+echo_green()
+{
+echo -e "\x1B[1;32m$1"
+echo -e '\x1B[0m'
+}
+
+echo_yellow()
+{
+echo -e "\x1B[1;33m$1"
+echo -e '\x1B[0m'
+}
+
+case $gamos in
+  [lL]inux)
+    gamos="linux"
+    case $gamarch in
+      x86_64) gamfile="linux-x86_64.tar.xz";;
+      i?86) gamfile="linux-i686.tar.xz";;
+      arm*) gamfile="linux-armv7l.tar.xz";;
+      *)
+        echo_red "ERROR: this installer currently only supports i386, x86_64 and arm Linux. Looks like you're running on $gamarch. Exiting."
+        exit
+    esac
+    ;;
+  [Mm]ac[Oo][sS]|[Dd]arwin)
+    osver=$(sw_vers -productVersion | awk -F'.' '{print $2}')
+    if (( $osver < 10 )); then
+      echo_red "ERROR: GAM currently requires MacOS 10.10 or newer. You are running MacOS 10.$osver. Please upgrade." 
+      exit
+    else
+      echo_green "Good, you're running MacOS 10.$osver..."
+    fi
+    gamos="macos"
+    gamfile="macos.tar.xz"
+    ;;
+  *)
+    echo_red "Sorry, this installer currently only supports Linux and MacOS. Looks like you're runnning on $gamos. Exiting."
+    exit
+    ;;
+esac
+
+if [ "$gamversion" == "latest" -o "$gamversion" == "prerelease" -o "$gamversion" == "draft" ]; then
+  release_url="https://api.github.com/repos/jay0lee/GAM/releases"
+else
+  release_url="https://api.github.com/repos/jay0lee/GAM/releases/tags/v$gamversion"
+fi
+
+echo_yellow "Checking GitHub URL $release_url for $gamversion GAM release..."
+release_json=$(curl -s $release_url 2>&1 /dev/null)
+
+echo_yellow "Getting file and download URL..."
+# Python is sadly the nearest to universal way to safely handle JSON with Bash
+# At least this code should be compatible with just about any Python version ever
+# unlike GAM itself. If some users don't have Python we can try grep / sed / etc
+# but that gets really ugly
+pycode="import json
+import sys
+
+attrib = sys.argv[1]
+gamversion = sys.argv[2]
+
+release = json.load(sys.stdin)
+if type(release) is list:
+  for a_release in release:
+    if a_release['prerelease'] and gamversion != 'prerelease':
+      continue
+    elif a_release['draft'] and gamversion != 'draft':
+      continue
+    release = a_release
+    break
+for asset in release['assets']:
+  if asset[sys.argv[1]].endswith('$gamfile'):
+    print(asset[sys.argv[1]])
+    break"
+
+pycmd="python"
+$pycmd -V >/dev/null 2>&1
+rc=$?
+if (( $rc != 0 )); then
+  pycmd="python3"
+fi
+$pycmd -V >/dev/null 2>&1
+rc=$?
+if (( $rc != 0 )); then
+  echo_red "ERROR: No version of python installed."
+  exit
+fi
+
+browser_download_url=$(echo "$release_json" | $pycmd -c "$pycode" browser_download_url $gamversion)
+name=$(echo "$release_json" | $pycmd -c "$pycode" name $gamversion)
+# Temp dir for archive
+#temp_archive_dir=$(mktemp -d)
+temp_archive_dir=$(mktemp -d 2>/dev/null || mktemp -d -t 'mytmpdir')
+echo_yellow "Downloading file $name from $browser_download_url to $temp_archive_dir."
+# Save archive to temp w/o losing our path
+(cd $temp_archive_dir && curl -O -L $browser_download_url)
+
+mkdir -p "$target_dir"
+
+echo_yellow "Extracting archive to $target_dir"
+tar xf $temp_archive_dir/$name -C "$target_dir"
+rc=$?
+if (( $rc != 0 )); then
+  echo_red "ERROR: extracting the GAM archive with tar failed with error $rc. Exiting."
+  exit
+else
+  echo_green "Finished extracting GAM archive."
+fi
+
+if [ "$upgrade_only" = true ]; then
+  echo_green "Here's information about your GAM upgrade:"
+  "$target_dir/gam/gam" version
+  rc=$?
+  if (( $rc != 0 )); then
+    echo_red "ERROR: Failed running GAM for the first time with $rc. Please report this error to GAM mailing list. Exiting."
+    exit
+  fi
+
+  echo_green "GAM upgrade complete!"
+  exit
+fi
+
+# Update profile to add gam command
+if [ "$update_profile" = true ]; then
+  alias_line="alias gam=\"$target_dir/gam/gam\""
+  if [ "$gamos" == "linux" ]; then
+    update_profile "$HOME/.bashrc" || update_profile "$HOME/.bash_profile"
+  elif [ "$gamos" == "macos" ]; then
+    update_profile "$HOME/.profile" || update_profile "$HOME/.bash_profile"
+  fi
+else
+  echo_yellow "skipping profile update."
+fi
+
+while true; do
+  read -p "Can you run a full browser on this machine? (usually Y for MacOS, N for Linux if you SSH into this machine) " yn
+  case $yn in
+    [Yy]*)
+      break
+      ;;
+    [Nn]*)
+      touch "$target_dir/gam/nobrowser.txt" > /dev/null 2>&1
+      break
+      ;;
+    *)
+      echo_red "Please answer yes or no."
+      ;;
+  esac
+done
+echo
+
+project_created=false
+while true; do
+  read -p "GAM is now installed. Are you ready to set up a Google API project for GAM? (yes or no) " yn
+  case $yn in
+    [Yy]*)
+      if [ "$adminuser" == "" ]; then
+        read -p "Please enter your G Suite admin email address: " adminuser
+      fi
+      "$target_dir/gam/gam" create project $adminuser
+      rc=$?
+      if (( $rc == 0 )); then
+        echo_green "Project creation complete."
+        project_created=true
+        break
+      else
+        echo_red "Project creation failed. Trying again. Say N to skip project creation."
+      fi
+      ;;
+    [Nn]*)
+      echo -e "\nYou can create an API project later by running:\n\ngam create project\n"
+      break
+      ;;
+    *)
+      echo_red "Please answer yes or no."
+      ;;
+  esac
+done
+
+admin_authorized=false
+while $project_created; do
+  read -p "Are you ready to authorize GAM to perform G Suite management operations as your admin account? (yes or no) " yn
+  case $yn in
+    [Yy]*)
+      "$target_dir/gam/gam" oauth create $adminuser
+      rc=$?
+      if (( $rc == 0 )); then
+        echo_green "Admin authorization complete."
+        admin_authorized=true
+        break
+      else
+        echo_red "Admin authorization failed. Trying again. Say N to skip admin authorization."
+      fi
+      ;;
+     [Nn]*)
+       echo -e "\nYou can authorize an admin later by running:\n\ngam oauth create\n"
+       break
+       ;;
+     *)
+       echo_red "Please answer yes or no."
+       ;;
+  esac
+done
+
+service_account_authorized=false
+while $project_created; do
+  read -p "Are you ready to authorize GAM to manage G Suite user data and settings? (yes or no) " yn
+  case $yn in
+    [Yy]*)
+      if [ "$regularuser" == "" ]; then
+        read -p "Please enter the email address of a regular G Suite user: " regularuser
+      fi
+      echo_yellow "Great! Checking service account scopes.This will fail the first time. Follow the steps to authorize and retry. It can take a few minutes for scopes to PASS after they've been authorized in the admin console."
+      "$target_dir/gam/gam" user $adminuser check serviceaccount
+      rc=$?
+      if (( $rc == 0 )); then
+        echo_green "Service account authorization complete."
+        service_account_authorized=true
+        break
+      else
+        echo_red "Service account authorization failed. Confirm you entered the scopes correctly in the admin console. It can take a few minutes for scopes to PASS after they are entered in the admin console so if you're sure you entered them correctly, go grab a coffee and then hit Y to try again. Say N to skip admin authorization."
+      fi
+      ;;
+     [Nn]*)
+       echo -e "\nYou can authorize a service account later by running:\n\ngam check serviceaccount\n"
+       break
+       ;;
+     *)
+       echo_red "Please answer yes or no."
+       ;;
+  esac
+done
+
+echo_green "Here's information about your new GAM installation:"
+"$target_dir/gam/gam" version
+rc=$?
+if (( $rc != 0 )); then
+  echo_red "ERROR: Failed running GAM for the first time with $rc. Please report this error to GAM mailing list. Exiting."
+  exit
+fi
+
+echo_green "GAM installation and setup complete!"
+if [ "$update_profile" = true ]; then
+  echo_green "Please restart your terminal shell or to get started right away run:\n\n$alias_line"
+fi
+
+# Clean up after ourselves even if we are killed with CTRL-C
+trap "rm -rf $temp_archive_dir" EXIT

src/gam-setup.bat

@@ -0,0 +1,75 @@
+@echo(
+@set /p adminemail= "Please enter your G Suite admin email address: "
+
+:createproject
+@echo(
+@set /p yn= "Are you ready to set up a Google API project for GAM? [y or n] "
+@if /I "%yn%"=="n" (
+@  echo(
+@  echo You can create an API project later by running:
+@  echo(
+@  echo gam create project
+@  goto alldone
+   )
+@if /I not "%yn%"=="y" (
+@  echo(
+@  echo Please answer y or n.
+@  goto createproject
+   )
+@gam create project %adminemail%
+@if not ERRORLEVEL 1 goto projectdone
+@echo(
+@echo Projection creation failed. Trying again. Say n to skip projection creation.
+@goto createproject
+:projectdone
+
+:adminauth
+@echo(
+@set /p yn= "Are you ready to authorize GAM to perform G Suite management operations as your admin account? [y or n] "
+@if /I "%yn%"=="n" (
+@  echo(
+@  echo You can authorize an admin later by running:
+@  echo(
+@  echo gam oauth create %adminemail%
+@  goto admindone
+   )
+@if /I not "%yn%"=="y" (
+@  echo(
+@  echo Please answer y or n.
+@  goto adminauth
+   )
+@gam oauth create %adminemail%
+@if not ERRORLEVEL 1 goto admindone
+@echo(
+@echo Admin authorization failed. Trying again. Say n to skip admin authorization.
+@goto adminauth
+:admindone
+
+:saauth
+@echo(
+@set /p yn= "Are you ready to authorize GAM to manage G Suite user data and settings? [y or n] "
+@if /I "%yn%"=="n" (
+@  echo(
+@  echo You can authorize a service account later by running:
+@  echo(
+@  echo gam user %adminemail% check serviceaccount
+@  goto sadone
+   )
+@if /I not "%yn%"=="y" (
+@  echo(
+@  echo Please answer y or n.
+@  goto saauth
+   )
+@echo(
+@set /p regularuser= "Please enter the email address of a regular G Suite user: "
+@echo Great! Checking service account scopes. This will fail the first time. Follow the steps to authorize and retry. It can take a few minutes for scopes to PASS after they've been authorized in the admin console.
+@gam user %regularuser% check serviceaccount
+@if not ERRORLEVEL 1 goto sadone
+@echo(
+@echo Service account authorization failed. Confirm you entered the scopes correctly in the admin console. It can take a few minutes for scopes to PASS after they are entered in the admin console so if you're sure you entered them correctly, go grab a coffee and then hit Y to try again. Say N to skip admin authorization.
+@goto saauth
+:sadone
+
+@echo GAM installation and setup complete!
+:alldone
+@pause

src/gam.wxs

@@ -44,6 +44,7 @@
         Source="gam-64">
       <Component Id="gam_exe" Guid="886abc07-73c5-4acc-9f71-58daf62aabc1">
         <File Name="gam.exe" KeyPath="yes" />
+        <Environment Id="PATH" Name="PATH" Value="[INSTALLFOLDER]" Permanent="yes" Part="last" Action="set" System="yes" />
       </Component>
       <Component Id="license" Guid="7a15de2e-fb91-4d0a-b8bf-c8b19c68f569">
         <File Name="LICENSE" KeyPath="yes" />
@@ -51,6 +52,12 @@
       <Component Id="whatsnew_txt" Guid="6aa9863c-90d9-412f-9b73-fda82549a950">
         <File Name="whatsnew.txt" KeyPath="yes" />
       </Component>
+      <Component Id="gam_setup_bat" Guid="ef01f93a-4b50-488a-9c04-ec5e13e66218">
+        <File Name="gam-setup.bat" KeyPath="yes" />
+      </Component>
+      <Component Id="gamcommands_txt" Guid="58ff9c45-a7c9-4e22-8845-a9a92610c1f3">
+        <File Name="gamcommands.txt" KeyPath="yes" />
+      </Component>
     </ComponentGroup>
   </Fragment>
 
@@ -58,7 +65,10 @@
     <InstallUISequence>
       <ExecuteAction />
       <Show Dialog="WelcomeDlg" Before="ProgressDlg" />
-      <!-- <Show Dialog="ProgressDlg" After="" /> -->
     </InstallUISequence>
+    <CustomAction Id="setup_gam" ExeCommand="[INSTALLFOLDER]gam-setup.bat" Directory="INSTALLFOLDER" Execute="commit" Impersonate="yes" Return="asyncWait"/>
+    <InstallExecuteSequence>
+     <Custom Action="setup_gam" After="InstallFiles" >NOT Installed AND NOT UPGRADINGPRODUCTCODE AND NOT WIX_UPGRADE_DETECTED</Custom>
+    </InstallExecuteSequence>
   </Fragment>
 </Wix>

src/googleapiclient/__init__.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = "1.5.2"
+__version__ = "1.6.2"
 
 # Set default logging handler to avoid "No handler found" warnings.
 import logging

src/googleapiclient/_auth.py

@@ -0,0 +1,92 @@
+# Copyright 2016 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helpers for authentication using oauth2client or google-auth."""
+
+try:
+    import google.auth
+    import google.auth.credentials
+    import google_auth_httplib2
+    HAS_GOOGLE_AUTH = True
+except ImportError:  # pragma: NO COVER
+    HAS_GOOGLE_AUTH = False
+
+try:
+    import oauth2client
+    import oauth2client.client
+    HAS_OAUTH2CLIENT = True
+except ImportError:  # pragma: NO COVER
+    HAS_OAUTH2CLIENT = False
+
+from googleapiclient.http import build_http
+
+
+def default_credentials():
+    """Returns Application Default Credentials."""
+    if HAS_GOOGLE_AUTH:
+        credentials, _ = google.auth.default()
+        return credentials
+    elif HAS_OAUTH2CLIENT:
+        return oauth2client.client.GoogleCredentials.get_application_default()
+    else:
+        raise EnvironmentError(
+            'No authentication library is available. Please install either '
+            'google-auth or oauth2client.')
+
+
+def with_scopes(credentials, scopes):
+    """Scopes the credentials if necessary.
+
+    Args:
+        credentials (Union[
+            google.auth.credentials.Credentials,
+            oauth2client.client.Credentials]): The credentials to scope.
+        scopes (Sequence[str]): The list of scopes.
+
+    Returns:
+        Union[google.auth.credentials.Credentials,
+            oauth2client.client.Credentials]: The scoped credentials.
+    """
+    if HAS_GOOGLE_AUTH and isinstance(
+            credentials, google.auth.credentials.Credentials):
+        return google.auth.credentials.with_scopes_if_required(
+            credentials, scopes)
+    else:
+        try:
+            if credentials.create_scoped_required():
+                return credentials.create_scoped(scopes)
+            else:
+                return credentials
+        except AttributeError:
+            return credentials
+
+
+def authorized_http(credentials):
+    """Returns an http client that is authorized with the given credentials.
+
+    Args:
+        credentials (Union[
+            google.auth.credentials.Credentials,
+            oauth2client.client.Credentials]): The credentials to use.
+
+    Returns:
+        Union[httplib2.Http, google_auth_httplib2.AuthorizedHttp]: An
+            authorized http client.
+    """
+    if HAS_GOOGLE_AUTH and isinstance(
+            credentials, google.auth.credentials.Credentials):
+        return google_auth_httplib2.AuthorizedHttp(credentials,
+                                                   http=build_http())
+    else:
+        return credentials.authorize(build_http())

src/googleapiclient/channel.py

@@ -61,7 +61,6 @@ import datetime
 import uuid
 
 from googleapiclient import errors
-from oauth2client import util
 import six
 
 # Oauth2client < 3 has the positional helper in 'util', >= 3 has it

src/googleapiclient/discovery.py

@@ -53,6 +53,7 @@ import httplib2
 import uritemplate
 
 # Local imports
+from googleapiclient import _auth
 from googleapiclient import mimeparse
 from googleapiclient.errors import HttpError
 from googleapiclient.errors import InvalidJsonError
@@ -60,7 +61,10 @@ from googleapiclient.errors import MediaUploadSizeError
 from googleapiclient.errors import UnacceptableMimeTypeError
 from googleapiclient.errors import UnknownApiNameOrVersion
 from googleapiclient.errors import UnknownFileType
+from googleapiclient.http import build_http
 from googleapiclient.http import BatchHttpRequest
+from googleapiclient.http import HttpMock
+from googleapiclient.http import HttpMockSequence
 from googleapiclient.http import HttpRequest
 from googleapiclient.http import MediaFileUpload
 from googleapiclient.http import MediaUpload
@@ -94,6 +98,7 @@ V2_DISCOVERY_URI = ('https://{api}.googleapis.com/$discovery/rest?'
                     'version={apiVersion}')
 DEFAULT_METHOD_DOC = 'A description of how to use this function'
 HTTP_PAYLOAD_METHODS = frozenset(['PUT', 'POST', 'PATCH'])
+
 _MEDIA_SIZE_BIT_SHIFTS = {'KB': 10, 'MB': 20, 'GB': 30, 'TB': 40}
 BODY_PARAMETER_DEFAULT_VALUE = {
     'description': 'The request body.',
@@ -106,6 +111,13 @@ MEDIA_BODY_PARAMETER_DEFAULT_VALUE = {
     'type': 'string',
     'required': False,
 }
+MEDIA_MIME_TYPE_PARAMETER_DEFAULT_VALUE = {
+    'description': ('The MIME type of the media request body, or an instance '
+                    'of a MediaUpload object.'),
+    'type': 'string',
+    'required': False,
+}
+_PAGE_TOKEN_NAMES = ('pageToken', 'nextPageToken')
 
 # Parameters accepted by the stack, but not visible via discovery.
 # TODO(dhermes): Remove 'userip' in 'v2'.
@@ -189,7 +201,8 @@ def build(serviceName,
     model: googleapiclient.Model, converts to and from the wire format.
     requestBuilder: googleapiclient.http.HttpRequest, encapsulator for an HTTP
       request.
-    credentials: oauth2client.Credentials, credentials to be used for
+    credentials: oauth2client.Credentials or
+      google.auth.credentials.Credentials, credentials to be used for
       authentication.
     cache_discovery: Boolean, whether or not to cache the discovery doc.
     cache: googleapiclient.discovery_cache.base.CacheBase, an optional
@@ -204,14 +217,16 @@ def build(serviceName,
       }
 
   if http is None:
-    http = httplib2.Http()
+    discovery_http = build_http()
+  else:
+    discovery_http = http
 
   for discovery_url in (discoveryServiceUrl, V2_DISCOVERY_URI,):
     requested_url = uritemplate.expand(discovery_url, params)
 
     try:
-      content = _retrieve_discovery_doc(requested_url, http, cache_discovery,
-                                        cache)
+      content = _retrieve_discovery_doc(
+        requested_url, discovery_http, cache_discovery, cache)
       return build_from_document(content, base=discovery_url, http=http,
           developerKey=developerKey, model=model, requestBuilder=requestBuilder,
           credentials=credentials)
@@ -308,17 +323,20 @@ def build_from_document(
     model: Model class instance that serializes and de-serializes requests and
       responses.
     requestBuilder: Takes an http request and packages it up to be executed.
-    credentials: object, credentials to be used for authentication.
+    credentials: oauth2client.Credentials or
+      google.auth.credentials.Credentials, credentials to be used for
+      authentication.
 
   Returns:
     A Resource object with methods for interacting with the service.
   """
 
-  if http is None:
-    http = httplib2.Http()
+  if http is not None and credentials is not None:
+    raise ValueError('Arguments http and credentials are mutually exclusive.')
 
-  # future is no longer used.
-  future = {}
+  if developerKey is not None and credentials is not None:
+    raise ValueError(
+      'Arguments developerKey and credentials are mutually exclusive.')
 
   if isinstance(service, six.string_types):
     service = json.loads(service)
@@ -334,31 +352,37 @@ def build_from_document(
   base = urljoin(service['rootUrl'], service['servicePath'])
   schema = Schemas(service)
 
-  if credentials:
-    # If credentials were passed in, we could have two cases:
-    # 1. the scopes were specified, in which case the given credentials
-    #    are used for authorizing the http;
-    # 2. the scopes were not provided (meaning the Application Default
-    #    Credentials are to be used). In this case, the Application Default
-    #    Credentials are built and used instead of the original credentials.
-    #    If there are no scopes found (meaning the given service requires no
-    #    authentication), there is no authorization of the http.
-    if (isinstance(credentials, GoogleCredentials) and
-        credentials.create_scoped_required()):
-      scopes = service.get('auth', {}).get('oauth2', {}).get('scopes', {})
-      if scopes:
-        credentials = credentials.create_scoped(list(scopes.keys()))
-      else:
-        # No need to authorize the http object
-        # if the service does not require authentication.
-        credentials = None
+  # If the http client is not specified, then we must construct an http client
+  # to make requests. If the service has scopes, then we also need to setup
+  # authentication.
+  if http is None:
+    # Does the service require scopes?
+    scopes = list(
+      service.get('auth', {}).get('oauth2', {}).get('scopes', {}).keys())
 
-    if credentials:
-      http = credentials.authorize(http)
+    # If so, then the we need to setup authentication if no developerKey is
+    # specified.
+    if scopes and not developerKey:
+      # If the user didn't pass in credentials, attempt to acquire application
+      # default credentials.
+      if credentials is None:
+        credentials = _auth.default_credentials()
+
+      # The credentials need to be scoped.
+      credentials = _auth.with_scopes(credentials, scopes)
+
+      # Create an authorized http instance
+      http = _auth.authorized_http(credentials)
+
+    # If the service doesn't require scopes then there is no need for
+    # authentication.
+    else:
+      http = build_http()
 
   if model is None:
     features = service.get('features', [])
     model = JsonModel('dataWrapper' in features)
+
   return Resource(http=http, baseUrl=base, model=model,
                   developerKey=developerKey, requestBuilder=requestBuilder,
                   resourceDesc=service, rootDesc=service, schema=schema)
@@ -479,7 +503,7 @@ def _fix_up_parameters(method_desc, root_desc, http_method):
 
 
 def _fix_up_media_upload(method_desc, root_desc, path_url, parameters):
-  """Updates parameters of API by adding 'media_body' if supported by method.
+  """Adds 'media_body' and 'media_mime_type' parameters if supported by method.
 
   SIDE EFFECTS: If the method supports media upload and has a required body,
   sets body to be optional (required=False) instead. Also, if there is a
@@ -516,6 +540,7 @@ def _fix_up_media_upload(method_desc, root_desc, path_url, parameters):
   if media_upload:
     media_path_url = _media_path_url_from_info(root_desc, path_url)
     parameters['media_body'] = MEDIA_BODY_PARAMETER_DEFAULT_VALUE.copy()
+    parameters['media_mime_type'] = MEDIA_MIME_TYPE_PARAMETER_DEFAULT_VALUE.copy()
     if 'body' in parameters:
       parameters['body']['required'] = False
 
@@ -704,7 +729,11 @@ def createMethod(methodName, methodDesc, rootDesc, schema):
 
     for name in parameters.required_params:
       if name not in kwargs:
-        raise TypeError('Missing required parameter "%s"' % name)
+        # temporary workaround for non-paging methods incorrectly requiring
+        # page token parameter (cf. drive.changes.watch vs. drive.changes.list)
+        if name not in _PAGE_TOKEN_NAMES or _findPageTokenName(
+            _methodProperties(methodDesc, schema, 'response')):
+          raise TypeError('Missing required parameter "%s"' % name)
 
     for name, regex in six.iteritems(parameters.pattern_params):
       if name in kwargs:
@@ -749,6 +778,7 @@ def createMethod(methodName, methodDesc, rootDesc, schema):
         actual_path_params[parameters.argmap[key]] = cast_value
     body_value = kwargs.get('body', None)
     media_filename = kwargs.get('media_body', None)
+    media_mime_type = kwargs.get('media_mime_type', None)
 
     if self._developerKey:
       actual_query_params['key'] = self._developerKey
@@ -772,7 +802,11 @@ def createMethod(methodName, methodDesc, rootDesc, schema):
     if media_filename:
       # Ensure we end up with a valid MediaUpload object.
       if isinstance(media_filename, six.string_types):
-        (media_mime_type, encoding) = mimetypes.guess_type(media_filename)
+        if media_mime_type is None:
+          logger.warning(
+              'media_mime_type argument not specified: trying to auto-detect for %s',
+              media_filename)
+          media_mime_type, _ = mimetypes.guess_type(media_filename)
         if media_mime_type is None:
           raise UnknownFileType(media_filename)
         if not mimeparse.best_match([media_mime_type], ','.join(accept)):
@@ -902,13 +936,20 @@ def createMethod(methodName, methodDesc, rootDesc, schema):
   return (methodName, method)
 
 
-def createNextMethod(methodName):
+def createNextMethod(methodName,
+                     pageTokenName='pageToken',
+                     nextPageTokenName='nextPageToken',
+                     isPageTokenParameter=True):
   """Creates any _next methods for attaching to a Resource.
 
   The _next methods allow for easy iteration through list() responses.
 
   Args:
     methodName: string, name of the method to use.
+    pageTokenName: string, name of request page token field.
+    nextPageTokenName: string, name of response page token field.
+    isPageTokenParameter: Boolean, True if request page token is a query
+        parameter, False if request page token is a field of the request body.
   """
   methodName = fix_method_name(methodName)
 
@@ -926,24 +967,24 @@ Returns:
     # Retrieve nextPageToken from previous_response
     # Use as pageToken in previous_request to create new request.
 
-    if 'nextPageToken' not in previous_response or not previous_response['nextPageToken']:
+    nextPageToken = previous_response.get(nextPageTokenName, None)
+    if not nextPageToken:
       return None
 
     request = copy.copy(previous_request)
 
-    pageToken = previous_response['nextPageToken']
-    parsed = list(urlparse(request.uri))
-    q = parse_qsl(parsed[4])
-
-    # Find and remove old 'pageToken' value from URI
-    newq = [(key, value) for (key, value) in q if key != 'pageToken']
-    newq.append(('pageToken', pageToken))
-    parsed[4] = urlencode(newq)
-    uri = urlunparse(parsed)
-
-    request.uri = uri
-
-    logger.info('URL being requested: %s %s' % (methodName,uri))
+    if isPageTokenParameter:
+        # Replace pageToken value in URI
+        request.uri = _add_query_parameter(
+            request.uri, pageTokenName, nextPageToken)
+        logger.info('Next page request URL: %s %s' % (methodName, request.uri))
+    else:
+        # Replace pageToken value in request body
+        model = self._model
+        body = model.deserialize(request.body)
+        body[pageTokenName] = nextPageToken
+        request.body = model.serialize(body)
+        logger.info('Next page request body: %s %s' % (methodName, body))
 
     return request
 
@@ -1091,19 +1132,59 @@ class Resource(object):
                                method.__get__(self, self.__class__))
 
   def _add_next_methods(self, resourceDesc, schema):
-    # Add _next() methods
-    # Look for response bodies in schema that contain nextPageToken, and methods
-    # that take a pageToken parameter.
-    if 'methods' in resourceDesc:
-      for methodName, methodDesc in six.iteritems(resourceDesc['methods']):
-        if 'response' in methodDesc:
-          responseSchema = methodDesc['response']
-          if '$ref' in responseSchema:
-            responseSchema = schema.get(responseSchema['$ref'])
-          hasNextPageToken = 'nextPageToken' in responseSchema.get('properties',
-                                                                   {})
-          hasPageToken = 'pageToken' in methodDesc.get('parameters', {})
-          if hasNextPageToken and hasPageToken:
-            fixedMethodName, method = createNextMethod(methodName + '_next')
-            self._set_dynamic_attr(fixedMethodName,
-                                   method.__get__(self, self.__class__))
+    # Add _next() methods if and only if one of the names 'pageToken' or
+    # 'nextPageToken' occurs among the fields of both the method's response
+    # type either the method's request (query parameters) or request body.
+    if 'methods' not in resourceDesc:
+      return
+    for methodName, methodDesc in six.iteritems(resourceDesc['methods']):
+      nextPageTokenName = _findPageTokenName(
+          _methodProperties(methodDesc, schema, 'response'))
+      if not nextPageTokenName:
+        continue
+      isPageTokenParameter = True
+      pageTokenName = _findPageTokenName(methodDesc.get('parameters', {}))
+      if not pageTokenName:
+        isPageTokenParameter = False
+        pageTokenName = _findPageTokenName(
+            _methodProperties(methodDesc, schema, 'request'))
+      if not pageTokenName:
+        continue
+      fixedMethodName, method = createNextMethod(
+          methodName + '_next', pageTokenName, nextPageTokenName,
+          isPageTokenParameter)
+      self._set_dynamic_attr(fixedMethodName,
+                             method.__get__(self, self.__class__))
+
+
+def _findPageTokenName(fields):
+  """Search field names for one like a page token.
+
+  Args:
+    fields: container of string, names of fields.
+
+  Returns:
+    First name that is either 'pageToken' or 'nextPageToken' if one exists,
+    otherwise None.
+  """
+  return next((tokenName for tokenName in _PAGE_TOKEN_NAMES
+              if tokenName in fields), None)
+
+def _methodProperties(methodDesc, schema, name):
+  """Get properties of a field in a method description.
+
+  Args:
+    methodDesc: object, fragment of deserialized discovery document that
+      describes the method.
+    schema: object, mapping of schema names to schema descriptions.
+    name: string, name of top-level field in method description.
+
+  Returns:
+    Object representing fragment of deserialized discovery document
+    corresponding to 'properties' field of object corresponding to named field
+    in method description, if it exists, otherwise empty dict.
+  """
+  desc = methodDesc.get(name, {})
+  if '$ref' in desc:
+    desc = schema.get(desc['$ref'], {})
+  return desc.get('properties', {})

src/googleapiclient/discovery_cache/file_cache.py

@@ -33,7 +33,12 @@ try:
   from oauth2client.contrib.locked_file import LockedFile
 except ImportError:
   # oauth2client < 2.0.0
-  from oauth2client.locked_file import LockedFile
+  try:
+    from oauth2client.locked_file import LockedFile
+  except ImportError:
+    # oauth2client > 4.0.0
+    raise ImportError(
+      'file_cache is unavailable when using oauth2client >= 4.0.0')
 
 from . import base
 from ..discovery_cache import DISCOVERY_DOC_MAX_AGE

src/googleapiclient/errors.py

@@ -52,8 +52,12 @@ class HttpError(Error):
     reason = self.resp.reason
     try:
       data = json.loads(self.content.decode('utf-8'))
-      reason = data['error']['message']
-    except (ValueError, KeyError):
+      if isinstance(data, dict):
+        reason = data['error']['message']
+      elif isinstance(data, list) and len(data) > 0:
+        first_error = data[0]
+        reason = first_error['error']['message']
+    except (ValueError, KeyError, TypeError):
       pass
     if reason is None:
       reason = ''
@@ -122,6 +126,9 @@ class BatchError(HttpError):
     self.reason = reason
 
   def __repr__(self):
+    if getattr(self.resp, 'status', None) is None:
+      return '<BatchError "%s">' % (self.reason)
+    else:
       return '<BatchError %s "%s">' % (self.resp.status, self.reason)
 
   __str__ = __repr__

src/googleapiclient/http.py

@@ -80,6 +80,8 @@ MAX_URI_LENGTH = 2048
 
 _TOO_MANY_REQUESTS = 429
 
+DEFAULT_HTTP_TIMEOUT_SEC = 60
+
 
 def _should_retry_response(resp_status, content):
   """Determines whether a response should be retried.
@@ -815,6 +817,7 @@ class HttpRequest(object):
     if 'content-length' not in self.headers:
       self.headers['content-length'] = str(self.body_size)
     # If the request URI is too long then turn it into a POST request.
+    # Assume that a GET request never contains a request body.
     if len(self.uri) > MAX_URI_LENGTH and self.method == 'GET':
       self.method = 'POST'
       self.headers['x-http-method-override'] = 'GET'
@@ -996,7 +999,11 @@ class HttpRequest(object):
     elif resp.status == 308:
       self._in_error_state = False
       # A "308 Resume Incomplete" indicates we are not done.
-      self.resumable_progress = int(resp['range'].split('-')[1]) + 1
+      try:
+        self.resumable_progress = int(resp['range'].split('-')[1]) + 1
+      except KeyError:
+        # If resp doesn't contain range header, resumable progress is 0
+        self.resumable_progress = 0
       if 'location' in resp:
         self.resumable_uri = resp['location']
     else:
@@ -1728,3 +1735,21 @@ def tunnel_patch(http):
 
   http.request = new_request
   return http
+
+
+def build_http():
+  """Builds httplib2.Http object
+
+  Returns:
+  A httplib2.Http object, which is used to make http requests, and which has timeout set by default.
+  To override default timeout call
+
+    socket.setdefaulttimeout(timeout_in_sec)
+
+  before interacting with this method.
+  """
+  if socket.getdefaulttimeout() is not None:
+    http_timeout = socket.getdefaulttimeout()
+  else:
+    http_timeout = DEFAULT_HTTP_TIMEOUT_SEC
+  return httplib2.Http(timeout=http_timeout)

src/googleapiclient/sample_tools.py

@@ -23,10 +23,10 @@ __all__ = ['init']
 
 
 import argparse
-import httplib2
 import os
 
 from googleapiclient import discovery
+from googleapiclient.http import build_http
 from oauth2client import client
 from oauth2client import file
 from oauth2client import tools
@@ -88,7 +88,7 @@ def init(argv, name, version, doc, filename, scope=None, parents=[], discovery_f
   credentials = storage.get()
   if credentials is None or credentials.invalid:
     credentials = tools.run_flow(flow, storage, flags)
-  http = credentials.authorize(http = httplib2.Http())
+  http = credentials.authorize(http=build_http())
 
   if discovery_filename is None:
     # Construct a service object via the discovery service.

src/googleapiclient/schema.py

@@ -161,13 +161,14 @@ class Schemas(object):
     # Return with trailing comma and newline removed.
     return self._prettyPrintSchema(schema, dent=1)[:-2]
 
-  def get(self, name):
+  def get(self, name, default=None):
     """Get deserialized JSON schema from the schema name.
 
     Args:
       name: string, Schema name.
+      default: object, return value if name not found.
     """
-    return self.schemas[name]
+    return self.schemas.get(name, default)
 
 
 class _SchemaToStruct(object):

src/httplib2/__init__.py

@@ -23,7 +23,7 @@ __contributors__ = ["Thomas Broyer (t.broyer@ltgt.net)",
                     "Louis Nyffenegger",
                     "Alex Yu"]
 __license__ = "MIT"
-__version__ = "0.9.2"
+__version__ = "0.10.3"
 
 import re
 import sys
@@ -65,42 +65,54 @@ except ImportError:
         socks = None
 
 # Build the appropriate socket wrapper for ssl
+ssl = None
+ssl_SSLError = None
+ssl_CertificateError = None
 try:
-    import ssl # python 2.6
-    ssl_SSLError = ssl.SSLError
-    def _ssl_wrap_socket(sock, key_file, cert_file, disable_validation,
-                         ca_certs, ssl_version, hostname):
-        if disable_validation:
-            cert_reqs = ssl.CERT_NONE
-        else:
-            cert_reqs = ssl.CERT_REQUIRED
-        if ssl_version is None:
-            ssl_version = ssl.PROTOCOL_SSLv23
+    import ssl  # python 2.6
+except ImportError:
+    pass
+if ssl is not None:
+    ssl_SSLError = getattr(ssl, 'SSLError', None)
+    ssl_CertificateError = getattr(ssl, 'CertificateError', None)
 
-        if hasattr(ssl, 'SSLContext'): # Python 2.7.9
-            context = ssl.SSLContext(ssl_version)
-            context.verify_mode = cert_reqs
-            context.check_hostname = (cert_reqs != ssl.CERT_NONE)
-            if cert_file:
-                context.load_cert_chain(cert_file, key_file)
-            if ca_certs:
-                context.load_verify_locations(ca_certs)
-            return context.wrap_socket(sock, server_hostname=hostname)
-        else:
-            return ssl.wrap_socket(sock, keyfile=key_file, certfile=cert_file,
-                                   cert_reqs=cert_reqs, ca_certs=ca_certs,
-                                   ssl_version=ssl_version)
-except (AttributeError, ImportError):
-    ssl_SSLError = None
-    def _ssl_wrap_socket(sock, key_file, cert_file, disable_validation,
-                         ca_certs, ssl_version, hostname):
-        if not disable_validation:
-            raise CertificateValidationUnsupported(
-                    "SSL certificate validation is not supported without "
-                    "the ssl module installed. To avoid this error, install "
-                    "the ssl module, or explicity disable validation.")
-        ssl_sock = socket.ssl(sock, key_file, cert_file)
-        return httplib.FakeSocket(sock, ssl_sock)
+
+def _ssl_wrap_socket(sock, key_file, cert_file, disable_validation,
+                     ca_certs, ssl_version, hostname):
+    if disable_validation:
+        cert_reqs = ssl.CERT_NONE
+    else:
+        cert_reqs = ssl.CERT_REQUIRED
+    if ssl_version is None:
+        ssl_version = ssl.PROTOCOL_SSLv23
+
+    if hasattr(ssl, 'SSLContext'):  # Python 2.7.9
+        context = ssl.SSLContext(ssl_version)
+        context.verify_mode = cert_reqs
+        context.check_hostname = (cert_reqs != ssl.CERT_NONE)
+        if cert_file:
+            context.load_cert_chain(cert_file, key_file)
+        if ca_certs:
+            context.load_verify_locations(ca_certs)
+        return context.wrap_socket(sock, server_hostname=hostname)
+    else:
+        return ssl.wrap_socket(sock, keyfile=key_file, certfile=cert_file,
+                               cert_reqs=cert_reqs, ca_certs=ca_certs,
+                               ssl_version=ssl_version)
+
+
+def _ssl_wrap_socket_unsupported(sock, key_file, cert_file, disable_validation,
+                                 ca_certs, ssl_version, hostname):
+    if not disable_validation:
+        raise CertificateValidationUnsupported(
+                "SSL certificate validation is not supported without "
+                "the ssl module installed. To avoid this error, install "
+                "the ssl module, or explicity disable validation.")
+    ssl_sock = socket.ssl(sock, key_file, cert_file)
+    return httplib.FakeSocket(sock, ssl_sock)
+
+if ssl is None:
+    _ssl_wrap_socket = _ssl_wrap_socket_unsupported
 
 
 if sys.version_info >= (2,3):
@@ -269,8 +281,8 @@ def safename(filename):
     filename = re_slash.sub(",", filename)
 
     # limit length of filename
-    if len(filename)>64:
-        filename=filename[:64]
+    if len(filename)>200:
+        filename=filename[:200]
     return ",".join((filename, filemd5))
 
 NORMALIZE_SPACE = re.compile(r'(?:\r\n)?[ \t]+')
@@ -1066,7 +1078,7 @@ class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
                         raise CertificateHostnameMismatch(
                             'Server presented certificate that does not match '
                             'host %s: %s' % (hostname, cert), hostname, cert)
-            except ssl_SSLError, e:
+            except (ssl_SSLError, ssl_CertificateError, CertificateHostnameMismatch), e:
                 if sock:
                     sock.close()
                 if self.sock:
@@ -1076,7 +1088,7 @@ class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
                 # to get at more detailed error information, in particular
                 # whether the error is due to certificate validation or
                 # something else (such as SSL protocol mismatch).
-                if e.errno == ssl.SSL_ERROR_SSL:
+                if getattr(e, 'errno', None) == ssl.SSL_ERROR_SSL:
                     raise SSLHandshakeError(e)
                 else:
                     raise
@@ -1155,18 +1167,11 @@ try:
               server_software.startswith('Development/')):
         raise NotRunningAppEngineEnvironment()
 
-    try:
-        from google.appengine.api import apiproxy_stub_map
-        if apiproxy_stub_map.apiproxy.GetStub('urlfetch') is None:
-            raise ImportError  # Bail out; we're not actually running on App Engine.
-        from google.appengine.api.urlfetch import fetch
-        from google.appengine.api.urlfetch import InvalidURLError
-    except (ImportError, AttributeError):
-        from google3.apphosting.api import apiproxy_stub_map
-        if apiproxy_stub_map.apiproxy.GetStub('urlfetch') is None:
-            raise ImportError  # Bail out; we're not actually running on App Engine.
-        from google3.apphosting.api.urlfetch import fetch
-        from google3.apphosting.api.urlfetch import InvalidURLError
+    from google.appengine.api import apiproxy_stub_map
+    if apiproxy_stub_map.apiproxy.GetStub('urlfetch') is None:
+        raise ImportError  # Bail out; we're not actually running on App Engine.
+    from google.appengine.api.urlfetch import fetch
+    from google.appengine.api.urlfetch import InvalidURLError
 
     # Update the connection classes to use the Googel App Engine specific ones.
     SCHEME_TO_CONNECTION = {

src/linux-build.sh

@@ -1,11 +1,12 @@
 rm -rf gam
 rm -rf build
-rm -rf  dist
+rm -rf dist
 rm -rf gam-$1-linux-$(arch).tar.xz
 
 export LD_LIBRARY_PATH=/usr/local/lib
 pyinstaller --clean -F --distpath=gam linux-gam.spec
 cp LICENSE gam
 cp whatsnew.txt gam
+cp GamCommands.txt gam
 
 tar cfJ gam-$1-linux-$(arch).tar.xz gam/ 

src/linux-gam.spec

@@ -8,9 +8,8 @@ for d in a.datas:
     if 'pyconfig' in d[0]:
         a.datas.remove(d)
         break
-a.datas += [('httplib2/cacerts.txt', 'httplib2\cacerts.txt', 'DATA')]
+a.datas += [('httplib2/cacerts.txt', 'httplib2/cacerts.txt', 'DATA')]
 a.datas += [('cloudprint-v2.json', 'cloudprint-v2.json', 'DATA')]
-a.datas += [('email-audit-v1.json', 'email-audit-v1.json', 'DATA')]
 a.datas += [('email-settings-v2.json', 'email-settings-v2.json', 'DATA')]
 pyz = PYZ(a.pure)
 exe = EXE(pyz,
@@ -21,5 +20,5 @@ exe = EXE(pyz,
           name='gam',
           debug=False,
           strip=None,
-          upx=True,
+          upx=False,
           console=True )

src/macos-build.sh

@@ -1,10 +1,11 @@
-rmdir /q /s gam
-rmdir /q /s build
-rmdir /q /s dist
+rm -rf gam
+rm -rf build
+rm -rf dist
 rm -rf gam-$1-macos.tar.xz
 
-pyinstaller --clean -F --distpath=gam macos-gam.spec
+/Library/Frameworks/Python.framework/Versions/2.7/bin/pyinstaller --clean -F --distpath=gam macos-gam.spec
 cp LICENSE gam
 cp whatsnew.txt gam
+cp GamCommands.txt gam
 
 tar cfJ gam-$1-macos.tar.xz gam/ 

src/macos-gam.spec

@@ -8,9 +8,8 @@ for d in a.datas:
     if 'pyconfig' in d[0]:
         a.datas.remove(d)
         break
-a.datas += [('httplib2/cacerts.txt', 'httplib2\cacerts.txt', 'DATA')]
+a.datas += [('httplib2/cacerts.txt', 'httplib2/cacerts.txt', 'DATA')]
 a.datas += [('cloudprint-v2.json', 'cloudprint-v2.json', 'DATA')]
-a.datas += [('email-audit-v1.json', 'email-audit-v1.json', 'DATA')]
 a.datas += [('email-settings-v2.json', 'email-settings-v2.json', 'DATA')]
 pyz = PYZ(a.pure)
 exe = EXE(pyz,
@@ -21,5 +20,5 @@ exe = EXE(pyz,
           name='gam',
           debug=False,
           strip=None,
-          upx=True,
+          upx=False,
           console=True )

src/oauth2client/__init__.py

@@ -14,7 +14,7 @@
 
 """Client library for using OAuth2, especially with Google APIs."""
 
-__version__ = '3.0.0'
+__version__ = '4.1.0'
 
 GOOGLE_AUTH_URI = 'https://accounts.google.com/o/oauth2/v2/auth'
 GOOGLE_DEVICE_URI = 'https://accounts.google.com/o/oauth2/device/code'

src/oauth2client/_helpers.py

@@ -11,12 +11,248 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 """Helper functions for commonly used utilities."""
 
 import base64
+import functools
+import inspect
 import json
+import logging
+import os
+import warnings
 
 import six
+from six.moves import urllib
+
+
+logger = logging.getLogger(__name__)
+
+POSITIONAL_WARNING = 'WARNING'
+POSITIONAL_EXCEPTION = 'EXCEPTION'
+POSITIONAL_IGNORE = 'IGNORE'
+POSITIONAL_SET = frozenset([POSITIONAL_WARNING, POSITIONAL_EXCEPTION,
+                            POSITIONAL_IGNORE])
+
+positional_parameters_enforcement = POSITIONAL_WARNING
+
+_SYM_LINK_MESSAGE = 'File: {0}: Is a symbolic link.'
+_IS_DIR_MESSAGE = '{0}: Is a directory'
+_MISSING_FILE_MESSAGE = 'Cannot access {0}: No such file or directory'
+
+
+def positional(max_positional_args):
+    """A decorator to declare that only the first N arguments my be positional.
+
+    This decorator makes it easy to support Python 3 style keyword-only
+    parameters. For example, in Python 3 it is possible to write::
+
+        def fn(pos1, *, kwonly1=None, kwonly1=None):
+            ...
+
+    All named parameters after ``*`` must be a keyword::
+
+        fn(10, 'kw1', 'kw2')  # Raises exception.
+        fn(10, kwonly1='kw1')  # Ok.
+
+    Example
+    ^^^^^^^
+
+    To define a function like above, do::
+
+        @positional(1)
+        def fn(pos1, kwonly1=None, kwonly2=None):
+            ...
+
+    If no default value is provided to a keyword argument, it becomes a
+    required keyword argument::
+
+        @positional(0)
+        def fn(required_kw):
+            ...
+
+    This must be called with the keyword parameter::
+
+        fn()  # Raises exception.
+        fn(10)  # Raises exception.
+        fn(required_kw=10)  # Ok.
+
+    When defining instance or class methods always remember to account for
+    ``self`` and ``cls``::
+
+        class MyClass(object):
+
+            @positional(2)
+            def my_method(self, pos1, kwonly1=None):
+                ...
+
+            @classmethod
+            @positional(2)
+            def my_method(cls, pos1, kwonly1=None):
+                ...
+
+    The positional decorator behavior is controlled by
+    ``_helpers.positional_parameters_enforcement``, which may be set to
+    ``POSITIONAL_EXCEPTION``, ``POSITIONAL_WARNING`` or
+    ``POSITIONAL_IGNORE`` to raise an exception, log a warning, or do
+    nothing, respectively, if a declaration is violated.
+
+    Args:
+        max_positional_arguments: Maximum number of positional arguments. All
+                                  parameters after the this index must be
+                                  keyword only.
+
+    Returns:
+        A decorator that prevents using arguments after max_positional_args
+        from being used as positional parameters.
+
+    Raises:
+        TypeError: if a key-word only argument is provided as a positional
+                   parameter, but only if
+                   _helpers.positional_parameters_enforcement is set to
+                   POSITIONAL_EXCEPTION.
+    """
+
+    def positional_decorator(wrapped):
+        @functools.wraps(wrapped)
+        def positional_wrapper(*args, **kwargs):
+            if len(args) > max_positional_args:
+                plural_s = ''
+                if max_positional_args != 1:
+                    plural_s = 's'
+                message = ('{function}() takes at most {args_max} positional '
+                           'argument{plural} ({args_given} given)'.format(
+                               function=wrapped.__name__,
+                               args_max=max_positional_args,
+                               args_given=len(args),
+                               plural=plural_s))
+                if positional_parameters_enforcement == POSITIONAL_EXCEPTION:
+                    raise TypeError(message)
+                elif positional_parameters_enforcement == POSITIONAL_WARNING:
+                    logger.warning(message)
+            return wrapped(*args, **kwargs)
+        return positional_wrapper
+
+    if isinstance(max_positional_args, six.integer_types):
+        return positional_decorator
+    else:
+        args, _, _, defaults = inspect.getargspec(max_positional_args)
+        return positional(len(args) - len(defaults))(max_positional_args)
+
+
+def scopes_to_string(scopes):
+    """Converts scope value to a string.
+
+    If scopes is a string then it is simply passed through. If scopes is an
+    iterable then a string is returned that is all the individual scopes
+    concatenated with spaces.
+
+    Args:
+        scopes: string or iterable of strings, the scopes.
+
+    Returns:
+        The scopes formatted as a single string.
+    """
+    if isinstance(scopes, six.string_types):
+        return scopes
+    else:
+        return ' '.join(scopes)
+
+
+def string_to_scopes(scopes):
+    """Converts stringifed scope value to a list.
+
+    If scopes is a list then it is simply passed through. If scopes is an
+    string then a list of each individual scope is returned.
+
+    Args:
+        scopes: a string or iterable of strings, the scopes.
+
+    Returns:
+        The scopes in a list.
+    """
+    if not scopes:
+        return []
+    elif isinstance(scopes, six.string_types):
+        return scopes.split(' ')
+    else:
+        return scopes
+
+
+def parse_unique_urlencoded(content):
+    """Parses unique key-value parameters from urlencoded content.
+
+    Args:
+        content: string, URL-encoded key-value pairs.
+
+    Returns:
+        dict, The key-value pairs from ``content``.
+
+    Raises:
+        ValueError: if one of the keys is repeated.
+    """
+    urlencoded_params = urllib.parse.parse_qs(content)
+    params = {}
+    for key, value in six.iteritems(urlencoded_params):
+        if len(value) != 1:
+            msg = ('URL-encoded content contains a repeated value:'
+                   '%s -> %s' % (key, ', '.join(value)))
+            raise ValueError(msg)
+        params[key] = value[0]
+    return params
+
+
+def update_query_params(uri, params):
+    """Updates a URI with new query parameters.
+
+    If a given key from ``params`` is repeated in the ``uri``, then
+    the URI will be considered invalid and an error will occur.
+
+    If the URI is valid, then each value from ``params`` will
+    replace the corresponding value in the query parameters (if
+    it exists).
+
+    Args:
+        uri: string, A valid URI, with potential existing query parameters.
+        params: dict, A dictionary of query parameters.
+
+    Returns:
+        The same URI but with the new query parameters added.
+    """
+    parts = urllib.parse.urlparse(uri)
+    query_params = parse_unique_urlencoded(parts.query)
+    query_params.update(params)
+    new_query = urllib.parse.urlencode(query_params)
+    new_parts = parts._replace(query=new_query)
+    return urllib.parse.urlunparse(new_parts)
+
+
+def _add_query_parameter(url, name, value):
+    """Adds a query parameter to a url.
+
+    Replaces the current value if it already exists in the URL.
+
+    Args:
+        url: string, url to add the query parameter to.
+        name: string, query parameter name.
+        value: string, query parameter value.
+
+    Returns:
+        Updated query parameter. Does not update the url if value is None.
+    """
+    if value is None:
+        return url
+    else:
+        return update_query_params(url, {name: value})
+
+
+def validate_file(filename):
+    if os.path.islink(filename):
+        raise IOError(_SYM_LINK_MESSAGE.format(filename))
+    elif os.path.isdir(filename):
+        raise IOError(_IS_DIR_MESSAGE.format(filename))
+    elif not os.path.isfile(filename):
+        warnings.warn(_MISSING_FILE_MESSAGE.format(filename))
 
 
 def _parse_pem_key(raw_key_input):

src/oauth2client/_pkce.py

@@ -0,0 +1,67 @@
+# Copyright 2016 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Utility functions for implementing Proof Key for Code Exchange (PKCE) by OAuth
+Public Clients
+
+See RFC7636.
+"""
+
+import base64
+import hashlib
+import os
+
+
+def code_verifier(n_bytes=64):
+    """
+    Generates a 'code_verifier' as described in section 4.1 of RFC 7636.
+
+    This is a 'high-entropy cryptographic random string' that will be
+    impractical for an attacker to guess.
+
+    Args:
+        n_bytes: integer between 31 and 96, inclusive. default: 64
+            number of bytes of entropy to include in verifier.
+
+    Returns:
+        Bytestring, representing urlsafe base64-encoded random data.
+    """
+    verifier = base64.urlsafe_b64encode(os.urandom(n_bytes)).rstrip(b'=')
+    # https://tools.ietf.org/html/rfc7636#section-4.1
+    # minimum length of 43 characters and a maximum length of 128 characters.
+    if len(verifier) < 43:
+        raise ValueError("Verifier too short. n_bytes must be > 30.")
+    elif len(verifier) > 128:
+        raise ValueError("Verifier too long. n_bytes must be < 97.")
+    else:
+        return verifier
+
+
+def code_challenge(verifier):
+    """
+    Creates a 'code_challenge' as described in section 4.2 of RFC 7636
+    by taking the sha256 hash of the verifier and then urlsafe
+    base64-encoding it.
+
+    Args:
+        verifier: bytestring, representing a code_verifier as generated by
+            code_verifier().
+
+    Returns:
+        Bytestring, representing a urlsafe base64-encoded sha256 hash digest,
+            without '=' padding.
+    """
+    digest = hashlib.sha256(verifier).digest()
+    return base64.urlsafe_b64encode(digest).rstrip(b'=')

src/oauth2client/client.py

@@ -34,13 +34,11 @@ from six.moves import urllib
 
 import oauth2client
 from oauth2client import _helpers
+from oauth2client import _pkce
 from oauth2client import clientsecrets
 from oauth2client import transport
-from oauth2client import util
 
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
-
 HAS_OPENSSL = False
 HAS_CRYPTO = False
 try:
@@ -100,20 +98,20 @@ AccessTokenInfo = collections.namedtuple(
 DEFAULT_ENV_NAME = 'UNKNOWN'
 
 # If set to True _get_environment avoid GCE check (_detect_gce_environment)
-NO_GCE_CHECK = os.environ.setdefault('NO_GCE_CHECK', 'False')
+NO_GCE_CHECK = os.getenv('NO_GCE_CHECK', 'False')
 
 # Timeout in seconds to wait for the GCE metadata server when detecting the
 # GCE environment.
 try:
-    GCE_METADATA_TIMEOUT = int(
-        os.environ.setdefault('GCE_METADATA_TIMEOUT', '3'))
+    GCE_METADATA_TIMEOUT = int(os.getenv('GCE_METADATA_TIMEOUT', 3))
 except ValueError:  # pragma: NO COVER
     GCE_METADATA_TIMEOUT = 3
 
 _SERVER_SOFTWARE = 'SERVER_SOFTWARE'
-_GCE_METADATA_HOST = '169.254.169.254'
-_METADATA_FLAVOR_HEADER = 'Metadata-Flavor'
+_GCE_METADATA_URI = 'http://' + os.getenv('GCE_METADATA_IP', '169.254.169.254')
+_METADATA_FLAVOR_HEADER = 'metadata-flavor'  # lowercase header
 _DESIRED_METADATA_FLAVOR = 'Google'
+_GCE_HEADERS = {_METADATA_FLAVOR_HEADER: _DESIRED_METADATA_FLAVOR}
 
 # Expose utcnow() at module level to allow for
 # easier testing (by replacing with a stub).
@@ -440,23 +438,6 @@ class Storage(object):
             self.release_lock()
 
 
-def _update_query_params(uri, params):
-    """Updates a URI with new query parameters.
-
-    Args:
-        uri: string, A valid URI, with potential existing query parameters.
-        params: dict, A dictionary of query parameters.
-
-    Returns:
-        The same URI but with the new query parameters added.
-    """
-    parts = urllib.parse.urlparse(uri)
-    query_params = dict(urllib.parse.parse_qsl(parts.query))
-    query_params.update(params)
-    new_parts = parts._replace(query=urllib.parse.urlencode(query_params))
-    return urllib.parse.urlunparse(new_parts)
-
-
 class OAuth2Credentials(Credentials):
     """Credentials object for OAuth 2.0.
 
@@ -466,11 +447,11 @@ class OAuth2Credentials(Credentials):
     OAuth2Credentials objects may be safely pickled and unpickled.
     """
 
-    @util.positional(8)
+    @_helpers.positional(8)
     def __init__(self, access_token, client_id, client_secret, refresh_token,
                  token_expiry, token_uri, user_agent, revoke_uri=None,
                  id_token=None, token_response=None, scopes=None,
-                 token_info_uri=None):
+                 token_info_uri=None, id_token_jwt=None):
         """Create an instance of OAuth2Credentials.
 
         This constructor is not usually called by the user, instead
@@ -493,8 +474,11 @@ class OAuth2Credentials(Credentials):
                             because some providers (e.g. wordpress.com) include
                             extra fields that clients may want.
             scopes: list, authorized scopes for these credentials.
-          token_info_uri: string, the URI for the token info endpoint. Defaults
-                          to None; scopes can not be refreshed if this is None.
+            token_info_uri: string, the URI for the token info endpoint.
+                            Defaults to None; scopes can not be refreshed if
+                            this is None.
+            id_token_jwt: string, the encoded and signed identity JWT. The
+                          decoded version of this is stored in id_token.
 
         Notes:
             store: callable, A callable that when passed a Credential
@@ -512,8 +496,9 @@ class OAuth2Credentials(Credentials):
         self.user_agent = user_agent
         self.revoke_uri = revoke_uri
         self.id_token = id_token
+        self.id_token_jwt = id_token_jwt
         self.token_response = token_response
-        self.scopes = set(util.string_to_scopes(scopes or []))
+        self.scopes = set(_helpers.string_to_scopes(scopes or []))
         self.token_info_uri = token_info_uri
 
         # True if the credentials have been revoked or expired and can't be
@@ -557,7 +542,7 @@ class OAuth2Credentials(Credentials):
             http: httplib2.Http, an http object to be used to make the refresh
                   request.
         """
-        self._refresh(http.request)
+        self._refresh(http)
 
     def revoke(self, http):
         """Revokes a refresh_token and makes the credentials void.
@@ -566,7 +551,7 @@ class OAuth2Credentials(Credentials):
             http: httplib2.Http, an http object to be used to make the revoke
                   request.
         """
-        self._revoke(http.request)
+        self._revoke(http)
 
     def apply(self, headers):
         """Add the authorization to the headers.
@@ -592,7 +577,7 @@ class OAuth2Credentials(Credentials):
             not have scopes. In both cases, you can use refresh_scopes() to
             obtain the canonical set of scopes.
         """
-        scopes = util.string_to_scopes(scopes)
+        scopes = _helpers.string_to_scopes(scopes)
         return set(scopes).issubset(self.scopes)
 
     def retrieve_scopes(self, http):
@@ -607,7 +592,7 @@ class OAuth2Credentials(Credentials):
         Returns:
             A set of strings containing the canonical list of scopes.
         """
-        self._retrieve_scopes(http.request)
+        self._retrieve_scopes(http)
         return self.scopes
 
     @classmethod
@@ -640,6 +625,7 @@ class OAuth2Credentials(Credentials):
             data['user_agent'],
             revoke_uri=data.get('revoke_uri', None),
             id_token=data.get('id_token', None),
+            id_token_jwt=data.get('id_token_jwt', None),
             token_response=data.get('token_response', None),
             scopes=data.get('scopes', None),
             token_info_uri=data.get('token_info_uri', None))
@@ -746,7 +732,7 @@ class OAuth2Credentials(Credentials):
 
         return headers
 
-    def _refresh(self, http_request):
+    def _refresh(self, http):
         """Refreshes the access_token.
 
         This method first checks by reading the Storage object if available.
@@ -754,15 +740,13 @@ class OAuth2Credentials(Credentials):
         refresh is completed.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          refresh request.
+            http: an object to be used to make HTTP requests.
 
         Raises:
             HttpAccessTokenRefreshError: When the refresh fails.
         """
         if not self.store:
-            self._do_refresh_request(http_request)
+            self._do_refresh_request(http)
         else:
             self.store.acquire_lock()
             try:
@@ -774,17 +758,15 @@ class OAuth2Credentials(Credentials):
                     logger.info('Updated access_token read from Storage')
                     self._updateFromCredential(new_cred)
                 else:
-                    self._do_refresh_request(http_request)
+                    self._do_refresh_request(http)
             finally:
                 self.store.release_lock()
 
-    def _do_refresh_request(self, http_request):
+    def _do_refresh_request(self, http):
         """Refresh the access_token using the refresh_token.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          refresh request.
+            http: an object to be used to make HTTP requests.
 
         Raises:
             HttpAccessTokenRefreshError: When the refresh fails.
@@ -793,8 +775,9 @@ class OAuth2Credentials(Credentials):
         headers = self._generate_refresh_request_headers()
 
         logger.info('Refreshing access_token')
-        resp, content = http_request(
-            self.token_uri, method='POST', body=body, headers=headers)
+        resp, content = transport.request(
+            http, self.token_uri, method='POST',
+            body=body, headers=headers)
         content = _helpers._from_bytes(content)
         if resp.status == http_client.OK:
             d = json.loads(content)
@@ -808,8 +791,10 @@ class OAuth2Credentials(Credentials):
                 self.token_expiry = None
             if 'id_token' in d:
                 self.id_token = _extract_id_token(d['id_token'])
+                self.id_token_jwt = d['id_token']
             else:
                 self.id_token = None
+                self.id_token_jwt = None
             # On temporary refresh errors, the user does not actually have to
             # re-authorize, so we unflag here.
             self.invalid = False
@@ -819,7 +804,7 @@ class OAuth2Credentials(Credentials):
             # An {'error':...} response body means the token is expired or
             # revoked, so we flag the credentials as such.
             logger.info('Failed to retrieve access token: %s', content)
-            error_msg = 'Invalid response {0}.'.format(resp['status'])
+            error_msg = 'Invalid response {0}.'.format(resp.status)
             try:
                 d = json.loads(content)
                 if 'error' in d:
@@ -833,23 +818,19 @@ class OAuth2Credentials(Credentials):
                 pass
             raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
 
-    def _revoke(self, http_request):
+    def _revoke(self, http):
         """Revokes this credential and deletes the stored copy (if it exists).
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          revoke request.
+            http: an object to be used to make HTTP requests.
         """
-        self._do_revoke(http_request, self.refresh_token or self.access_token)
+        self._do_revoke(http, self.refresh_token or self.access_token)
 
-    def _do_revoke(self, http_request, token):
+    def _do_revoke(self, http, token):
         """Revokes this credential and deletes the stored copy (if it exists).
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          refresh request.
+            http: an object to be used to make HTTP requests.
             token: A string used as the token to be revoked. Can be either an
                    access_token or refresh_token.
 
@@ -859,8 +840,13 @@ class OAuth2Credentials(Credentials):
         """
         logger.info('Revoking token')
         query_params = {'token': token}
-        token_revoke_uri = _update_query_params(self.revoke_uri, query_params)
-        resp, content = http_request(token_revoke_uri)
+        token_revoke_uri = _helpers.update_query_params(
+            self.revoke_uri, query_params)
+        resp, content = transport.request(http, token_revoke_uri)
+        if resp.status == http_client.METHOD_NOT_ALLOWED:
+            body = urllib.parse.urlencode(query_params)
+            resp, content = transport.request(http, token_revoke_uri,
+                                              method='POST', body=body)
         if resp.status == http_client.OK:
             self.invalid = True
         else:
@@ -876,23 +862,19 @@ class OAuth2Credentials(Credentials):
         if self.store:
             self.store.delete()
 
-    def _retrieve_scopes(self, http_request):
+    def _retrieve_scopes(self, http):
         """Retrieves the list of authorized scopes from the OAuth2 provider.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          revoke request.
+            http: an object to be used to make HTTP requests.
         """
-        self._do_retrieve_scopes(http_request, self.access_token)
+        self._do_retrieve_scopes(http, self.access_token)
 
-    def _do_retrieve_scopes(self, http_request, token):
+    def _do_retrieve_scopes(self, http, token):
         """Retrieves the list of authorized scopes from the OAuth2 provider.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          refresh request.
+            http: an object to be used to make HTTP requests.
             token: A string used as the token to identify the credentials to
                    the provider.
 
@@ -902,13 +884,13 @@ class OAuth2Credentials(Credentials):
         """
         logger.info('Refreshing scopes')
         query_params = {'access_token': token, 'fields': 'scope'}
-        token_info_uri = _update_query_params(self.token_info_uri,
-                                              query_params)
-        resp, content = http_request(token_info_uri)
+        token_info_uri = _helpers.update_query_params(
+            self.token_info_uri, query_params)
+        resp, content = transport.request(http, token_info_uri)
         content = _helpers._from_bytes(content)
         if resp.status == http_client.OK:
             d = json.loads(content)
-            self.scopes = set(util.string_to_scopes(d.get('scope', '')))
+            self.scopes = set(_helpers.string_to_scopes(d.get('scope', '')))
         else:
             error_msg = 'Invalid response {0}.'.format(resp.status)
             try:
@@ -977,19 +959,25 @@ class AccessTokenCredentials(OAuth2Credentials):
             data['user_agent'])
         return retval
 
-    def _refresh(self, http_request):
+    def _refresh(self, http):
+        """Refreshes the access token.
+
+        Args:
+            http: unused HTTP object.
+
+        Raises:
+            AccessTokenCredentialsError: always
+        """
         raise AccessTokenCredentialsError(
             'The access_token is expired or invalid and can\'t be refreshed.')
 
-    def _revoke(self, http_request):
+    def _revoke(self, http):
         """Revokes the access_token and deletes the store if available.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          revoke request.
+            http: an object to be used to make HTTP requests.
         """
-        self._do_revoke(http_request, self.access_token)
+        self._do_revoke(http, self.access_token)
 
 
 def _detect_gce_environment():
@@ -1005,21 +993,16 @@ def _detect_gce_environment():
     #       could lead to false negatives in the event that we are on GCE, but
     #       the metadata resolution was particularly slow. The latter case is
     #       "unlikely".
-    connection = six.moves.http_client.HTTPConnection(
-        _GCE_METADATA_HOST, timeout=GCE_METADATA_TIMEOUT)
-
+    http = transport.get_http_object(timeout=GCE_METADATA_TIMEOUT)
     try:
-        headers = {_METADATA_FLAVOR_HEADER: _DESIRED_METADATA_FLAVOR}
-        connection.request('GET', '/', headers=headers)
-        response = connection.getresponse()
-        if response.status == http_client.OK:
-            return (response.getheader(_METADATA_FLAVOR_HEADER) ==
-                    _DESIRED_METADATA_FLAVOR)
+        response, _ = transport.request(
+            http, _GCE_METADATA_URI, headers=_GCE_HEADERS)
+        return (
+            response.status == http_client.OK and
+            response.get(_METADATA_FLAVOR_HEADER) == _DESIRED_METADATA_FLAVOR)
     except socket.error:  # socket.timeout or socket.error(64, 'Host is down')
         logger.info('Timeout attempting to reach GCE metadata service.')
         return False
-    finally:
-        connection.close()
 
 
 def _in_gae_environment():
@@ -1469,7 +1452,7 @@ class AssertionCredentials(GoogleCredentials):
     AssertionCredentials objects may be safely pickled and unpickled.
     """
 
-    @util.positional(2)
+    @_helpers.positional(2)
     def __init__(self, assertion_type, user_agent=None,
                  token_uri=oauth2client.GOOGLE_TOKEN_URI,
                  revoke_uri=oauth2client.GOOGLE_REVOKE_URI,
@@ -1511,15 +1494,13 @@ class AssertionCredentials(GoogleCredentials):
         """Generate assertion string to be used in the access token request."""
         raise NotImplementedError
 
-    def _revoke(self, http_request):
+    def _revoke(self, http):
         """Revokes the access_token and deletes the store if available.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          revoke request.
+            http: an object to be used to make HTTP requests.
         """
-        self._do_revoke(http_request, self.access_token)
+        self._do_revoke(http, self.access_token)
 
     def sign_blob(self, blob):
         """Cryptographically sign a blob (of bytes).
@@ -1545,7 +1526,7 @@ def _require_crypto_or_die():
         raise CryptoUnavailableError('No crypto library available')
 
 
-@util.positional(2)
+@_helpers.positional(2)
 def verify_id_token(id_token, audience, http=None,
                     cert_uri=ID_TOKEN_VERIFICATION_CERTS):
     """Verifies a signed JWT id_token.
@@ -1572,7 +1553,7 @@ def verify_id_token(id_token, audience, http=None,
     if http is None:
         http = transport.get_cached_http()
 
-    resp, content = http.request(cert_uri)
+    resp, content = transport.request(http, cert_uri)
     if resp.status == http_client.OK:
         certs = json.loads(_helpers._from_bytes(content))
         return crypt.verify_signed_jwt_with_certs(id_token, certs, audience)
@@ -1624,7 +1605,7 @@ def _parse_exchange_token_response(content):
     except Exception:
         # different JSON libs raise different exceptions,
         # so we just do a catch-all here
-        resp = dict(urllib.parse.parse_qsl(content))
+        resp = _helpers.parse_unique_urlencoded(content)
 
     # some providers respond with 'expires', others with 'expires_in'
     if resp and 'expires' in resp:
@@ -1633,7 +1614,7 @@ def _parse_exchange_token_response(content):
     return resp
 
 
-@util.positional(4)
+@_helpers.positional(4)
 def credentials_from_code(client_id, client_secret, scope, code,
                           redirect_uri='postmessage', http=None,
                           user_agent=None,
@@ -1641,7 +1622,9 @@ def credentials_from_code(client_id, client_secret, scope, code,
                           auth_uri=oauth2client.GOOGLE_AUTH_URI,
                           revoke_uri=oauth2client.GOOGLE_REVOKE_URI,
                           device_uri=oauth2client.GOOGLE_DEVICE_URI,
-                          token_info_uri=oauth2client.GOOGLE_TOKEN_INFO_URI):
+                          token_info_uri=oauth2client.GOOGLE_TOKEN_INFO_URI,
+                          pkce=False,
+                          code_verifier=None):
     """Exchanges an authorization code for an OAuth2Credentials object.
 
     Args:
@@ -1665,6 +1648,15 @@ def credentials_from_code(client_id, client_secret, scope, code,
         device_uri: string, URI for device authorization endpoint. For
                     convenience defaults to Google's endpoints but any OAuth
                     2.0 provider can be used.
+        pkce: boolean, default: False, Generate and include a "Proof Key
+              for Code Exchange" (PKCE) with your authorization and token
+              requests. This adds security for installed applications that
+              cannot protect a client_secret. See RFC 7636 for details.
+        code_verifier: bytestring or None, default: None, parameter passed
+                       as part of the code exchange when pkce=True. If
+                       None, a code_verifier will automatically be
+                       generated as part of step1_get_authorize_url(). See
+                       RFC 7636 for details.
 
     Returns:
         An OAuth2Credentials object.
@@ -1675,16 +1667,20 @@ def credentials_from_code(client_id, client_secret, scope, code,
     """
     flow = OAuth2WebServerFlow(client_id, client_secret, scope,
                                redirect_uri=redirect_uri,
-                               user_agent=user_agent, auth_uri=auth_uri,
-                               token_uri=token_uri, revoke_uri=revoke_uri,
+                               user_agent=user_agent,
+                               auth_uri=auth_uri,
+                               token_uri=token_uri,
+                               revoke_uri=revoke_uri,
                                device_uri=device_uri,
-                               token_info_uri=token_info_uri)
+                               token_info_uri=token_info_uri,
+                               pkce=pkce,
+                               code_verifier=code_verifier)
 
     credentials = flow.step2_exchange(code, http=http)
     return credentials
 
 
-@util.positional(3)
+@_helpers.positional(3)
 def credentials_from_clientsecrets_and_code(filename, scope, code,
                                             message=None,
                                             redirect_uri='postmessage',
@@ -1713,6 +1709,15 @@ def credentials_from_clientsecrets_and_code(filename, scope, code,
         cache: An optional cache service client that implements get() and set()
                methods. See clientsecrets.loadfile() for details.
         device_uri: string, OAuth 2.0 device authorization endpoint
+        pkce: boolean, default: False, Generate and include a "Proof Key
+              for Code Exchange" (PKCE) with your authorization and token
+              requests. This adds security for installed applications that
+              cannot protect a client_secret. See RFC 7636 for details.
+        code_verifier: bytestring or None, default: None, parameter passed
+                       as part of the code exchange when pkce=True. If
+                       None, a code_verifier will automatically be
+                       generated as part of step1_get_authorize_url(). See
+                       RFC 7636 for details.
 
     Returns:
         An OAuth2Credentials object.
@@ -1803,7 +1808,7 @@ class OAuth2WebServerFlow(Flow):
     OAuth2WebServerFlow objects may be safely pickled and unpickled.
     """
 
-    @util.positional(4)
+    @_helpers.positional(4)
     def __init__(self, client_id,
                  client_secret=None,
                  scope=None,
@@ -1816,6 +1821,8 @@ class OAuth2WebServerFlow(Flow):
                  device_uri=oauth2client.GOOGLE_DEVICE_URI,
                  token_info_uri=oauth2client.GOOGLE_TOKEN_INFO_URI,
                  authorization_header=None,
+                 pkce=False,
+                 code_verifier=None,
                  **kwargs):
         """Constructor for OAuth2WebServerFlow.
 
@@ -1853,6 +1860,15 @@ class OAuth2WebServerFlow(Flow):
                                   require a client to authenticate using a
                                   header value instead of passing client_secret
                                   in the POST body.
+            pkce: boolean, default: False, Generate and include a "Proof Key
+                  for Code Exchange" (PKCE) with your authorization and token
+                  requests. This adds security for installed applications that
+                  cannot protect a client_secret. See RFC 7636 for details.
+            code_verifier: bytestring or None, default: None, parameter passed
+                           as part of the code exchange when pkce=True. If
+                           None, a code_verifier will automatically be
+                           generated as part of step1_get_authorize_url(). See
+                           RFC 7636 for details.
             **kwargs: dict, The keyword arguments are all optional and required
                       parameters for the OAuth calls.
         """
@@ -1862,7 +1878,7 @@ class OAuth2WebServerFlow(Flow):
             raise TypeError("The value of scope must not be None")
         self.client_id = client_id
         self.client_secret = client_secret
-        self.scope = util.scopes_to_string(scope)
+        self.scope = _helpers.scopes_to_string(scope)
         self.redirect_uri = redirect_uri
         self.login_hint = login_hint
         self.user_agent = user_agent
@@ -1872,9 +1888,11 @@ class OAuth2WebServerFlow(Flow):
         self.device_uri = device_uri
         self.token_info_uri = token_info_uri
         self.authorization_header = authorization_header
+        self._pkce = pkce
+        self.code_verifier = code_verifier
         self.params = _oauth2_web_server_flow_params(kwargs)
 
-    @util.positional(1)
+    @_helpers.positional(1)
     def step1_get_authorize_url(self, redirect_uri=None, state=None):
         """Returns a URI to redirect to the provider.
 
@@ -1912,10 +1930,17 @@ class OAuth2WebServerFlow(Flow):
             query_params['state'] = state
         if self.login_hint is not None:
             query_params['login_hint'] = self.login_hint
-        query_params.update(self.params)
-        return _update_query_params(self.auth_uri, query_params)
+        if self._pkce:
+            if not self.code_verifier:
+                self.code_verifier = _pkce.code_verifier()
+            challenge = _pkce.code_challenge(self.code_verifier)
+            query_params['code_challenge'] = challenge
+            query_params['code_challenge_method'] = 'S256'
 
-    @util.positional(1)
+        query_params.update(self.params)
+        return _helpers.update_query_params(self.auth_uri, query_params)
+
+    @_helpers.positional(1)
     def step1_get_device_and_user_codes(self, http=None):
         """Returns a user code and the verification URL where to enter it
 
@@ -1940,8 +1965,8 @@ class OAuth2WebServerFlow(Flow):
         if http is None:
             http = transport.get_http_object()
 
-        resp, content = http.request(self.device_uri, method='POST', body=body,
-                                     headers=headers)
+        resp, content = transport.request(
+            http, self.device_uri, method='POST', body=body, headers=headers)
         content = _helpers._from_bytes(content)
         if resp.status == http_client.OK:
             try:
@@ -1963,7 +1988,7 @@ class OAuth2WebServerFlow(Flow):
                 pass
             raise OAuth2DeviceCodeError(error_msg)
 
-    @util.positional(2)
+    @_helpers.positional(2)
     def step2_exchange(self, code=None, http=None, device_flow_info=None):
         """Exchanges a code for OAuth2Credentials.
 
@@ -2006,6 +2031,8 @@ class OAuth2WebServerFlow(Flow):
         }
         if self.client_secret is not None:
             post_data['client_secret'] = self.client_secret
+        if self._pkce:
+            post_data['code_verifier'] = self.code_verifier
         if device_flow_info is not None:
             post_data['grant_type'] = 'http://oauth.net/grant_type/device/1.0'
         else:
@@ -2023,8 +2050,8 @@ class OAuth2WebServerFlow(Flow):
         if http is None:
             http = transport.get_http_object()
 
-        resp, content = http.request(self.token_uri, method='POST', body=body,
-                                     headers=headers)
+        resp, content = transport.request(
+            http, self.token_uri, method='POST', body=body, headers=headers)
         d = _parse_exchange_token_response(content)
         if resp.status == http_client.OK and 'access_token' in d:
             access_token = d['access_token']
@@ -2039,15 +2066,17 @@ class OAuth2WebServerFlow(Flow):
                 token_expiry = delta + _UTCNOW()
 
             extracted_id_token = None
+            id_token_jwt = None
             if 'id_token' in d:
                 extracted_id_token = _extract_id_token(d['id_token'])
+                id_token_jwt = d['id_token']
 
             logger.info('Successfully retrieved access token')
             return OAuth2Credentials(
                 access_token, self.client_id, self.client_secret,
                 refresh_token, token_expiry, self.token_uri, self.user_agent,
                 revoke_uri=self.revoke_uri, id_token=extracted_id_token,
-                token_response=d, scopes=self.scope,
+                id_token_jwt=id_token_jwt, token_response=d, scopes=self.scope,
                 token_info_uri=self.token_info_uri)
         else:
             logger.info('Failed to retrieve access token: %s', content)
@@ -2060,10 +2089,10 @@ class OAuth2WebServerFlow(Flow):
             raise FlowExchangeError(error_msg)
 
 
-@util.positional(2)
+@_helpers.positional(2)
 def flow_from_clientsecrets(filename, scope, redirect_uri=None,
                             message=None, cache=None, login_hint=None,
-                            device_uri=None):
+                            device_uri=None, pkce=None, code_verifier=None):
     """Create a Flow from a clientsecrets file.
 
     Will create the right kind of Flow based on the contents of the
@@ -2112,10 +2141,11 @@ def flow_from_clientsecrets(filename, scope, redirect_uri=None,
                 'login_hint': login_hint,
             }
             revoke_uri = client_info.get('revoke_uri')
-            if revoke_uri is not None:
-                constructor_kwargs['revoke_uri'] = revoke_uri
-            if device_uri is not None:
-                constructor_kwargs['device_uri'] = device_uri
+            optional = ('revoke_uri', 'device_uri', 'pkce', 'code_verifier')
+            for param in optional:
+                if locals()[param] is not None:
+                    constructor_kwargs[param] = locals()[param]
+
             return OAuth2WebServerFlow(
                 client_info['client_id'], client_info['client_secret'],
                 scope, **constructor_kwargs)

src/oauth2client/clientsecrets.py

@@ -22,7 +22,6 @@ import json
 
 import six
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
 
 # Properties that make a client_secrets.json file valid.
 TYPE_WEB = 'web'

src/oauth2client/contrib/_fcntl_opener.py

@@ -1,81 +0,0 @@
-# Copyright 2016 Google Inc. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import errno
-import fcntl
-import time
-
-from oauth2client.contrib import locked_file
-
-
-class _FcntlOpener(locked_file._Opener):
-    """Open, lock, and unlock a file using fcntl.lockf."""
-
-    def open_and_lock(self, timeout, delay):
-        """Open the file and lock it.
-
-        Args:
-            timeout: float, How long to try to lock for.
-            delay: float, How long to wait between retries
-
-        Raises:
-            AlreadyLockedException: if the lock is already acquired.
-            IOError: if the open fails.
-            CredentialsFileSymbolicLinkError: if the file is a symbolic
-                                              link.
-        """
-        if self._locked:
-            raise locked_file.AlreadyLockedException(
-                'File {0} is already locked'.format(self._filename))
-        start_time = time.time()
-
-        locked_file.validate_file(self._filename)
-        try:
-            self._fh = open(self._filename, self._mode)
-        except IOError as e:
-            # If we can't access with _mode, try _fallback_mode and
-            # don't lock.
-            if e.errno in (errno.EPERM, errno.EACCES):
-                self._fh = open(self._filename, self._fallback_mode)
-                return
-
-        # We opened in _mode, try to lock the file.
-        while True:
-            try:
-                fcntl.lockf(self._fh.fileno(), fcntl.LOCK_EX)
-                self._locked = True
-                return
-            except IOError as e:
-                # If not retrying, then just pass on the error.
-                if timeout == 0:
-                    raise
-                if e.errno != errno.EACCES:
-                    raise
-                # We could not acquire the lock. Try again.
-                if (time.time() - start_time) >= timeout:
-                    locked_file.logger.warn('Could not lock %s in %s seconds',
-                                            self._filename, timeout)
-                    if self._fh:
-                        self._fh.close()
-                    self._fh = open(self._filename, self._fallback_mode)
-                    return
-                time.sleep(delay)
-
-    def unlock_and_close(self):
-        """Close and unlock the file using the fcntl.lockf primitive."""
-        if self._locked:
-            fcntl.lockf(self._fh.fileno(), fcntl.LOCK_UN)
-        self._locked = False
-        if self._fh:
-            self._fh.close()

src/oauth2client/contrib/_metadata.py

@@ -19,29 +19,28 @@ See https://cloud.google.com/compute/docs/metadata
 
 import datetime
 import json
+import os
 
-import httplib2
 from six.moves import http_client
 from six.moves.urllib import parse as urlparse
 
 from oauth2client import _helpers
 from oauth2client import client
-from oauth2client import util
+from oauth2client import transport
 
 
-METADATA_ROOT = 'http://metadata.google.internal/computeMetadata/v1/'
+METADATA_ROOT = 'http://{}/computeMetadata/v1/'.format(
+    os.getenv('GCE_METADATA_ROOT', 'metadata.google.internal'))
 METADATA_HEADERS = {'Metadata-Flavor': 'Google'}
 
 
-def get(http_request, path, root=METADATA_ROOT, recursive=None):
+def get(http, path, root=METADATA_ROOT, recursive=None):
     """Fetch a resource from the metadata server.
 
     Args:
+        http: an object to be used to make HTTP requests.
         path: A string indicating the resource to retrieve. For example,
-            'instance/service-accounts/defualt'
-        http_request: A callable that matches the method
-            signature of httplib2.Http.request. Used to make the request to the
-            metadataserver.
+            'instance/service-accounts/default'
         root: A string indicating the full path to the metadata server root.
         recursive: A boolean indicating whether to do a recursive query of
             metadata. See
@@ -51,15 +50,14 @@ def get(http_request, path, root=METADATA_ROOT, recursive=None):
         A dictionary if the metadata server returns JSON, otherwise a string.
 
     Raises:
-        httplib2.Httplib2Error if an error corrured while retrieving metadata.
+        http_client.HTTPException if an error corrured while
+        retrieving metadata.
     """
     url = urlparse.urljoin(root, path)
-    url = util._add_query_parameter(url, 'recursive', recursive)
+    url = _helpers._add_query_parameter(url, 'recursive', recursive)
 
-    response, content = http_request(
-        url,
-        headers=METADATA_HEADERS
-    )
+    response, content = transport.request(
+        http, url, headers=METADATA_HEADERS)
 
     if response.status == http_client.OK:
         decoded = _helpers._from_bytes(content)
@@ -68,21 +66,20 @@ def get(http_request, path, root=METADATA_ROOT, recursive=None):
         else:
             return decoded
     else:
-        raise httplib2.HttpLib2Error(
+        raise http_client.HTTPException(
             'Failed to retrieve {0} from the Google Compute Engine'
             'metadata service. Response:\n{1}'.format(url, response))
 
 
-def get_service_account_info(http_request, service_account='default'):
+def get_service_account_info(http, service_account='default'):
     """Get information about a service account from the metadata server.
 
     Args:
+        http: an object to be used to make HTTP requests.
         service_account: An email specifying the service account for which to
             look up information. Default will be information for the "default"
             service account of the current compute engine instance.
-        http_request: A callable that matches the method
-            signature of httplib2.Http.request. Used to make the request to the
-            metadata server.
+
     Returns:
          A dictionary with information about the specified service account,
          for example:
@@ -94,21 +91,19 @@ def get_service_account_info(http_request, service_account='default'):
             }
     """
     return get(
-        http_request,
+        http,
         'instance/service-accounts/{0}/'.format(service_account),
         recursive=True)
 
 
-def get_token(http_request, service_account='default'):
+def get_token(http, service_account='default'):
     """Fetch an oauth token for the
 
     Args:
+        http: an object to be used to make HTTP requests.
         service_account: An email specifying the service account this token
             should represent. Default will be a token for the "default" service
             account of the current compute engine instance.
-        http_request: A callable that matches the method
-            signature of httplib2.Http.request. Used to make the request to the
-            metadataserver.
 
     Returns:
          A tuple of (access token, token expiration), where access token is the
@@ -116,7 +111,7 @@ def get_token(http_request, service_account='default'):
          that indicates when the access token will expire.
     """
     token_json = get(
-        http_request,
+        http,
         'instance/service-accounts/{0}/token'.format(service_account))
     token_expiry = client._UTCNOW() + datetime.timedelta(
         seconds=token_json['expires_in'])

src/oauth2client/contrib/_win32_opener.py

@@ -1,106 +0,0 @@
-# Copyright 2016 Google Inc. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import errno
-import time
-
-import pywintypes
-import win32con
-import win32file
-
-from oauth2client.contrib import locked_file
-
-
-class _Win32Opener(locked_file._Opener):
-    """Open, lock, and unlock a file using windows primitives."""
-
-    # Error #33:
-    #  'The process cannot access the file because another process'
-    FILE_IN_USE_ERROR = 33
-
-    # Error #158:
-    #  'The segment is already unlocked.'
-    FILE_ALREADY_UNLOCKED_ERROR = 158
-
-    def open_and_lock(self, timeout, delay):
-        """Open the file and lock it.
-
-        Args:
-            timeout: float, How long to try to lock for.
-            delay: float, How long to wait between retries
-
-        Raises:
-            AlreadyLockedException: if the lock is already acquired.
-            IOError: if the open fails.
-            CredentialsFileSymbolicLinkError: if the file is a symbolic
-                                              link.
-        """
-        if self._locked:
-            raise locked_file.AlreadyLockedException(
-                'File {0} is already locked'.format(self._filename))
-        start_time = time.time()
-
-        locked_file.validate_file(self._filename)
-        try:
-            self._fh = open(self._filename, self._mode)
-        except IOError as e:
-            # If we can't access with _mode, try _fallback_mode
-            # and don't lock.
-            if e.errno == errno.EACCES:
-                self._fh = open(self._filename, self._fallback_mode)
-                return
-
-        # We opened in _mode, try to lock the file.
-        while True:
-            try:
-                hfile = win32file._get_osfhandle(self._fh.fileno())
-                win32file.LockFileEx(
-                    hfile,
-                    (win32con.LOCKFILE_FAIL_IMMEDIATELY |
-                     win32con.LOCKFILE_EXCLUSIVE_LOCK), 0, -0x10000,
-                    pywintypes.OVERLAPPED())
-                self._locked = True
-                return
-            except pywintypes.error as e:
-                if timeout == 0:
-                    raise
-
-                # If the error is not that the file is already
-                # in use, raise.
-                if e[0] != _Win32Opener.FILE_IN_USE_ERROR:
-                    raise
-
-                # We could not acquire the lock. Try again.
-                if (time.time() - start_time) >= timeout:
-                    locked_file.logger.warn('Could not lock %s in %s seconds',
-                                            self._filename, timeout)
-                    if self._fh:
-                        self._fh.close()
-                    self._fh = open(self._filename, self._fallback_mode)
-                    return
-                time.sleep(delay)
-
-    def unlock_and_close(self):
-        """Close and unlock the file using the win32 primitive."""
-        if self._locked:
-            try:
-                hfile = win32file._get_osfhandle(self._fh.fileno())
-                win32file.UnlockFileEx(hfile, 0, -0x10000,
-                                       pywintypes.OVERLAPPED())
-            except pywintypes.error as e:
-                if e[0] != _Win32Opener.FILE_ALREADY_UNLOCKED_ERROR:
-                    raise
-        self._locked = False
-        if self._fh:
-            self._fh.close()

src/oauth2client/contrib/appengine.py

@@ -29,13 +29,13 @@ from google.appengine.api import memcache
 from google.appengine.api import users
 from google.appengine.ext import db
 from google.appengine.ext.webapp.util import login_required
-import httplib2
 import webapp2 as webapp
 
 import oauth2client
+from oauth2client import _helpers
 from oauth2client import client
 from oauth2client import clientsecrets
-from oauth2client import util
+from oauth2client import transport
 from oauth2client.contrib import xsrfutil
 
 # This is a temporary fix for a Google internal issue.
@@ -45,8 +45,6 @@ except ImportError:  # pragma: NO COVER
     _appengine_ndb = None
 
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
-
 logger = logging.getLogger(__name__)
 
 OAUTH2CLIENT_NAMESPACE = 'oauth2client#ns'
@@ -131,7 +129,7 @@ class AppAssertionCredentials(client.AssertionCredentials):
     information to generate and refresh its own access tokens.
     """
 
-    @util.positional(2)
+    @_helpers.positional(2)
     def __init__(self, scope, **kwargs):
         """Constructor for AppAssertionCredentials
 
@@ -143,7 +141,7 @@ class AppAssertionCredentials(client.AssertionCredentials):
                                 or unspecified, the default service account for
                                 the app is used.
         """
-        self.scope = util.scopes_to_string(scope)
+        self.scope = _helpers.scopes_to_string(scope)
         self._kwargs = kwargs
         self.service_account_id = kwargs.get('service_account_id', None)
         self._service_account_email = None
@@ -157,17 +155,15 @@ class AppAssertionCredentials(client.AssertionCredentials):
         data = json.loads(json_data)
         return AppAssertionCredentials(data['scope'])
 
-    def _refresh(self, http_request):
-        """Refreshes the access_token.
+    def _refresh(self, http):
+        """Refreshes the access token.
 
         Since the underlying App Engine app_identity implementation does its
         own caching we can skip all the storage hoops and just to a refresh
         using the API.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          refresh request.
+            http: unused HTTP object
 
         Raises:
             AccessTokenRefreshError: When the refresh fails.
@@ -305,7 +301,7 @@ class StorageByKeyName(client.Storage):
     and that entities are stored by key_name.
     """
 
-    @util.positional(4)
+    @_helpers.positional(4)
     def __init__(self, model, key_name, property_name, cache=None, user=None):
         """Constructor for Storage.
 
@@ -523,7 +519,7 @@ class OAuth2Decorator(object):
 
     flow = property(get_flow, set_flow)
 
-    @util.positional(4)
+    @_helpers.positional(4)
     def __init__(self, client_id, client_secret, scope,
                  auth_uri=oauth2client.GOOGLE_AUTH_URI,
                  token_uri=oauth2client.GOOGLE_TOKEN_URI,
@@ -590,7 +586,7 @@ class OAuth2Decorator(object):
         self.credentials = None
         self._client_id = client_id
         self._client_secret = client_secret
-        self._scope = util.scopes_to_string(scope)
+        self._scope = _helpers.scopes_to_string(scope)
         self._auth_uri = auth_uri
         self._token_uri = token_uri
         self._revoke_uri = revoke_uri
@@ -742,7 +738,8 @@ class OAuth2Decorator(object):
             *args: Positional arguments passed to httplib2.Http constructor.
             **kwargs: Positional arguments passed to httplib2.Http constructor.
         """
-        return self.credentials.authorize(httplib2.Http(*args, **kwargs))
+        return self.credentials.authorize(
+            transport.get_http_object(*args, **kwargs))
 
     @property
     def callback_path(self):
@@ -804,7 +801,7 @@ class OAuth2Decorator(object):
                     if (decorator._token_response_param and
                             credentials.token_response):
                         resp_json = json.dumps(credentials.token_response)
-                        redirect_uri = util._add_query_parameter(
+                        redirect_uri = _helpers._add_query_parameter(
                             redirect_uri, decorator._token_response_param,
                             resp_json)
 
@@ -848,7 +845,7 @@ class OAuth2DecoratorFromClientSecrets(OAuth2Decorator):
 
     """
 
-    @util.positional(3)
+    @_helpers.positional(3)
     def __init__(self, filename, scope, message=None, cache=None, **kwargs):
         """Constructor
 
@@ -891,7 +888,7 @@ class OAuth2DecoratorFromClientSecrets(OAuth2Decorator):
             self._message = 'Please configure your application for OAuth 2.0.'
 
 
-@util.positional(2)
+@_helpers.positional(2)
 def oauth2decorator_from_clientsecrets(filename, scope,
                                        message=None, cache=None):
     """Creates an OAuth2Decorator populated from a clientsecrets file.

src/oauth2client/contrib/devshell.py

@@ -37,6 +37,7 @@ class CommunicationError(Error):
 class NoDevshellServer(Error):
     """Error when no Developer Shell server can be contacted."""
 
+
 # The request for credential information to the Developer Shell client socket
 # is always an empty PBLite-formatted JSON object, so just define it as a
 # constant.
@@ -117,7 +118,12 @@ class DevshellCredentials(client.GoogleCredentials):
             user_agent)
         self._refresh(None)
 
-    def _refresh(self, http_request):
+    def _refresh(self, http):
+        """Refreshes the access token.
+
+        Args:
+            http: unused HTTP object
+        """
         self.devshell_response = _SendRecv()
         self.access_token = self.devshell_response.access_token
         expires_in = self.devshell_response.expires_in

src/oauth2client/contrib/django_util/__init__.py

@@ -52,6 +52,9 @@ Add the helper to your INSTALLED_APPS:
 
 This helper also requires the Django Session Middleware, so
 ``django.contrib.sessions.middleware`` should be in INSTALLED_APPS as well.
+MIDDLEWARE or MIDDLEWARE_CLASSES (in Django  versions <1.10) should also
+contain the string 'django.contrib.sessions.middleware.SessionMiddleware'.
+
 
 Add the client secrets created earlier to the settings. You can either
 specify the path to the credentials file in JSON format
@@ -228,10 +231,10 @@ import importlib
 import django.conf
 from django.core import exceptions
 from django.core import urlresolvers
-import httplib2
 from six.moves.urllib import parse
 
 from oauth2client import clientsecrets
+from oauth2client import transport
 from oauth2client.contrib import dictionary_storage
 from oauth2client.contrib.django_util import storage
 
@@ -335,16 +338,26 @@ class OAuth2Settings(object):
         self.request_prefix = getattr(settings_instance,
                                       'GOOGLE_OAUTH2_REQUEST_ATTRIBUTE',
                                       GOOGLE_OAUTH2_REQUEST_ATTRIBUTE)
-        self.client_id, self.client_secret = \
-            _get_oauth2_client_id_and_secret(settings_instance)
+        info = _get_oauth2_client_id_and_secret(settings_instance)
+        self.client_id, self.client_secret = info
 
-        if ('django.contrib.sessions.middleware.SessionMiddleware'
-           not in settings_instance.MIDDLEWARE_CLASSES):
+        # Django 1.10 deprecated MIDDLEWARE_CLASSES in favor of MIDDLEWARE
+        middleware_settings = getattr(settings_instance, 'MIDDLEWARE', None)
+        if middleware_settings is None:
+            middleware_settings = getattr(
+                settings_instance, 'MIDDLEWARE_CLASSES', None)
+        if middleware_settings is None:
             raise exceptions.ImproperlyConfigured(
-                  'The Google OAuth2 Helper requires session middleware to '
-                  'be installed. Edit your MIDDLEWARE_CLASSES setting'
-                  ' to include \'django.contrib.sessions.middleware.'
-                  'SessionMiddleware\'.')
+                'Django settings has neither MIDDLEWARE nor MIDDLEWARE_CLASSES'
+                'configured')
+
+        if ('django.contrib.sessions.middleware.SessionMiddleware' not in
+                middleware_settings):
+            raise exceptions.ImproperlyConfigured(
+                'The Google OAuth2 Helper requires session middleware to '
+                'be installed. Edit your MIDDLEWARE_CLASSES or MIDDLEWARE '
+                'setting to include \'django.contrib.sessions.middleware.'
+                'SessionMiddleware\'.')
         (self.storage_model, self.storage_model_user_property,
          self.storage_model_credentials_property) = _get_storage_model()
 
@@ -470,8 +483,7 @@ class UserOAuth2(object):
 
     @property
     def http(self):
-        """Helper method to create an HTTP client authorized with OAuth2
-        credentials."""
+        """Helper: create HTTP client authorized with OAuth2 credentials."""
         if self.has_credentials():
-            return self.credentials.authorize(httplib2.Http())
+            return self.credentials.authorize(transport.get_http_object())
         return None

src/oauth2client/contrib/django_util/models.py

@@ -19,6 +19,7 @@ import pickle
 
 from django.db import models
 from django.utils import encoding
+import jsonpickle
 
 import oauth2client
 
@@ -48,7 +49,12 @@ class CredentialsField(models.Field):
         elif isinstance(value, oauth2client.client.Credentials):
             return value
         else:
-            return pickle.loads(base64.b64decode(encoding.smart_bytes(value)))
+            try:
+                return jsonpickle.decode(
+                    base64.b64decode(encoding.smart_bytes(value)).decode())
+            except ValueError:
+                return pickle.loads(
+                    base64.b64decode(encoding.smart_bytes(value)))
 
     def get_prep_value(self, value):
         """Overrides ``models.Field`` method. This is used to convert
@@ -58,7 +64,8 @@ class CredentialsField(models.Field):
         if value is None:
             return None
         else:
-            return encoding.smart_text(base64.b64encode(pickle.dumps(value)))
+            return encoding.smart_text(
+                base64.b64encode(jsonpickle.encode(value).encode()))
 
     def value_to_string(self, obj):
         """Convert the field value from the provided model to a string.

src/oauth2client/contrib/django_util/views.py

@@ -22,13 +22,13 @@ in the configured storage."""
 import hashlib
 import json
 import os
-import pickle
 
 from django import http
 from django import shortcuts
 from django.conf import settings
 from django.core import urlresolvers
 from django.shortcuts import redirect
+import jsonpickle
 from six.moves.urllib import parse
 
 from oauth2client import client
@@ -71,7 +71,7 @@ def _make_flow(request, scopes, return_url=None):
             urlresolvers.reverse("google_oauth:callback")))
 
     flow_key = _FLOW_KEY.format(csrf_token)
-    request.session[flow_key] = pickle.dumps(flow)
+    request.session[flow_key] = jsonpickle.encode(flow)
     return flow
 
 
@@ -89,7 +89,7 @@ def _get_flow_for_token(csrf_token, request):
         CSRF token.
     """
     flow_pickle = request.session.get(_FLOW_KEY.format(csrf_token), None)
-    return None if flow_pickle is None else pickle.loads(flow_pickle)
+    return None if flow_pickle is None else jsonpickle.decode(flow_pickle)
 
 
 def oauth2_callback(request):
@@ -170,7 +170,10 @@ def oauth2_authorize(request):
          A redirect to Google OAuth2 Authorization.
     """
     return_url = request.GET.get('return_url', None)
+    if not return_url:
+        return_url = request.META.get('HTTP_REFERER', '/')
 
+    scopes = request.GET.getlist('scopes', django_util.oauth2_settings.scopes)
     # Model storage (but not session storage) requires a logged in user
     if django_util.oauth2_settings.storage_model:
         if not request.user.is_authenticated():
@@ -178,13 +181,11 @@ def oauth2_authorize(request):
                 settings.LOGIN_URL, parse.quote(request.get_full_path())))
         # This checks for the case where we ended up here because of a logged
         # out user but we had credentials for it in the first place
-        elif get_storage(request).get() is not None:
-            return redirect(return_url)
+        else:
+            user_oauth = django_util.UserOAuth2(request, scopes, return_url)
+            if user_oauth.has_credentials():
+                return redirect(return_url)
 
-    scopes = request.GET.getlist('scopes', django_util.oauth2_settings.scopes)
-
-    if not return_url:
-        return_url = request.META.get('HTTP_REFERER', '/')
     flow = _make_flow(request=request, scopes=scopes, return_url=return_url)
     auth_url = flow.step1_get_authorize_url()
     return shortcuts.redirect(auth_url)

src/oauth2client/contrib/flask_util.py

@@ -176,19 +176,18 @@ try:
     from flask import request
     from flask import session
     from flask import url_for
+    import markupsafe
 except ImportError:  # pragma: NO COVER
     raise ImportError('The flask utilities require flask 0.9 or newer.')
 
-import httplib2
 import six.moves.http_client as httplib
 
 from oauth2client import client
 from oauth2client import clientsecrets
+from oauth2client import transport
 from oauth2client.contrib import dictionary_storage
 
 
-__author__ = 'jonwayne@google.com (Jon Wayne Parrott)'
-
 _DEFAULT_SCOPES = ('email',)
 _CREDENTIALS_KEY = 'google_oauth2_credentials'
 _FLOW_KEY = 'google_oauth2_flow_{0}'
@@ -390,6 +389,7 @@ class UserOAuth2(object):
         if 'error' in request.args:
             reason = request.args.get(
                 'error_description', request.args.get('error', ''))
+            reason = markupsafe.escape(reason)
             return ('Authorization failed: {0}'.format(reason),
                     httplib.BAD_REQUEST)
 
@@ -553,4 +553,5 @@ class UserOAuth2(object):
         """
         if not self.credentials:
             raise ValueError('No credentials available.')
-        return self.credentials.authorize(httplib2.Http(*args, **kwargs))
+        return self.credentials.authorize(
+            transport.get_http_object(*args, **kwargs))

src/oauth2client/contrib/gce.py

@@ -20,14 +20,12 @@ Utilities for making it easier to use OAuth 2.0 on Google Compute Engine.
 import logging
 import warnings
 
-import httplib2
+from six.moves import http_client
 
 from oauth2client import client
 from oauth2client.contrib import _metadata
 
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
-
 logger = logging.getLogger(__name__)
 
 _SCOPES_WARNING = """\
@@ -98,44 +96,40 @@ class AppAssertionCredentials(client.AssertionCredentials):
         Returns:
             A set of strings containing the canonical list of scopes.
         """
-        self._retrieve_info(http.request)
+        self._retrieve_info(http)
         return self.scopes
 
-    def _retrieve_info(self, http_request):
-        """Validates invalid service accounts by retrieving service account info.
+    def _retrieve_info(self, http):
+        """Retrieves service account info for invalid credentials.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make the
-                          request to the metadata server
+            http: an object to be used to make HTTP requests.
         """
         if self.invalid:
             info = _metadata.get_service_account_info(
-                http_request,
+                http,
                 service_account=self.service_account_email or 'default')
             self.invalid = False
             self.service_account_email = info['email']
             self.scopes = info['scopes']
 
-    def _refresh(self, http_request):
-        """Refreshes the access_token.
+    def _refresh(self, http):
+        """Refreshes the access token.
 
         Skip all the storage hoops and just refresh using the API.
 
         Args:
-            http_request: callable, a callable that matches the method
-                          signature of httplib2.Http.request, used to make
-                          the refresh request.
+            http: an object to be used to make HTTP requests.
 
         Raises:
             HttpAccessTokenRefreshError: When the refresh fails.
         """
         try:
-            self._retrieve_info(http_request)
+            self._retrieve_info(http)
             self.access_token, self.token_expiry = _metadata.get_token(
-                http_request, service_account=self.service_account_email)
-        except httplib2.HttpLib2Error as e:
-            raise client.HttpAccessTokenRefreshError(str(e))
+                http, service_account=self.service_account_email)
+        except http_client.HTTPException as err:
+            raise client.HttpAccessTokenRefreshError(str(err))
 
     @property
     def serialization_data(self):

src/oauth2client/contrib/keyring_storage.py

@@ -24,9 +24,6 @@ import keyring
 from oauth2client import client
 
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
-
-
 class Storage(client.Storage):
     """Store and retrieve a single credential to and from the keyring.
 

src/oauth2client/contrib/locked_file.py

@@ -1,234 +0,0 @@
-# Copyright 2014 Google Inc. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Locked file interface that should work on Unix and Windows pythons.
-
-This module first tries to use fcntl locking to ensure serialized access
-to a file, then falls back on a lock file if that is unavialable.
-
-Usage::
-
-    f = LockedFile('filename', 'r+b', 'rb')
-    f.open_and_lock()
-    if f.is_locked():
-      print('Acquired filename with r+b mode')
-      f.file_handle().write('locked data')
-    else:
-      print('Acquired filename with rb mode')
-    f.unlock_and_close()
-
-"""
-
-from __future__ import print_function
-
-import errno
-import logging
-import os
-import time
-
-from oauth2client import util
-
-
-__author__ = 'cache@google.com (David T McWherter)'
-
-logger = logging.getLogger(__name__)
-
-
-class CredentialsFileSymbolicLinkError(Exception):
-    """Credentials files must not be symbolic links."""
-
-
-class AlreadyLockedException(Exception):
-    """Trying to lock a file that has already been locked by the LockedFile."""
-    pass
-
-
-def validate_file(filename):
-    if os.path.islink(filename):
-        raise CredentialsFileSymbolicLinkError(
-            'File: {0} is a symbolic link.'.format(filename))
-
-
-class _Opener(object):
-    """Base class for different locking primitives."""
-
-    def __init__(self, filename, mode, fallback_mode):
-        """Create an Opener.
-
-        Args:
-            filename: string, The pathname of the file.
-            mode: string, The preferred mode to access the file with.
-            fallback_mode: string, The mode to use if locking fails.
-        """
-        self._locked = False
-        self._filename = filename
-        self._mode = mode
-        self._fallback_mode = fallback_mode
-        self._fh = None
-        self._lock_fd = None
-
-    def is_locked(self):
-        """Was the file locked."""
-        return self._locked
-
-    def file_handle(self):
-        """The file handle to the file. Valid only after opened."""
-        return self._fh
-
-    def filename(self):
-        """The filename that is being locked."""
-        return self._filename
-
-    def open_and_lock(self, timeout, delay):
-        """Open the file and lock it.
-
-        Args:
-            timeout: float, How long to try to lock for.
-            delay: float, How long to wait between retries.
-        """
-        pass
-
-    def unlock_and_close(self):
-        """Unlock and close the file."""
-        pass
-
-
-class _PosixOpener(_Opener):
-    """Lock files using Posix advisory lock files."""
-
-    def open_and_lock(self, timeout, delay):
-        """Open the file and lock it.
-
-        Tries to create a .lock file next to the file we're trying to open.
-
-        Args:
-            timeout: float, How long to try to lock for.
-            delay: float, How long to wait between retries.
-
-        Raises:
-            AlreadyLockedException: if the lock is already acquired.
-            IOError: if the open fails.
-            CredentialsFileSymbolicLinkError if the file is a symbolic link.
-        """
-        if self._locked:
-            raise AlreadyLockedException(
-                'File {0} is already locked'.format(self._filename))
-        self._locked = False
-
-        validate_file(self._filename)
-        try:
-            self._fh = open(self._filename, self._mode)
-        except IOError as e:
-            # If we can't access with _mode, try _fallback_mode and don't lock.
-            if e.errno == errno.EACCES:
-                self._fh = open(self._filename, self._fallback_mode)
-                return
-
-        lock_filename = self._posix_lockfile(self._filename)
-        start_time = time.time()
-        while True:
-            try:
-                self._lock_fd = os.open(lock_filename,
-                                        os.O_CREAT | os.O_EXCL | os.O_RDWR)
-                self._locked = True
-                break
-
-            except OSError as e:
-                if e.errno != errno.EEXIST:
-                    raise
-                if (time.time() - start_time) >= timeout:
-                    logger.warn('Could not acquire lock %s in %s seconds',
-                                lock_filename, timeout)
-                    # Close the file and open in fallback_mode.
-                    if self._fh:
-                        self._fh.close()
-                    self._fh = open(self._filename, self._fallback_mode)
-                    return
-                time.sleep(delay)
-
-    def unlock_and_close(self):
-        """Unlock a file by removing the .lock file, and close the handle."""
-        if self._locked:
-            lock_filename = self._posix_lockfile(self._filename)
-            os.close(self._lock_fd)
-            os.unlink(lock_filename)
-            self._locked = False
-            self._lock_fd = None
-        if self._fh:
-            self._fh.close()
-
-    def _posix_lockfile(self, filename):
-        """The name of the lock file to use for posix locking."""
-        return '{0}.lock'.format(filename)
-
-
-class LockedFile(object):
-    """Represent a file that has exclusive access."""
-
-    @util.positional(4)
-    def __init__(self, filename, mode, fallback_mode, use_native_locking=True):
-        """Construct a LockedFile.
-
-        Args:
-            filename: string, The path of the file to open.
-            mode: string, The mode to try to open the file with.
-            fallback_mode: string, The mode to use if locking fails.
-            use_native_locking: bool, Whether or not fcntl/win32 locking is
-                                used.
-        """
-        opener = None
-        if not opener and use_native_locking:
-            try:
-                from oauth2client.contrib._win32_opener import _Win32Opener
-                opener = _Win32Opener(filename, mode, fallback_mode)
-            except ImportError:
-                try:
-                    from oauth2client.contrib._fcntl_opener import _FcntlOpener
-                    opener = _FcntlOpener(filename, mode, fallback_mode)
-                except ImportError:
-                    pass
-
-        if not opener:
-            opener = _PosixOpener(filename, mode, fallback_mode)
-
-        self._opener = opener
-
-    def filename(self):
-        """Return the filename we were constructed with."""
-        return self._opener._filename
-
-    def file_handle(self):
-        """Return the file_handle to the opened file."""
-        return self._opener.file_handle()
-
-    def is_locked(self):
-        """Return whether we successfully locked the file."""
-        return self._opener.is_locked()
-
-    def open_and_lock(self, timeout=0, delay=0.05):
-        """Open the file, trying to lock it.
-
-        Args:
-            timeout: float, The number of seconds to try to acquire the lock.
-            delay: float, The number of seconds to wait between retry attempts.
-
-        Raises:
-            AlreadyLockedException: if the lock is already acquired.
-            IOError: if the open fails.
-        """
-        self._opener.open_and_lock(timeout, delay)
-
-    def unlock_and_close(self):
-        """Unlock and close a file."""
-        self._opener.unlock_and_close()

src/oauth2client/contrib/multistore_file.py

@@ -1,505 +0,0 @@
-# Copyright 2014 Google Inc. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Multi-credential file store with lock support.
-
-This module implements a JSON credential store where multiple
-credentials can be stored in one file. That file supports locking
-both in a single process and across processes.
-
-The credential themselves are keyed off of:
-
-* client_id
-* user_agent
-* scope
-
-The format of the stored data is like so::
-
-    {
-      'file_version': 1,
-      'data': [
-          {
-              'key': {
-                  'clientId': '<client id>',
-                  'userAgent': '<user agent>',
-                  'scope': '<scope>'
-              },
-              'credential': {
-                  # JSON serialized Credentials.
-              }
-          }
-      ]
-    }
-
-"""
-
-import errno
-import json
-import logging
-import os
-import threading
-
-from oauth2client import client
-from oauth2client import util
-from oauth2client.contrib import locked_file
-
-__author__ = 'jbeda@google.com (Joe Beda)'
-
-logger = logging.getLogger(__name__)
-
-logger.warning(
-    'The oauth2client.contrib.multistore_file module has been deprecated and '
-    'will be removed in the next release of oauth2client. Please migrate to '
-    'multiprocess_file_storage.')
-
-# A dict from 'filename'->_MultiStore instances
-_multistores = {}
-_multistores_lock = threading.Lock()
-
-
-class Error(Exception):
-    """Base error for this module."""
-
-
-class NewerCredentialStoreError(Error):
-    """The credential store is a newer version than supported."""
-
-
-def _dict_to_tuple_key(dictionary):
-    """Converts a dictionary to a tuple that can be used as an immutable key.
-
-    The resulting key is always sorted so that logically equivalent
-    dictionaries always produce an identical tuple for a key.
-
-    Args:
-        dictionary: the dictionary to use as the key.
-
-    Returns:
-        A tuple representing the dictionary in it's naturally sorted ordering.
-    """
-    return tuple(sorted(dictionary.items()))
-
-
-@util.positional(4)
-def get_credential_storage(filename, client_id, user_agent, scope,
-                           warn_on_readonly=True):
-    """Get a Storage instance for a credential.
-
-    Args:
-        filename: The JSON file storing a set of credentials
-        client_id: The client_id for the credential
-        user_agent: The user agent for the credential
-        scope: string or iterable of strings, Scope(s) being requested
-        warn_on_readonly: if True, log a warning if the store is readonly
-
-    Returns:
-        An object derived from client.Storage for getting/setting the
-        credential.
-    """
-    # Recreate the legacy key with these specific parameters
-    key = {'clientId': client_id, 'userAgent': user_agent,
-           'scope': util.scopes_to_string(scope)}
-    return get_credential_storage_custom_key(
-        filename, key, warn_on_readonly=warn_on_readonly)
-
-
-@util.positional(2)
-def get_credential_storage_custom_string_key(filename, key_string,
-                                             warn_on_readonly=True):
-    """Get a Storage instance for a credential using a single string as a key.
-
-    Allows you to provide a string as a custom key that will be used for
-    credential storage and retrieval.
-
-    Args:
-        filename: The JSON file storing a set of credentials
-        key_string: A string to use as the key for storing this credential.
-        warn_on_readonly: if True, log a warning if the store is readonly
-
-    Returns:
-        An object derived from client.Storage for getting/setting the
-        credential.
-    """
-    # Create a key dictionary that can be used
-    key_dict = {'key': key_string}
-    return get_credential_storage_custom_key(
-        filename, key_dict, warn_on_readonly=warn_on_readonly)
-
-
-@util.positional(2)
-def get_credential_storage_custom_key(filename, key_dict,
-                                      warn_on_readonly=True):
-    """Get a Storage instance for a credential using a dictionary as a key.
-
-    Allows you to provide a dictionary as a custom key that will be used for
-    credential storage and retrieval.
-
-    Args:
-        filename: The JSON file storing a set of credentials
-        key_dict: A dictionary to use as the key for storing this credential.
-                  There is no ordering of the keys in the dictionary. Logically
-                  equivalent dictionaries will produce equivalent storage keys.
-        warn_on_readonly: if True, log a warning if the store is readonly
-
-    Returns:
-        An object derived from client.Storage for getting/setting the
-        credential.
-    """
-    multistore = _get_multistore(filename, warn_on_readonly=warn_on_readonly)
-    key = _dict_to_tuple_key(key_dict)
-    return multistore._get_storage(key)
-
-
-@util.positional(1)
-def get_all_credential_keys(filename, warn_on_readonly=True):
-    """Gets all the registered credential keys in the given Multistore.
-
-    Args:
-        filename: The JSON file storing a set of credentials
-        warn_on_readonly: if True, log a warning if the store is readonly
-
-    Returns:
-        A list of the credential keys present in the file.  They are returned
-        as dictionaries that can be passed into
-        get_credential_storage_custom_key to get the actual credentials.
-    """
-    multistore = _get_multistore(filename, warn_on_readonly=warn_on_readonly)
-    multistore._lock()
-    try:
-        return multistore._get_all_credential_keys()
-    finally:
-        multistore._unlock()
-
-
-@util.positional(1)
-def _get_multistore(filename, warn_on_readonly=True):
-    """A helper method to initialize the multistore with proper locking.
-
-    Args:
-        filename: The JSON file storing a set of credentials
-        warn_on_readonly: if True, log a warning if the store is readonly
-
-    Returns:
-        A multistore object
-    """
-    filename = os.path.expanduser(filename)
-    _multistores_lock.acquire()
-    try:
-        multistore = _multistores.setdefault(
-            filename, _MultiStore(filename, warn_on_readonly=warn_on_readonly))
-    finally:
-        _multistores_lock.release()
-    return multistore
-
-
-class _MultiStore(object):
-    """A file backed store for multiple credentials."""
-
-    @util.positional(2)
-    def __init__(self, filename, warn_on_readonly=True):
-        """Initialize the class.
-
-        This will create the file if necessary.
-        """
-        self._file = locked_file.LockedFile(filename, 'r+', 'r')
-        self._thread_lock = threading.Lock()
-        self._read_only = False
-        self._warn_on_readonly = warn_on_readonly
-
-        self._create_file_if_needed()
-
-        # Cache of deserialized store. This is only valid after the
-        # _MultiStore is locked or _refresh_data_cache is called. This is
-        # of the form of:
-        #
-        # ((key, value), (key, value)...) -> OAuth2Credential
-        #
-        # If this is None, then the store hasn't been read yet.
-        self._data = None
-
-    class _Storage(client.Storage):
-        """A Storage object that can read/write a single credential."""
-
-        def __init__(self, multistore, key):
-            self._multistore = multistore
-            self._key = key
-
-        def acquire_lock(self):
-            """Acquires any lock necessary to access this Storage.
-
-            This lock is not reentrant.
-            """
-            self._multistore._lock()
-
-        def release_lock(self):
-            """Release the Storage lock.
-
-            Trying to release a lock that isn't held will result in a
-            RuntimeError.
-            """
-            self._multistore._unlock()
-
-        def locked_get(self):
-            """Retrieve credential.
-
-            The Storage lock must be held when this is called.
-
-            Returns:
-                oauth2client.client.Credentials
-            """
-            credential = self._multistore._get_credential(self._key)
-            if credential:
-                credential.set_store(self)
-            return credential
-
-        def locked_put(self, credentials):
-            """Write a credential.
-
-            The Storage lock must be held when this is called.
-
-            Args:
-                credentials: Credentials, the credentials to store.
-            """
-            self._multistore._update_credential(self._key, credentials)
-
-        def locked_delete(self):
-            """Delete a credential.
-
-            The Storage lock must be held when this is called.
-
-            Args:
-                credentials: Credentials, the credentials to store.
-            """
-            self._multistore._delete_credential(self._key)
-
-    def _create_file_if_needed(self):
-        """Create an empty file if necessary.
-
-        This method will not initialize the file. Instead it implements a
-        simple version of "touch" to ensure the file has been created.
-        """
-        if not os.path.exists(self._file.filename()):
-            old_umask = os.umask(0o177)
-            try:
-                open(self._file.filename(), 'a+b').close()
-            finally:
-                os.umask(old_umask)
-
-    def _lock(self):
-        """Lock the entire multistore."""
-        self._thread_lock.acquire()
-        try:
-            self._file.open_and_lock()
-        except (IOError, OSError) as e:
-            if e.errno == errno.ENOSYS:
-                logger.warn('File system does not support locking the '
-                            'credentials file.')
-            elif e.errno == errno.ENOLCK:
-                logger.warn('File system is out of resources for writing the '
-                            'credentials file (is your disk full?).')
-            elif e.errno == errno.EDEADLK:
-                logger.warn('Lock contention on multistore file, opening '
-                            'in read-only mode.')
-            elif e.errno == errno.EACCES:
-                logger.warn('Cannot access credentials file.')
-            else:
-                raise
-        if not self._file.is_locked():
-            self._read_only = True
-            if self._warn_on_readonly:
-                logger.warn('The credentials file (%s) is not writable. '
-                            'Opening in read-only mode. Any refreshed '
-                            'credentials will only be '
-                            'valid for this run.', self._file.filename())
-
-        if os.path.getsize(self._file.filename()) == 0:
-            logger.debug('Initializing empty multistore file')
-            # The multistore is empty so write out an empty file.
-            self._data = {}
-            self._write()
-        elif not self._read_only or self._data is None:
-            # Only refresh the data if we are read/write or we haven't
-            # cached the data yet. If we are readonly, we assume is isn't
-            # changing out from under us and that we only have to read it
-            # once. This prevents us from whacking any new access keys that
-            # we have cached in memory but were unable to write out.
-            self._refresh_data_cache()
-
-    def _unlock(self):
-        """Release the lock on the multistore."""
-        self._file.unlock_and_close()
-        self._thread_lock.release()
-
-    def _locked_json_read(self):
-        """Get the raw content of the multistore file.
-
-        The multistore must be locked when this is called.
-
-        Returns:
-            The contents of the multistore decoded as JSON.
-        """
-        assert self._thread_lock.locked()
-        self._file.file_handle().seek(0)
-        return json.load(self._file.file_handle())
-
-    def _locked_json_write(self, data):
-        """Write a JSON serializable data structure to the multistore.
-
-        The multistore must be locked when this is called.
-
-        Args:
-            data: The data to be serialized and written.
-        """
-        assert self._thread_lock.locked()
-        if self._read_only:
-            return
-        self._file.file_handle().seek(0)
-        json.dump(data, self._file.file_handle(),
-                  sort_keys=True, indent=2, separators=(',', ': '))
-        self._file.file_handle().truncate()
-
-    def _refresh_data_cache(self):
-        """Refresh the contents of the multistore.
-
-        The multistore must be locked when this is called.
-
-        Raises:
-            NewerCredentialStoreError: Raised when a newer client has written
-            the store.
-        """
-        self._data = {}
-        try:
-            raw_data = self._locked_json_read()
-        except Exception:
-            logger.warn('Credential data store could not be loaded. '
-                        'Will ignore and overwrite.')
-            return
-
-        version = 0
-        try:
-            version = raw_data['file_version']
-        except Exception:
-            logger.warn('Missing version for credential data store. It may be '
-                        'corrupt or an old version. Overwriting.')
-        if version > 1:
-            raise NewerCredentialStoreError(
-                'Credential file has file_version of {0}. '
-                'Only file_version of 1 is supported.'.format(version))
-
-        credentials = []
-        try:
-            credentials = raw_data['data']
-        except (TypeError, KeyError):
-            pass
-
-        for cred_entry in credentials:
-            try:
-                key, credential = self._decode_credential_from_json(cred_entry)
-                self._data[key] = credential
-            except:
-                # If something goes wrong loading a credential, just ignore it
-                logger.info('Error decoding credential, skipping',
-                            exc_info=True)
-
-    def _decode_credential_from_json(self, cred_entry):
-        """Load a credential from our JSON serialization.
-
-        Args:
-            cred_entry: A dict entry from the data member of our format
-
-        Returns:
-            (key, cred) where the key is the key tuple and the cred is the
-            OAuth2Credential object.
-        """
-        raw_key = cred_entry['key']
-        key = _dict_to_tuple_key(raw_key)
-        credential = None
-        credential = client.Credentials.new_from_json(
-            json.dumps(cred_entry['credential']))
-        return (key, credential)
-
-    def _write(self):
-        """Write the cached data back out.
-
-        The multistore must be locked.
-        """
-        raw_data = {'file_version': 1}
-        raw_creds = []
-        raw_data['data'] = raw_creds
-        for (cred_key, cred) in self._data.items():
-            raw_key = dict(cred_key)
-            raw_cred = json.loads(cred.to_json())
-            raw_creds.append({'key': raw_key, 'credential': raw_cred})
-        self._locked_json_write(raw_data)
-
-    def _get_all_credential_keys(self):
-        """Gets all the registered credential keys in the multistore.
-
-        Returns:
-            A list of dictionaries corresponding to all the keys currently
-            registered
-        """
-        return [dict(key) for key in self._data.keys()]
-
-    def _get_credential(self, key):
-        """Get a credential from the multistore.
-
-        The multistore must be locked.
-
-        Args:
-            key: The key used to retrieve the credential
-
-        Returns:
-            The credential specified or None if not present
-        """
-        return self._data.get(key, None)
-
-    def _update_credential(self, key, cred):
-        """Update a credential and write the multistore.
-
-        This must be called when the multistore is locked.
-
-        Args:
-            key: The key used to retrieve the credential
-            cred: The OAuth2Credential to update/set
-        """
-        self._data[key] = cred
-        self._write()
-
-    def _delete_credential(self, key):
-        """Delete a credential and write the multistore.
-
-        This must be called when the multistore is locked.
-
-        Args:
-            key: The key used to retrieve the credential
-        """
-        try:
-            del self._data[key]
-        except KeyError:
-            pass
-        self._write()
-
-    def _get_storage(self, key):
-        """Get a Storage object to get/set a credential.
-
-        This Storage is a 'view' into the multistore.
-
-        Args:
-            key: The key used to retrieve the credential
-
-        Returns:
-            A Storage object that can be used to get/set this cred
-        """
-        return self._Storage(self, key)

src/oauth2client/contrib/xsrfutil.py

@@ -20,12 +20,7 @@ import hmac
 import time
 
 from oauth2client import _helpers
-from oauth2client import util
 
-__authors__ = [
-    '"Doug Coker" <dcoker@google.com>',
-    '"Joe Gregorio" <jcgregorio@google.com>',
-]
 
 # Delimiter character
 DELIMITER = b':'
@@ -34,7 +29,7 @@ DELIMITER = b':'
 DEFAULT_TIMEOUT_SECS = 60 * 60
 
 
-@util.positional(2)
+@_helpers.positional(2)
 def generate_token(key, user_id, action_id='', when=None):
     """Generates a URL-safe token for the given user, action, time tuple.
 
@@ -62,7 +57,7 @@ def generate_token(key, user_id, action_id='', when=None):
     return token
 
 
-@util.positional(3)
+@_helpers.positional(3)
 def validate_token(key, token, user_id, action_id="", current_time=None):
     """Validates that the given token authorizes the user for the action.
 

src/oauth2client/file.py

@@ -21,16 +21,10 @@ credentials.
 import os
 import threading
 
+from oauth2client import _helpers
 from oauth2client import client
 
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
-
-
-class CredentialsFileSymbolicLinkError(Exception):
-    """Credentials files must not be symbolic links."""
-
-
 class Storage(client.Storage):
     """Store and retrieve a single credential to and from a file."""
 
@@ -38,11 +32,6 @@ class Storage(client.Storage):
         super(Storage, self).__init__(lock=threading.Lock())
         self._filename = filename
 
-    def _validate_file(self):
-        if os.path.islink(self._filename):
-            raise CredentialsFileSymbolicLinkError(
-                'File: {0} is a symbolic link.'.format(self._filename))
-
     def locked_get(self):
         """Retrieve Credential from file.
 
@@ -50,10 +39,10 @@ class Storage(client.Storage):
             oauth2client.client.Credentials
 
         Raises:
-            CredentialsFileSymbolicLinkError if the file is a symbolic link.
+            IOError if the file is a symbolic link.
         """
         credentials = None
-        self._validate_file()
+        _helpers.validate_file(self._filename)
         try:
             f = open(self._filename, 'rb')
             content = f.read()
@@ -89,10 +78,10 @@ class Storage(client.Storage):
             credentials: Credentials, the credentials to store.
 
         Raises:
-            CredentialsFileSymbolicLinkError if the file is a symbolic link.
+            IOError if the file is a symbolic link.
         """
         self._create_file_if_needed()
-        self._validate_file()
+        _helpers.validate_file(self._filename)
         f = open(self._filename, 'w')
         f.write(credentials.to_json())
         f.close()

src/oauth2client/service_account.py

@@ -25,7 +25,6 @@ from oauth2client import _helpers
 from oauth2client import client
 from oauth2client import crypt
 from oauth2client import transport
-from oauth2client import util
 
 
 _PASSWORD_DEFAULT = 'notasecret'
@@ -110,7 +109,7 @@ class ServiceAccountCredentials(client.AssertionCredentials):
 
         self._service_account_email = service_account_email
         self._signer = signer
-        self._scopes = util.scopes_to_string(scopes)
+        self._scopes = _helpers.scopes_to_string(scopes)
         self._private_key_id = private_key_id
         self.client_id = client_id
         self._user_agent = user_agent
@@ -650,9 +649,22 @@ class _JWTAccessCredentials(ServiceAccountCredentials):
         return result
 
     def refresh(self, http):
+        """Refreshes the access_token.
+
+        The HTTP object is unused since no request needs to be made to
+        get a new token, it can just be generated locally.
+
+        Args:
+            http: unused HTTP object
+        """
         self._refresh(None)
 
-    def _refresh(self, http_request):
+    def _refresh(self, http):
+        """Refreshes the access_token.
+
+        Args:
+            http: unused HTTP object
+        """
         self.access_token, self.token_expiry = self._create_token()
 
     def _create_token(self, additional_claims=None):

src/oauth2client/tools.py

@@ -30,11 +30,10 @@ from six.moves import http_client
 from six.moves import input
 from six.moves import urllib
 
+from oauth2client import _helpers
 from oauth2client import client
-from oauth2client import util
 
 
-__author__ = 'jcgregorio@google.com (Joe Gregorio)'
 __all__ = ['argparser', 'run_flow', 'message_if_missing']
 
 _CLIENT_SECRETS_MESSAGE = """WARNING: Please configure OAuth 2.0
@@ -93,6 +92,7 @@ def _CreateArgumentParser():
         help='Set the logging level of detail.')
     return parser
 
+
 # argparser is an ArgumentParser that contains command-line options expected
 # by tools.run(). Pass it in as part of the 'parents' argument to your own
 # ArgumentParser.
@@ -123,22 +123,22 @@ class ClientRedirectHandler(BaseHTTPServer.BaseHTTPRequestHandler):
         if an error occurred.
         """
         self.send_response(http_client.OK)
-        self.send_header("Content-type", "text/html")
+        self.send_header('Content-type', 'text/html')
         self.end_headers()
-        query = self.path.split('?', 1)[-1]
-        query = dict(urllib.parse.parse_qsl(query))
+        parts = urllib.parse.urlparse(self.path)
+        query = _helpers.parse_unique_urlencoded(parts.query)
         self.server.query_params = query
         self.wfile.write(
-            b"<html><head><title>Authentication Status</title></head>")
+            b'<html><head><title>Authentication Status</title></head>')
         self.wfile.write(
-            b"<body><p>The authentication flow has completed.</p>")
-        self.wfile.write(b"</body></html>")
+            b'<body><p>The authentication flow has completed.</p>')
+        self.wfile.write(b'</body></html>')
 
     def log_message(self, format, *args):
         """Do not log messages to stdout while running as cmd. line program."""
 
 
-@util.positional(3)
+@_helpers.positional(3)
 def run_flow(flow, storage, flags=None, http=None):
     """Core code for a command-line application.
 
@@ -218,16 +218,6 @@ def run_flow(flow, storage, flags=None, http=None):
     flow.redirect_uri = oauth_callback
     authorize_url = flow.step1_get_authorize_url()
 
-    if flags.short_url:
-        try:
-            from googleapiclient.discovery import build
-            service = build('urlshortener', 'v1', http=http)
-            url_result = service.url().insert(body={'longUrl': authorize_url},
-              key=u'AIzaSyBlmgbii8QfJSYmC9VTMOfqrAt5Vj5wtzE').execute()
-            authorize_url = url_result['id']
-        except:
-          pass
-
     if not flags.noauth_local_webserver:
         import webbrowser
         webbrowser.open(authorize_url, new=1, autoraise=True)

src/oauth2client/transport.py

@@ -18,7 +18,7 @@ import httplib2
 import six
 from six.moves import http_client
 
-from oauth2client._helpers import _to_bytes
+from oauth2client import _helpers
 
 
 _LOGGER = logging.getLogger(__name__)
@@ -58,13 +58,19 @@ def get_cached_http():
     return _CACHED_HTTP
 
 
-def get_http_object():
+def get_http_object(*args, **kwargs):
     """Return a new HTTP object.
 
+    Args:
+        *args: tuple, The positional arguments to be passed when
+               contructing a new HTTP object.
+        **kwargs: dict, The keyword arguments to be passed when
+                  contructing a new HTTP object.
+
     Returns:
         httplib2.Http, an HTTP object.
     """
-    return httplib2.Http()
+    return httplib2.Http(*args, **kwargs)
 
 
 def _initialize_headers(headers):
@@ -121,7 +127,7 @@ def clean_headers(headers):
                 k = str(k)
             if not isinstance(v, six.binary_type):
                 v = str(v)
-            clean[_to_bytes(k)] = _to_bytes(v)
+            clean[_helpers._to_bytes(k)] = _helpers._to_bytes(v)
     except UnicodeEncodeError:
         from oauth2client.client import NonAsciiHeaderError
         raise NonAsciiHeaderError(k, ': ', v)
@@ -164,9 +170,9 @@ def wrap_http_for_auth(credentials, http):
                _STREAM_PROPERTIES):
             body_stream_position = body.tell()
 
-        resp, content = orig_request_method(uri, method, body,
-                                            clean_headers(headers),
-                                            redirections, connection_type)
+        resp, content = request(orig_request_method, uri, method, body,
+                                clean_headers(headers),
+                                redirections, connection_type)
 
         # A stored token may expire between the time it is retrieved and
         # the time the request is made, so we may need to try twice.
@@ -182,9 +188,9 @@ def wrap_http_for_auth(credentials, http):
             if body_stream_position is not None:
                 body.seek(body_stream_position)
 
-            resp, content = orig_request_method(uri, method, body,
-                                                clean_headers(headers),
-                                                redirections, connection_type)
+            resp, content = request(orig_request_method, uri, method, body,
+                                    clean_headers(headers),
+                                    redirections, connection_type)
 
         return resp, content
 
@@ -192,7 +198,7 @@ def wrap_http_for_auth(credentials, http):
     http.request = new_request
 
     # Set credentials as a property of the request method.
-    setattr(http.request, 'credentials', credentials)
+    http.request.credentials = credentials
 
 
 def wrap_http_for_jwt_access(credentials, http):
@@ -222,9 +228,9 @@ def wrap_http_for_jwt_access(credentials, http):
             if (credentials.access_token is None or
                     credentials.access_token_expired):
                 credentials.refresh(None)
-            return authenticated_request_method(uri, method, body,
-                                                headers, redirections,
-                                                connection_type)
+            return request(authenticated_request_method, uri,
+                           method, body, headers, redirections,
+                           connection_type)
         else:
             # If we don't have an 'aud' (audience) claim,
             # create a 1-time token with the uri root as the audience
@@ -234,12 +240,46 @@ def wrap_http_for_jwt_access(credentials, http):
             token, unused_expiry = credentials._create_token({'aud': uri_root})
 
             headers['Authorization'] = 'Bearer ' + token
-            return orig_request_method(uri, method, body,
-                                       clean_headers(headers),
-                                       redirections, connection_type)
+            return request(orig_request_method, uri, method, body,
+                           clean_headers(headers),
+                           redirections, connection_type)
 
     # Replace the request method with our own closure.
     http.request = new_request
 
+    # Set credentials as a property of the request method.
+    http.request.credentials = credentials
+
+
+def request(http, uri, method='GET', body=None, headers=None,
+            redirections=httplib2.DEFAULT_MAX_REDIRECTS,
+            connection_type=None):
+    """Make an HTTP request with an HTTP object and arguments.
+
+    Args:
+        http: httplib2.Http, an http object to be used to make requests.
+        uri: string, The URI to be requested.
+        method: string, The HTTP method to use for the request. Defaults
+                to 'GET'.
+        body: string, The payload / body in HTTP request. By default
+              there is no payload.
+        headers: dict, Key-value pairs of request headers. By default
+                 there are no headers.
+        redirections: int, The number of allowed 203 redirects for
+                      the request. Defaults to 5.
+        connection_type: httplib.HTTPConnection, a subclass to be used for
+                         establishing connection. If not set, the type
+                         will be determined from the ``uri``.
+
+    Returns:
+        tuple, a pair of a httplib2.Response with the status code and other
+        headers and the bytes of the content returned.
+    """
+    # NOTE: Allowing http or http.request is temporary (See Issue 601).
+    http_callable = getattr(http, 'request', http)
+    return http_callable(uri, method=method, body=body, headers=headers,
+                         redirections=redirections,
+                         connection_type=connection_type)
+
 
 _CACHED_HTTP = httplib2.Http(MemoryCache())

src/oauth2client/util.py

@@ -1,206 +0,0 @@
-# Copyright 2014 Google Inc. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Common utility library."""
-
-import functools
-import inspect
-import logging
-
-import six
-from six.moves import urllib
-
-
-__author__ = [
-    'rafek@google.com (Rafe Kaplan)',
-    'guido@google.com (Guido van Rossum)',
-]
-
-__all__ = [
-    'positional',
-    'POSITIONAL_WARNING',
-    'POSITIONAL_EXCEPTION',
-    'POSITIONAL_IGNORE',
-]
-
-logger = logging.getLogger(__name__)
-
-POSITIONAL_WARNING = 'WARNING'
-POSITIONAL_EXCEPTION = 'EXCEPTION'
-POSITIONAL_IGNORE = 'IGNORE'
-POSITIONAL_SET = frozenset([POSITIONAL_WARNING, POSITIONAL_EXCEPTION,
-                            POSITIONAL_IGNORE])
-
-positional_parameters_enforcement = POSITIONAL_WARNING
-
-
-def positional(max_positional_args):
-    """A decorator to declare that only the first N arguments my be positional.
-
-    This decorator makes it easy to support Python 3 style keyword-only
-    parameters. For example, in Python 3 it is possible to write::
-
-        def fn(pos1, *, kwonly1=None, kwonly1=None):
-            ...
-
-    All named parameters after ``*`` must be a keyword::
-
-        fn(10, 'kw1', 'kw2')  # Raises exception.
-        fn(10, kwonly1='kw1')  # Ok.
-
-    Example
-    ^^^^^^^
-
-    To define a function like above, do::
-
-        @positional(1)
-        def fn(pos1, kwonly1=None, kwonly2=None):
-            ...
-
-    If no default value is provided to a keyword argument, it becomes a
-    required keyword argument::
-
-        @positional(0)
-        def fn(required_kw):
-            ...
-
-    This must be called with the keyword parameter::
-
-        fn()  # Raises exception.
-        fn(10)  # Raises exception.
-        fn(required_kw=10)  # Ok.
-
-    When defining instance or class methods always remember to account for
-    ``self`` and ``cls``::
-
-        class MyClass(object):
-
-            @positional(2)
-            def my_method(self, pos1, kwonly1=None):
-                ...
-
-            @classmethod
-            @positional(2)
-            def my_method(cls, pos1, kwonly1=None):
-                ...
-
-    The positional decorator behavior is controlled by
-    ``util.positional_parameters_enforcement``, which may be set to
-    ``POSITIONAL_EXCEPTION``, ``POSITIONAL_WARNING`` or
-    ``POSITIONAL_IGNORE`` to raise an exception, log a warning, or do
-    nothing, respectively, if a declaration is violated.
-
-    Args:
-        max_positional_arguments: Maximum number of positional arguments. All
-                                  parameters after the this index must be
-                                  keyword only.
-
-    Returns:
-        A decorator that prevents using arguments after max_positional_args
-        from being used as positional parameters.
-
-    Raises:
-        TypeError: if a key-word only argument is provided as a positional
-                   parameter, but only if
-                   util.positional_parameters_enforcement is set to
-                   POSITIONAL_EXCEPTION.
-    """
-
-    def positional_decorator(wrapped):
-        @functools.wraps(wrapped)
-        def positional_wrapper(*args, **kwargs):
-            if len(args) > max_positional_args:
-                plural_s = ''
-                if max_positional_args != 1:
-                    plural_s = 's'
-                message = ('{function}() takes at most {args_max} positional '
-                           'argument{plural} ({args_given} given)'.format(
-                               function=wrapped.__name__,
-                               args_max=max_positional_args,
-                               args_given=len(args),
-                               plural=plural_s))
-                if positional_parameters_enforcement == POSITIONAL_EXCEPTION:
-                    raise TypeError(message)
-                elif positional_parameters_enforcement == POSITIONAL_WARNING:
-                    logger.warning(message)
-            return wrapped(*args, **kwargs)
-        return positional_wrapper
-
-    if isinstance(max_positional_args, six.integer_types):
-        return positional_decorator
-    else:
-        args, _, _, defaults = inspect.getargspec(max_positional_args)
-        return positional(len(args) - len(defaults))(max_positional_args)
-
-
-def scopes_to_string(scopes):
-    """Converts scope value to a string.
-
-    If scopes is a string then it is simply passed through. If scopes is an
-    iterable then a string is returned that is all the individual scopes
-    concatenated with spaces.
-
-    Args:
-        scopes: string or iterable of strings, the scopes.
-
-    Returns:
-        The scopes formatted as a single string.
-    """
-    if isinstance(scopes, six.string_types):
-        return scopes
-    else:
-        return ' '.join(scopes)
-
-
-def string_to_scopes(scopes):
-    """Converts stringifed scope value to a list.
-
-    If scopes is a list then it is simply passed through. If scopes is an
-    string then a list of each individual scope is returned.
-
-    Args:
-        scopes: a string or iterable of strings, the scopes.
-
-    Returns:
-        The scopes in a list.
-    """
-    if not scopes:
-        return []
-    if isinstance(scopes, six.string_types):
-        return scopes.split(' ')
-    else:
-        return scopes
-
-
-def _add_query_parameter(url, name, value):
-    """Adds a query parameter to a url.
-
-    Replaces the current value if it already exists in the URL.
-
-    Args:
-        url: string, url to add the query parameter to.
-        name: string, query parameter name.
-        value: string, query parameter value.
-
-    Returns:
-        Updated query parameter. Does not update the url if value is None.
-    """
-    if value is None:
-        return url
-    else:
-        parsed = list(urllib.parse.urlparse(url))
-        q = dict(urllib.parse.parse_qsl(parsed[4]))
-        q[name] = value
-        parsed[4] = urllib.parse.urlencode(q)
-        return urllib.parse.urlunparse(parsed)

src/project-apis.txt

@@ -0,0 +1,12 @@
+admin.googleapis.com
+appsactivity.googleapis.com
+calendar-json.googleapis.com
+classroom.googleapis.com
+contacts.googleapis.com
+drive
+gmail.googleapis.com
+groupssettings.googleapis.com
+licensing.googleapis.com
+plus.googleapis.com
+reseller.googleapis.com
+siteverification.googleapis.com

src/utils.py

@@ -0,0 +1,80 @@
+import collections
+import re
+import sys
+from htmlentitydefs import name2codepoint
+from HTMLParser import HTMLParser
+from var import GM_Globals, GM_WINDOWS, GM_SYS_ENCODING
+
+def convertUTF8(data):
+  if isinstance(data, str):
+    return data
+  if isinstance(data, unicode):
+    if GM_Globals[GM_WINDOWS]:
+      return data
+    return data.encode(GM_Globals[GM_SYS_ENCODING])
+  if isinstance(data, collections.Mapping):
+    return dict(map(convertUTF8, data.iteritems()))
+  if isinstance(data, collections.Iterable):
+    return type(data)(map(convertUTF8, data))
+  return data
+
+class _DeHTMLParser(HTMLParser):
+  def __init__(self):
+    HTMLParser.__init__(self)
+    self.__text = []
+
+  def handle_data(self, data):
+    self.__text.append(data)
+
+  def handle_charref(self, name):
+    self.__text.append(unichr(int(name[1:], 16)) if name.startswith('x') else unichr(int(name)))
+
+  def handle_entityref(self, name):
+    cp = name2codepoint.get(name)
+    if cp:
+      self.__text.append(unichr(cp))
+    else:
+      self.__text.append(u'&'+name)
+
+  def handle_starttag(self, tag, attrs):
+    if tag == 'p':
+      self.__text.append('\n\n')
+    elif tag == 'br':
+      self.__text.append('\n')
+    elif tag == 'a':
+      for attr in attrs:
+        if attr[0] == 'href':
+          self.__text.append('({0}) '.format(attr[1]))
+          break
+    elif tag == 'div':
+      if not attrs:
+        self.__text.append('\n')
+    elif tag in ['http:', 'https']:
+      self.__text.append(' ({0}//{1}) '.format(tag, attrs[0][0]))
+
+  def handle_startendtag(self, tag, attrs):
+    if tag == 'br':
+      self.__text.append('\n\n')
+
+  def text(self):
+    return re.sub(r'\n{2}\n+', '\n\n', re.sub(r'\n +', '\n', ''.join(self.__text))).strip()
+
+def dehtml(text):
+  try:
+    parser = _DeHTMLParser()
+    parser.feed(text.encode(u'utf-8'))
+    parser.close()
+    return parser.text()
+  except:
+    from traceback import print_exc
+    print_exc(file=sys.stderr)
+    return text
+
+def indentMultiLineText(message, n=0):
+  return message.replace(u'\n', u'\n{0}'.format(u' '*n)).rstrip()
+
+def formatMilliSeconds(millis):
+  seconds, millis = divmod(millis, 1000)
+  minutes, seconds = divmod(seconds, 60)
+  hours, minutes = divmod(minutes, 60)
+  return u'%02d:%02d:%02d' % (hours, minutes, seconds)

src/var.py

@@ -0,0 +1,730 @@
+import os
+import sys
+import platform
+import re
+
+gam_author = u'Jay Lee <jay0lee@gmail.com>'
+gam_version = u'4.23'
+gam_license = u'Apache License 2.0 (http://www.apache.org/licenses/LICENSE-2.0)'
+
+GAM_URL = u'https://git.io/gam'
+GAM_INFO = u'GAM {0} - {1} / {2} / Python {3}.{4}.{5} {6} / {7} {8} /'.format(gam_version, GAM_URL,
+                                                                              gam_author,
+                                                                              sys.version_info[0], sys.version_info[1],
+                                                                              sys.version_info[2], sys.version_info[3],
+                                                                              platform.platform(), platform.machine())
+
+GAM_RELEASES = u'https://github.com/jay0lee/GAM/releases'
+GAM_WIKI = u'https://github.com/jay0lee/GAM/wiki'
+GAM_ALL_RELEASES = u'https://api.github.com/repos/jay0lee/GAM/releases'
+GAM_LATEST_RELEASE = GAM_ALL_RELEASES+u'/latest'
+GAM_PROJECT_APIS = u'https://raw.githubusercontent.com/jay0lee/GAM/master/src/project-apis.txt'
+
+TRUE = u'true'
+FALSE = u'false'
+true_values = [u'on', u'yes', u'enabled', u'true', u'1']
+false_values = [u'off', u'no', u'disabled', u'false', u'0']
+usergroup_types = [u'user', u'users', u'group', u'ou', u'org',
+                   u'ou_and_children', u'ou_and_child', u'query',
+                   u'license', u'licenses', u'licence', u'licences', u'file', u'csv', u'all',
+                   u'cros']
+ERROR = u'ERROR'
+ERROR_PREFIX = ERROR+u': '
+WARNING = u'WARNING'
+WARNING_PREFIX = WARNING+u': '
+DEFAULT_CHARSET = [u'mbcs', u'utf-8'][os.name != u'nt']
+ONE_KILO_BYTES = 1000
+ONE_MEGA_BYTES = 1000000
+ONE_GIGA_BYTES = 1000000000
+FN_CLIENT_SECRETS_JSON = u'client_secrets.json'
+FN_EXTRA_ARGS_TXT = u'extra-args.txt'
+FN_LAST_UPDATE_CHECK_TXT = u'lastupdatecheck.txt'
+FN_OAUTH2SERVICE_JSON = u'oauth2service.json'
+FN_OAUTH2_TXT = u'oauth2.txt'
+MY_CUSTOMER = u'my_customer'
+SKUS = {
+  u'Google-Apps': {
+    u'product': u'Google-Apps', u'aliases': [u'standard', u'free'], u'displayName': u'G Suite Free/Standard'},
+  u'Google-Apps-For-Business': {
+    u'product': u'Google-Apps', u'aliases': [u'gafb', u'gafw', u'basic', u'gsuitebasic'], u'displayName': u'G Suite Basic'},
+  u'Google-Apps-For-Government': {
+    u'product': u'Google-Apps', u'aliases': [u'gafg', u'gsuitegovernment', u'gsuitegov'], u'displayName': u'G Suite Government'},
+  u'Google-Apps-For-Postini': {
+    u'product': u'Google-Apps', u'aliases': [u'gams', u'postini', u'gsuitegams', u'gsuitepostini', u'gsuitemessagesecurity'], u'displayName': u'G Suite Message Security'},
+  u'Google-Apps-Lite': {
+    u'product': u'Google-Apps', u'aliases': [u'gal', u'lite', u'gsuitelite'], u'displayName': u'G Suite Lite'},
+  u'Google-Apps-Unlimited': {
+    u'product': u'Google-Apps', u'aliases': [u'gau', u'unlimited', u'gsuitebusiness'], u'displayName': u'G Suite Business'},
+  u'1010020020': {
+    u'product': u'Google-Apps', u'aliases': [u'gae', u'enterprise', u'gsuiteenterprise'], u'displayName': u'G Suite Enterprise'},
+  u'Google-Drive-storage-20GB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive20gb', u'20gb', u'googledrivestorage20gb'], u'displayName': u'Google Drive Storage 20GB'},
+  u'Google-Drive-storage-50GB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive50gb', u'50gb', u'googledrivestorage50gb'], u'displayName': u'Google Drive Storage 50GB'},
+  u'Google-Drive-storage-200GB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive200gb', u'200gb', u'googledrivestorage200gb'], u'displayName': u'Google Drive Storage 200GB'},
+  u'Google-Drive-storage-400GB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive400gb', u'400gb', u'googledrivestorage400gb'], u'displayName': u'Google Drive Storage 400GB'},
+  u'Google-Drive-storage-1TB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive1tb', u'1tb', u'googledrivestorage1tb'], u'displayName': u'Google Drive Storage 1TB'},
+  u'Google-Drive-storage-2TB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive2tb', u'2tb', u'googledrivestorage2tb'], u'displayName': u'Google Drive Storage 2TB'},
+  u'Google-Drive-storage-4TB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive4tb', u'4tb', u'googledrivestorage4tb'], u'displayName': u'Google Drive Storage 4TB'},
+  u'Google-Drive-storage-8TB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive8tb', u'8tb', u'googledrivestorage8tb'], u'displayName': u'Google Drive Storage 8TB'},
+  u'Google-Drive-storage-16TB': {
+    u'product': u'Google-Drive-storage', u'aliases': [u'drive16tb', u'16tb', u'googledrivestorage16tb'], u'displayName': u'Google Drive Storage 16TB'},
+  u'Google-Vault': {
+    u'product': u'Google-Vault', u'aliases': [u'vault', u'googlevault'], u'displayName': u'Google Vault'},
+  u'Google-Vault-Former-Employee': {
+    u'product': u'Google-Vault', u'aliases': [u'vfe', u'googlevaultformeremployee'], u'displayName': u'Google Vault Former Employee'},
+  u'Google-Coordinate': {
+    u'product': u'Google-Coordinate', u'aliases': [u'coordinate', u'googlecoordinate'], u'displayName': u'Google Coordinate'},
+  u'Google-Chrome-Device-Management': {
+    u'product': u'Google-Chrome-Device-Management', u'aliases': [u'chrome', u'cdm', u'googlechromedevicemanagement'], u'displayName': u'Google Chrome Device Management'}
+  }
+
+API_VER_MAPPING = {
+  u'appsactivity': u'v1',
+  u'calendar': u'v3',
+  u'classroom': u'v1',
+  u'cloudprint': u'v2',
+  u'datatransfer': u'datatransfer_v1',
+  u'directory': u'directory_v1',
+  u'drive': u'v2',
+  u'drive3': u'v3',
+  u'email-settings': u'v2',
+  u'gmail': u'v1',
+  u'groupssettings': u'v1',
+  u'licensing': u'v1',
+  u'oauth2': u'v2',
+  u'plus': u'v1',
+  u'reports': u'reports_v1',
+  u'reseller': u'v1',
+  u'siteVerification': u'v1',
+  u'urlshortener': u'v1',
+  }
+
+API_SCOPE_MAPPING = {
+  u'appsactivity': [u'https://www.googleapis.com/auth/activity',
+                    u'https://www.googleapis.com/auth/drive'],
+  u'calendar': [u'https://www.googleapis.com/auth/calendar',],
+  u'drive': [u'https://www.googleapis.com/auth/drive',],
+  u'drive3': [u'https://www.googleapis.com/auth/drive',],
+  u'gmail': [u'https://mail.google.com/',
+             u'https://www.googleapis.com/auth/gmail.settings.basic',
+             u'https://www.googleapis.com/auth/gmail.settings.sharing',],
+  u'plus': [u'https://www.googleapis.com/auth/plus.me',],
+}
+
+ADDRESS_FIELDS_PRINT_ORDER = [
+  u'contactName', u'organizationName',
+  u'addressLine1', u'addressLine2', u'addressLine3',
+  u'locality', u'region', u'postalCode', u'countryCode',
+  ]
+
+ADDRESS_FIELDS_ARGUMENT_MAP = {
+  u'contact': u'contactName', u'contactname': u'contactName',
+  u'name': u'organizationName', u'organizationname': u'organizationName',
+  u'address1': u'addressLine1', u'addressline1': u'addressLine1',
+  u'address2': u'addressLine2', u'addressline2': u'addressLine2',
+  u'address3': u'addressLine3', u'addressline3': u'addressLine3',
+  u'locality': u'locality',
+  u'region': u'region',
+  u'postalcode': u'postalCode',
+  u'country': u'countryCode', u'countrycode': u'countryCode',
+  }
+
+SERVICE_NAME_CHOICES_MAP = {
+  u'drive': u'Drive and Docs',
+  u'drive and docs': u'Drive and Docs',
+  u'googledrive': u'Drive and Docs',
+  u'gdrive': u'Drive and Docs',
+  }
+
+PRINTJOB_ASCENDINGORDER_MAP = {
+  u'createtime': u'CREATE_TIME',
+  u'status': u'STATUS',
+  u'title': u'TITLE',
+  }
+PRINTJOB_DESCENDINGORDER_MAP = {
+  u'CREATE_TIME': u'CREATE_TIME_DESC',
+  u'STATUS':  u'STATUS_DESC',
+  u'TITLE': u'TITLE_DESC',
+  }
+
+PRINTJOBS_DEFAULT_JOB_LIMIT = 0
+PRINTJOBS_DEFAULT_MAX_RESULTS = 100
+
+CALENDAR_REMINDER_METHODS = [u'email', u'sms', u'popup',]
+CALENDAR_NOTIFICATION_METHODS = [u'email', u'sms',]
+CALENDAR_NOTIFICATION_TYPES_MAP = {
+  u'eventcreation': u'eventCreation',
+  u'eventchange': u'eventChange',
+  u'eventcancellation': u'eventCancellation',
+  u'eventresponse': u'eventResponse',
+  u'agenda': u'agenda',
+  }
+
+DRIVEFILE_FIELDS_CHOICES_MAP = {
+  u'alternatelink': u'alternateLink',
+  u'appdatacontents': u'appDataContents',
+  u'cancomment': u'canComment',
+  u'canreadrevisions': u'canReadRevisions',
+  u'copyable': u'copyable',
+  u'createddate': u'createdDate',
+  u'createdtime': u'createdDate',
+  u'description': u'description',
+  u'editable': u'editable',
+  u'explicitlytrashed': u'explicitlyTrashed',
+  u'fileextension': u'fileExtension',
+  u'filesize': u'fileSize',
+  u'foldercolorrgb': u'folderColorRgb',
+  u'fullfileextension': u'fullFileExtension',
+  u'headrevisionid': u'headRevisionId',
+  u'iconlink': u'iconLink',
+  u'id': u'id',
+  u'lastmodifyinguser': u'lastModifyingUser',
+  u'lastmodifyingusername': u'lastModifyingUserName',
+  u'lastviewedbyme': u'lastViewedByMeDate',
+  u'lastviewedbymedate': u'lastViewedByMeDate',
+  u'lastviewedbymetime': u'lastViewedByMeDate',
+  u'lastviewedbyuser': u'lastViewedByMeDate',
+  u'md5': u'md5Checksum',
+  u'md5checksum': u'md5Checksum',
+  u'md5sum': u'md5Checksum',
+  u'mime': u'mimeType',
+  u'mimetype': u'mimeType',
+  u'modifiedbyme': u'modifiedByMeDate',
+  u'modifiedbymedate': u'modifiedByMeDate',
+  u'modifiedbymetime': u'modifiedByMeDate',
+  u'modifiedbyuser': u'modifiedByMeDate',
+  u'modifieddate': u'modifiedDate',
+  u'modifiedtime': u'modifiedDate',
+  u'name': u'title',
+  u'originalfilename': u'originalFilename',
+  u'ownedbyme': u'ownedByMe',
+  u'ownernames': u'ownerNames',
+  u'owners': u'owners',
+  u'parents': u'parents',
+  u'permissions': u'permissions',
+  u'quotabytesused': u'quotaBytesUsed',
+  u'quotaused': u'quotaBytesUsed',
+  u'shareable': u'shareable',
+  u'shared': u'shared',
+  u'sharedwithmedate': u'sharedWithMeDate',
+  u'sharedwithmetime': u'sharedWithMeDate',
+  u'sharinguser': u'sharingUser',
+  u'spaces': u'spaces',
+  u'thumbnaillink': u'thumbnailLink',
+  u'title': u'title',
+  u'userpermission': u'userPermission',
+  u'version': u'version',
+  u'viewedbyme': u'labels(viewed)',
+  u'viewedbymedate': u'lastViewedByMeDate',
+  u'viewedbymetime': u'lastViewedByMeDate',
+  u'viewerscancopycontent': u'labels(restricted)',
+  u'webcontentlink': u'webContentLink',
+  u'webviewlink': u'webViewLink',
+  u'writerscanshare': u'writersCanShare',
+  }
+
+DRIVEFILE_LABEL_CHOICES_MAP = {
+  u'restricted': u'restricted',
+  u'restrict': u'restricted',
+  u'starred': u'starred',
+  u'star': u'starred',
+  u'trashed': u'trashed',
+  u'trash': u'trashed',
+  u'viewed': u'viewed',
+  u'view': u'viewed',
+}
+
+DRIVEFILE_ORDERBY_CHOICES_MAP = {
+  u'createddate': u'createdDate',
+  u'folder': u'folder',
+  u'lastviewedbyme': u'lastViewedByMeDate',
+  u'lastviewedbymedate': u'lastViewedByMeDate',
+  u'lastviewedbyuser': u'lastViewedByMeDate',
+  u'modifiedbyme': u'modifiedByMeDate',
+  u'modifiedbymedate': u'modifiedByMeDate',
+  u'modifiedbyuser': u'modifiedByMeDate',
+  u'modifieddate': u'modifiedDate',
+  u'name': u'title',
+  u'quotabytesused': u'quotaBytesUsed',
+  u'quotaused': u'quotaBytesUsed',
+  u'recency': u'recency',
+  u'sharedwithmedate': u'sharedWithMeDate',
+  u'starred': u'starred',
+  u'title': u'title',
+  u'viewedbymedate': u'lastViewedByMeDate',
+  }
+
+DELETE_DRIVEFILE_FUNCTION_TO_ACTION_MAP = {
+  u'delete': u'purging',
+  u'trash': u'trashing',
+  u'untrash': u'untrashing',
+  }
+
+DRIVEFILE_LABEL_CHOICES_MAP = {
+  u'restricted': u'restricted',
+  u'restrict': u'restricted',
+  u'starred': u'starred',
+  u'star': u'starred',
+  u'trashed': u'trashed',
+  u'trash': u'trashed',
+  u'viewed': u'viewed',
+  u'view': u'viewed',
+  }
+
+APPLICATION_VND_GOOGLE_APPS = u'application/vnd.google-apps.'
+MIMETYPE_GA_DOCUMENT = APPLICATION_VND_GOOGLE_APPS+u'document'
+MIMETYPE_GA_DRAWING = APPLICATION_VND_GOOGLE_APPS+u'drawing'
+MIMETYPE_GA_FOLDER = APPLICATION_VND_GOOGLE_APPS+u'folder'
+MIMETYPE_GA_FORM = APPLICATION_VND_GOOGLE_APPS+u'form'
+MIMETYPE_GA_FUSIONTABLE = APPLICATION_VND_GOOGLE_APPS+u'fusiontable'
+MIMETYPE_GA_MAP = APPLICATION_VND_GOOGLE_APPS+u'map'
+MIMETYPE_GA_PRESENTATION = APPLICATION_VND_GOOGLE_APPS+u'presentation'
+MIMETYPE_GA_SCRIPT = APPLICATION_VND_GOOGLE_APPS+u'script'
+MIMETYPE_GA_SITES = APPLICATION_VND_GOOGLE_APPS+u'sites'
+MIMETYPE_GA_SPREADSHEET = APPLICATION_VND_GOOGLE_APPS+u'spreadsheet'
+
+MIMETYPE_CHOICES_MAP = {
+  u'gdoc': MIMETYPE_GA_DOCUMENT,
+  u'gdocument': MIMETYPE_GA_DOCUMENT,
+  u'gdrawing': MIMETYPE_GA_DRAWING,
+  u'gfolder': MIMETYPE_GA_FOLDER,
+  u'gdirectory': MIMETYPE_GA_FOLDER,
+  u'gform': MIMETYPE_GA_FORM,
+  u'gfusion': MIMETYPE_GA_FUSIONTABLE,
+  u'gpresentation': MIMETYPE_GA_PRESENTATION,
+  u'gscript': MIMETYPE_GA_SCRIPT,
+  u'gsite': MIMETYPE_GA_SITES,
+  u'gsheet': MIMETYPE_GA_SPREADSHEET,
+  u'gspreadsheet': MIMETYPE_GA_SPREADSHEET,
+  }
+
+DFA_CONVERT = u'convert'
+DFA_LOCALFILEPATH = u'localFilepath'
+DFA_LOCALFILENAME = u'localFilename'
+DFA_LOCALMIMETYPE = u'localMimeType'
+DFA_OCR = u'ocr'
+DFA_OCRLANGUAGE = u'ocrLanguage'
+DFA_PARENTQUERY = u'parentQuery'
+
+DOCUMENT_FORMATS_MAP = {
+  u'csv': [{u'mime': u'text/csv', u'ext': u'.csv'}],
+  u'html': [{u'mime': u'text/html', u'ext': u'.html'}],
+  u'txt': [{u'mime': u'text/plain', u'ext': u'.txt'}],
+  u'tsv': [{u'mime': u'text/tsv', u'ext': u'.tsv'}],
+  u'jpeg': [{u'mime': u'image/jpeg', u'ext': u'.jpeg'}],
+  u'jpg': [{u'mime': u'image/jpeg', u'ext': u'.jpg'}],
+  u'png': [{u'mime': u'image/png', u'ext': u'.png'}],
+  u'svg': [{u'mime': u'image/svg+xml', u'ext': u'.svg'}],
+  u'pdf': [{u'mime': u'application/pdf', u'ext': u'.pdf'}],
+  u'rtf': [{u'mime': u'application/rtf', u'ext': u'.rtf'}],
+  u'zip': [{u'mime': u'application/zip', u'ext': u'.zip'}],
+  u'pptx': [{u'mime': u'application/vnd.openxmlformats-officedocument.presentationml.presentation', u'ext': u'.pptx'}],
+  u'xlsx': [{u'mime': u'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', u'ext': u'.xlsx'}],
+  u'docx': [{u'mime': u'application/vnd.openxmlformats-officedocument.wordprocessingml.document', u'ext': u'.docx'}],
+  u'ms': [{u'mime': u'application/vnd.openxmlformats-officedocument.presentationml.presentation', u'ext': u'.pptx'},
+          {u'mime': u'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', u'ext': u'.xlsx'},
+          {u'mime': u'application/vnd.openxmlformats-officedocument.wordprocessingml.document', u'ext': u'.docx'}],
+  u'microsoft': [{u'mime': u'application/vnd.openxmlformats-officedocument.presentationml.presentation', u'ext': u'.pptx'},
+                 {u'mime': u'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', u'ext': u'.xlsx'},
+                 {u'mime': u'application/vnd.openxmlformats-officedocument.wordprocessingml.document', u'ext': u'.docx'}],
+  u'micro$oft': [{u'mime': u'application/vnd.openxmlformats-officedocument.presentationml.presentation', u'ext': u'.pptx'},
+                 {u'mime': u'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', u'ext': u'.xlsx'},
+                 {u'mime': u'application/vnd.openxmlformats-officedocument.wordprocessingml.document', u'ext': u'.docx'}],
+  u'odt': [{u'mime': u'application/vnd.oasis.opendocument.text', u'ext': u'.odt'}],
+  u'ods': [{u'mime': u'application/x-vnd.oasis.opendocument.spreadsheet', u'ext': u'.ods'}],
+  u'openoffice': [{u'mime': u'application/vnd.oasis.opendocument.text', u'ext': u'.odt'},
+                  {u'mime': u'application/x-vnd.oasis.opendocument.spreadsheet', u'ext': u'.ods'}],
+  }
+
+EMAILSETTINGS_OLD_NEW_OLD_FORWARD_ACTION_MAP = {
+  u'ARCHIVE': u'archive',
+  u'DELETE': u'trash',
+  u'KEEP': u'leaveInInBox',
+  u'MARK_READ': u'markRead',
+  u'archive': u'ARCHIVE',
+  u'trash': u'DELETE',
+  u'leaveInInbox': u'KEEP',
+  u'markRead': u'MARK_READ',
+  }
+
+EMAILSETTINGS_IMAP_EXPUNGE_BEHAVIOR_CHOICES_MAP = {
+  u'archive': u'archive',
+  u'deleteforever': u'deleteForever',
+  u'trash': u'trash',
+  }
+
+EMAILSETTINGS_IMAP_MAX_FOLDER_SIZE_CHOICES = [u'0', u'1000', u'2000', u'5000', u'10000']
+
+EMAILSETTINGS_POP_ENABLE_FOR_CHOICES_MAP = {
+  u'allmail': u'allMail',
+  u'fromnowon': u'fromNowOn',
+  u'mailfromnowon': u'fromNowOn',
+  u'newmail': u'fromNowOn',
+  }
+
+EMAILSETTINGS_FORWARD_POP_ACTION_CHOICES_MAP = {
+  u'archive': u'archive',
+  u'delete': u'trash',
+  u'keep': u'leaveInInbox',
+  u'leaveininbox': u'leaveInInbox',
+  u'markread': u'markRead',
+  u'trash': u'trash',
+  }
+
+RT_PATTERN = re.compile(r'(?s){RT}.*?{(.+?)}.*?{/RT}')
+RT_OPEN_PATTERN = re.compile(r'{RT}')
+RT_CLOSE_PATTERN = re.compile(r'{/RT}')
+RT_STRIP_PATTERN = re.compile(r'(?s){RT}.*?{/RT}')
+RT_TAG_REPLACE_PATTERN = re.compile(r'{(.*?)}')
+
+FILTER_ADD_LABEL_TO_ARGUMENT_MAP = {
+  u'IMPORTANT': u'important',
+  u'STARRED': u'star',
+  u'TRASH': u'trash',
+  }
+
+FILTER_REMOVE_LABEL_TO_ARGUMENT_MAP = {
+  u'IMPORTANT': u'notimportant',
+  u'UNREAD': u'markread',
+  u'INBOX': u'archive',
+  u'SPAM': u'neverspam',
+  }
+
+FILTER_CRITERIA_CHOICES_MAP = {
+  u'excludechats': u'excludeChats',
+  u'from': u'from',
+  u'hasattachment': u'hasAttachment',
+  u'haswords': u'query',
+  u'musthaveattachment': u'hasAttachment',
+  u'negatedquery': u'negatedQuery',
+  u'nowords': u'negatedQuery',
+  u'query': u'query',
+  u'size': u'size',
+  u'subject': u'subject',
+  u'to': u'to',
+  }
+FILTER_ACTION_CHOICES = [
+  u'archive', u'forward', u'important', u'label',
+  u'markread', u'neverspam', u'notimportant', u'star', u'trash',
+  ]
+
+CROS_ARGUMENT_TO_PROPERTY_MAP = {
+  u'activetimeranges': [u'activeTimeRanges.activeTime', u'activeTimeRanges.date'],
+  u'annotatedassetid': [u'annotatedAssetId',],
+  u'annotatedlocation': [u'annotatedLocation',],
+  u'annotateduser': [u'annotatedUser',],
+  u'asset': [u'annotatedAssetId',],
+  u'assetid': [u'annotatedAssetId',],
+  u'bootmode': [u'bootMode',],
+  u'deviceid': [u'deviceId',],
+  u'ethernetmacaddress': [u'ethernetMacAddress',],
+  u'firmwareversion': [u'firmwareVersion',],
+  u'lastenrollmenttime': [u'lastEnrollmentTime',],
+  u'lastsync': [u'lastSync',],
+  u'location': [u'annotatedLocation',],
+  u'macaddress': [u'macAddress',],
+  u'meid': [u'meid',],
+  u'model': [u'model',],
+  u'notes': [u'notes',],
+  u'ordernumber': [u'orderNumber',],
+  u'org': [u'orgUnitPath',],
+  u'orgunitpath': [u'orgUnitPath',],
+  u'osversion': [u'osVersion',],
+  u'ou': [u'orgUnitPath',],
+  u'platformversion': [u'platformVersion',],
+  u'recentusers': [u'recentUsers.email', u'recentUsers.type'],
+  u'serialnumber': [u'serialNumber',],
+  u'status': [u'status',],
+  u'supportenddate': [u'supportEndDate',],
+  u'tag': [u'annotatedAssetId',],
+  u'timeranges': [u'activeTimeRanges.activeTime', u'activeTimeRanges.date'],
+  u'times': [u'activeTimeRanges.activeTime', u'activeTimeRanges.date'],
+  u'user': [u'annotatedUser',],
+  u'users': [u'recentUsers.email', u'recentUsers.type'],
+  u'willautorenew': [u'willAutoRenew',],
+  }
+
+CROS_BASIC_FIELDS_LIST = [u'deviceId', u'annotatedAssetId', u'annotatedLocation', u'annotatedUser', u'lastSync', u'notes', u'serialNumber', u'status']
+
+CROS_SCALAR_PROPERTY_PRINT_ORDER = [
+  u'orgUnitPath',
+  u'annotatedAssetId',
+  u'annotatedLocation',
+  u'annotatedUser',
+  u'lastSync',
+  u'notes',
+  u'serialNumber',
+  u'status',
+  u'model',
+  u'firmwareVersion',
+  u'platformVersion',
+  u'osVersion',
+  u'bootMode',
+  u'meid',
+  u'ethernetMacAddress',
+  u'macAddress',
+  u'lastEnrollmentTime',
+  u'orderNumber',
+  u'supportEndDate',
+  u'willAutoRenew',
+  ]
+
+CROS_RECENT_USERS_ARGUMENTS = [u'recentusers', u'users']
+CROS_ACTIVE_TIME_RANGES_ARGUMENTS = [u'timeranges', u'activetimeranges', u'times']
+CROS_START_ARGUMENTS = [u'start', u'startdate', u'oldestdate']
+CROS_END_ARGUMENTS = [u'end', u'enddate']
+
+#
+# Global variables
+#
+# The following GM_XXX constants are arbitrary but must be unique
+# Most errors print a message and bail out with a return code
+# Some commands want to set a non-zero return code but not bail
+GM_SYSEXITRC = u'sxrc'
+# Path to gam
+GM_GAM_PATH = u'gpth'
+# Are we on Windows?
+GM_WINDOWS = u'wndo'
+# Encodings
+GM_SYS_ENCODING = u'syen'
+# Extra arguments to pass to GAPI functions
+GM_EXTRA_ARGS_DICT = u'exad'
+# Current API user
+GM_CURRENT_API_USER = u'capu'
+# Current API scope
+GM_CURRENT_API_SCOPES = u'scoc'
+# Values retrieved from oauth2service.json
+GM_OAUTH2SERVICE_JSON_DATA = u'oajd'
+GM_OAUTH2SERVICE_ACCOUNT_CLIENT_ID = u'oaci'
+# File containing time of last GAM update check
+GM_LAST_UPDATE_CHECK_TXT = u'lupc'
+# Dictionary mapping OrgUnit ID to Name
+GM_MAP_ORGUNIT_ID_TO_NAME = u'oi2n'
+# Dictionary mapping Role ID to Name
+GM_MAP_ROLE_ID_TO_NAME = u'ri2n'
+# Dictionary mapping Role Name to ID
+GM_MAP_ROLE_NAME_TO_ID = u'rn2i'
+# Dictionary mapping User ID to Name
+GM_MAP_USER_ID_TO_NAME = u'ui2n'
+# GAM cache directory. If no_cache is True, this variable will be set to None
+GM_CACHE_DIR = u'gacd'
+# Reset GAM cache directory after discovery
+GM_CACHE_DISCOVERY_ONLY = u'gcdo'
+#
+GM_Globals = {
+  GM_SYSEXITRC: 0,
+  GM_GAM_PATH: os.path.dirname(os.path.realpath(__file__)) if not getattr(sys, u'frozen', False) else os.path.dirname(sys.executable),
+  GM_WINDOWS: os.name == u'nt',
+  GM_SYS_ENCODING: DEFAULT_CHARSET,
+  GM_EXTRA_ARGS_DICT:  {u'prettyPrint': False},
+  GM_CURRENT_API_USER: None,
+  GM_CURRENT_API_SCOPES: [],
+  GM_OAUTH2SERVICE_JSON_DATA: None,
+  GM_OAUTH2SERVICE_ACCOUNT_CLIENT_ID: None,
+  GM_LAST_UPDATE_CHECK_TXT: u'',
+  GM_MAP_ORGUNIT_ID_TO_NAME: None,
+  GM_MAP_ROLE_ID_TO_NAME: None,
+  GM_MAP_ROLE_NAME_TO_ID: None,
+  GM_MAP_USER_ID_TO_NAME: None,
+  GM_CACHE_DIR: None,
+  GM_CACHE_DISCOVERY_ONLY: True,
+  }
+#
+# Global variables defined by environment variables/signal files
+#
+# When retrieving lists of Google Drive activities from API, how many should be retrieved in each chunk
+GC_ACTIVITY_MAX_RESULTS = u'activity_max_results'
+# Automatically generate gam batch command if number of users specified in gam users xxx command exceeds this number
+# Default: 0, don't automatically generate gam batch commands
+GC_AUTO_BATCH_MIN = u'auto_batch_min'
+# When processing items in batches, how many should be processed in each batch
+GC_BATCH_SIZE = u'batch_size'
+# GAM cache directory. If no_cache is specified, this variable will be set to None
+GC_CACHE_DIR = u'cache_dir'
+# GAM cache discovery only. If no_cache is False, only API discovery calls will be cached
+GC_CACHE_DISCOVERY_ONLY = u'cache_discovery_only'
+# Character set of batch, csv, data files
+GC_CHARSET = u'charset'
+# Path to client_secrets.json
+GC_CLIENT_SECRETS_JSON = u'client_secrets_json'
+# GAM config directory containing client_secrets.json, oauth2.txt, oauth2service.json, extra_args.txt
+GC_CONFIG_DIR = u'config_dir'
+# custmerId from gam.cfg or retrieved from Google
+GC_CUSTOMER_ID = u'customer_id'
+# If debug_level > 0: extra_args[u'prettyPrint'] = True, httplib2.debuglevel = gam_debug_level, appsObj.debug = True
+GC_DEBUG_LEVEL = u'debug_level'
+# When retrieving lists of ChromeOS/Mobile devices from API, how many should be retrieved in each chunk
+GC_DEVICE_MAX_RESULTS = u'device_max_results'
+# Domain obtained from gam.cfg or oauth2.txt
+GC_DOMAIN = u'domain'
+# Google Drive download directory
+GC_DRIVE_DIR = u'drive_dir'
+# When retrieving lists of Drive files/folders from API, how many should be retrieved in each chunk
+GC_DRIVE_MAX_RESULTS = u'drive_max_results'
+# If no_browser is False, writeCSVfile won't open a browser when todrive is set
+# and doRequestOAuth prints a link and waits for the verification code when oauth2.txt is being created
+GC_NO_BROWSER = u'no_browser'
+# Disable GAM API caching
+GC_NO_CACHE = u'no_cache'
+# Disable GAM update check
+GC_NO_UPDATE_CHECK = u'no_update_check'
+# Disable SSL certificate validation
+GC_NO_VERIFY_SSL = u'no_verify_ssl'
+# Number of threads for gam batch
+GC_NUM_THREADS = u'num_threads'
+# Path to oauth2.txt
+GC_OAUTH2_TXT = u'oauth2_txt'
+# Path to oauth2service.json
+GC_OAUTH2SERVICE_JSON = u'oauth2service_json'
+# Default section to use for processing
+GC_SECTION = u'section'
+# Add (n/m) to end of messages if number of items to be processed exceeds this number
+GC_SHOW_COUNTS_MIN = u'show_counts_min'
+# Enable/disable "Getting ... " messages
+GC_SHOW_GETTINGS = u'show_gettings'
+# GAM config directory containing json discovery files
+GC_SITE_DIR = u'site_dir'
+# When retrieving lists of Users from API, how many should be retrieved in each chunk
+GC_USER_MAX_RESULTS = u'user_max_results'
+
+GC_Defaults = {
+  GC_ACTIVITY_MAX_RESULTS: 100,
+  GC_AUTO_BATCH_MIN: 0,
+  GC_BATCH_SIZE: 50,
+  GC_CACHE_DIR: u'',
+  GC_CACHE_DISCOVERY_ONLY: True,
+  GC_CHARSET: DEFAULT_CHARSET,
+  GC_CLIENT_SECRETS_JSON: FN_CLIENT_SECRETS_JSON,
+  GC_CONFIG_DIR: u'',
+  GC_CUSTOMER_ID: MY_CUSTOMER,
+  GC_DEBUG_LEVEL: 0,
+  GC_DEVICE_MAX_RESULTS: 500,
+  GC_DOMAIN: u'',
+  GC_DRIVE_DIR: u'',
+  GC_DRIVE_MAX_RESULTS: 1000,
+  GC_NO_BROWSER: False,
+  GC_NO_CACHE: False,
+  GC_NO_UPDATE_CHECK: False,
+  GC_NO_VERIFY_SSL: False,
+  GC_NUM_THREADS: 25,
+  GC_OAUTH2_TXT: FN_OAUTH2_TXT,
+  GC_OAUTH2SERVICE_JSON: FN_OAUTH2SERVICE_JSON,
+  GC_SECTION: u'',
+  GC_SHOW_COUNTS_MIN: 0,
+  GC_SHOW_GETTINGS: True,
+  GC_SITE_DIR: u'',
+  GC_USER_MAX_RESULTS: 500,
+  }
+
+GC_Values = {}
+
+GC_TYPE_BOOLEAN = u'bool'
+GC_TYPE_CHOICE = u'choi'
+GC_TYPE_DIRECTORY = u'dire'
+GC_TYPE_EMAIL = u'emai'
+GC_TYPE_FILE = u'file'
+GC_TYPE_INTEGER = u'inte'
+GC_TYPE_LANGUAGE = u'lang'
+GC_TYPE_STRING = u'stri'
+
+GC_VAR_TYPE = u'type'
+GC_VAR_LIMITS = u'lmit'
+
+GC_VAR_INFO = {
+  GC_ACTIVITY_MAX_RESULTS: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (1, 500)},
+  GC_AUTO_BATCH_MIN: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (0, None)},
+  GC_BATCH_SIZE: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (1, 1000)},
+  GC_CACHE_DIR: {GC_VAR_TYPE: GC_TYPE_DIRECTORY},
+  GC_CACHE_DISCOVERY_ONLY: {GC_VAR_TYPE: GC_TYPE_BOOLEAN},
+  GC_CHARSET: {GC_VAR_TYPE: GC_TYPE_STRING},
+  GC_CLIENT_SECRETS_JSON: {GC_VAR_TYPE: GC_TYPE_FILE},
+  GC_CONFIG_DIR: {GC_VAR_TYPE: GC_TYPE_DIRECTORY},
+  GC_CUSTOMER_ID: {GC_VAR_TYPE: GC_TYPE_STRING},
+  GC_DEBUG_LEVEL: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (0, None)},
+  GC_DEVICE_MAX_RESULTS: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (1, 1000)},
+  GC_DOMAIN: {GC_VAR_TYPE: GC_TYPE_STRING},
+  GC_DRIVE_DIR: {GC_VAR_TYPE: GC_TYPE_DIRECTORY},
+  GC_DRIVE_MAX_RESULTS: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (1, 1000)},
+  GC_NO_BROWSER: {GC_VAR_TYPE: GC_TYPE_BOOLEAN},
+  GC_NO_CACHE: {GC_VAR_TYPE: GC_TYPE_BOOLEAN},
+  GC_NO_UPDATE_CHECK: {GC_VAR_TYPE: GC_TYPE_BOOLEAN},
+  GC_NO_VERIFY_SSL: {GC_VAR_TYPE: GC_TYPE_BOOLEAN},
+  GC_NUM_THREADS: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (1, None)},
+  GC_OAUTH2_TXT: {GC_VAR_TYPE: GC_TYPE_FILE},
+  GC_OAUTH2SERVICE_JSON: {GC_VAR_TYPE: GC_TYPE_FILE},
+  GC_SECTION: {GC_VAR_TYPE: GC_TYPE_STRING},
+  GC_SHOW_COUNTS_MIN: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (0, None)},
+  GC_SHOW_GETTINGS: {GC_VAR_TYPE: GC_TYPE_BOOLEAN},
+  GC_SITE_DIR: {GC_VAR_TYPE: GC_TYPE_DIRECTORY},
+  GC_USER_MAX_RESULTS: {GC_VAR_TYPE: GC_TYPE_INTEGER, GC_VAR_LIMITS: (1, 500)},
+  }
+# Google API constants
+
+NEVER_TIME = u'1970-01-01T00:00:00.000Z'
+NEVER_START_DATE = u'1970-01-01'
+NEVER_END_DATE = u'1969-12-31'
+ROLE_MANAGER = u'MANAGER'
+ROLE_MEMBER = u'MEMBER'
+ROLE_OWNER = u'OWNER'
+ROLE_USER = u'USER'
+ROLE_MANAGER_MEMBER = u','.join([ROLE_MANAGER, ROLE_MEMBER])
+ROLE_MANAGER_OWNER = u','.join([ROLE_MANAGER, ROLE_OWNER])
+ROLE_MANAGER_MEMBER_OWNER = u','.join([ROLE_MANAGER, ROLE_MEMBER, ROLE_OWNER])
+ROLE_MEMBER_OWNER = u','.join([ROLE_MEMBER, ROLE_OWNER])
+PROJECTION_CHOICES_MAP = {u'basic': u'BASIC', u'full': u'FULL',}
+SORTORDER_CHOICES_MAP = {u'ascending': u'ASCENDING', u'descending': u'DESCENDING',}
+#
+CLEAR_NONE_ARGUMENT = [u'clear', u'none',]
+#
+MESSAGE_API_ACCESS_CONFIG = u'API access is configured in your Control Panel under: Security-Show more-Advanced settings-Manage API client access'
+MESSAGE_API_ACCESS_DENIED = u'API access Denied.\n\nPlease make sure the Client ID: {0} is authorized for the API Scope(s): {1}'
+MESSAGE_GAM_EXITING_FOR_UPDATE = u'GAM is now exiting so that you can overwrite this old version with the latest release'
+MESSAGE_GAM_OUT_OF_MEMORY = u'GAM has run out of memory. If this is a large G Suite instance, you should use a 64-bit version of GAM on Windows or a 64-bit version of Python on other systems.'
+MESSAGE_HEADER_NOT_FOUND_IN_CSV_HEADERS = u'Header "{0}" not found in CSV headers of "{1}".'
+MESSAGE_HIT_CONTROL_C_TO_UPDATE = u'\n\nHit CTRL+C to visit the GAM website and download the latest release or wait 15 seconds continue with this boring old version. GAM won\'t bother you with this announcement for 1 week or you can create a file named noupdatecheck.txt in the same location as gam.py or gam.exe and GAM won\'t ever check for updates.'
+MESSAGE_INVALID_JSON = u'The file {0} has an invalid format.'
+MESSAGE_NO_DISCOVERY_INFORMATION = u'No online discovery doc and {0} does not exist locally'
+MESSAGE_NO_PYTHON_SSL = u'You don\'t have the Python SSL module installed so we can\'t verify SSL Certificates. You can fix this by installing the Python SSL module or you can live on the edge and turn SSL validation off by creating a file named noverifyssl.txt in the same location as gam.exe / gam.py'
+MESSAGE_NO_TRANSFER_LACK_OF_DISK_SPACE = u'Cowardly refusing to perform migration due to lack of target drive space. Source size: {0}mb Target Free: {1}mb'
+MESSAGE_REQUEST_COMPLETED_NO_FILES = u'Request completed but no results/files were returned, try requesting again'
+MESSAGE_REQUEST_NOT_COMPLETE = u'Request needs to be completed before downloading, current status is: {0}'
+MESSAGE_RESULTS_TOO_LARGE_FOR_GOOGLE_SPREADSHEET = u'Results are too large for Google Spreadsheets. Uploading as a regular CSV file.'
+MESSAGE_SERVICE_NOT_APPLICABLE = u'Service not applicable for this address: {0}. Please make sure service is enabled for user and run\n\ngam user <user> check serviceaccount\n\nfor further instructions'
+MESSAGE_INSTRUCTIONS_OAUTH2SERVICE_JSON = u'Please run\n\ngam create project\ngam user <user> check serviceaccount\n\nto create and configure a service account.'
+MESSAGE_OAUTH2SERVICE_JSON_INVALID = u'The file {0} is missing required keys (client_email, client_id or private_key). Please remove it and recreate with the commands:\n\ngam create project\ngam user <user> check serviceaccount'
+# oauth errors
+OAUTH2_TOKEN_ERRORS = [
+  u'access_denied',
+  u'access_denied: Requested client not authorized',
+  u'internal_failure: Backend Error',
+  u'invalid_grant: Bad Request',
+  u'invalid_grant: Invalid email or User ID',
+  u'invalid_grant: Not a valid email',
+  u'invalid_request: Invalid impersonation prn email address',
+  u'unauthorized_client: Client is unauthorized to retrieve access tokens using this method',
+  u'unauthorized_client: Unauthorized client or scope in request',
+  ]
+#
+# callGAPI throw reasons
+GAPI_BACKEND_ERROR = u'backendError'
+GAPI_BAD_REQUEST = u'badRequest'
+GAPI_FORBIDDEN = u'forbidden'
+GAPI_INTERNAL_ERROR = u'internalError'
+GAPI_INVALID = u'invalid'
+GAPI_NOT_FOUND = u'notFound'
+GAPI_QUOTA_EXCEEDED = u'quotaExceeded'
+GAPI_RATE_LIMIT_EXCEEDED = u'rateLimitExceeded'
+GAPI_SERVICE_NOT_AVAILABLE = u'serviceNotAvailable'
+GAPI_USER_NOT_FOUND = u'userNotFound'
+GAPI_USER_RATE_LIMIT_EXCEEDED = u'userRateLimitExceeded'
+#
+GAPI_DEFAULT_RETRY_REASONS = [GAPI_QUOTA_EXCEEDED, GAPI_RATE_LIMIT_EXCEEDED, GAPI_USER_RATE_LIMIT_EXCEEDED, GAPI_BACKEND_ERROR, GAPI_INTERNAL_ERROR]
+GAPI_GMAIL_THROW_REASONS = [GAPI_SERVICE_NOT_AVAILABLE]
+GAPI_GPLUS_THROW_REASONS = [GAPI_SERVICE_NOT_AVAILABLE]

src/whatsnew.txt

@@ -1,3 +1,51 @@
+GAM 4.22
+ - Validate client id and secret from user input (reduces user errors on create project)
+ - Ross - optimize "gam print printjobs"
+ - Ross- countsonly option for gam print courses
+
+GAM 4.21
+ - Drive v3 fixes/updates by Ross
+ - "gam print crosactivty" command outputs active users and times
+ - SMime and calendar ACL fixes by Ross
+ - standardized cros info/print functionality by Ross
+
+GAM 4.2
+ - Create, Update, Delete and List Team Drives
+ - Start moving to Drive API v3
+ - Disable GAM cache by default to prevent errors (Ross)
+ - Use service accounts for all Calendar, Drive and Gmail operations to reduce scopes
+ - Fix "Unknown" errors due to a scope issue (may require "gam oauth revoke" and re-authentication)
+ - "gam info domain" shows basic user / license sums again
+ - "gam report customer" now shows more browser usage stats
+ - Fix project creation ToS error (Ross)
+
+GAM 4.12
+ - Realtime 2SV user status in gam info user and gam print users. Thanks hajdbo!
+ - Reseller API support. Create and manage customers and subscriptions.
+ - Delete or modify Gmail threads. Improved message delete/modify performance.
+ - Support for the new G Suite Enterprise license
+ - Manage S/MIME certificates for G Suite Enterprise users
+ - Many fixes and improvements by Ross.
+
+GAM 4.11
+ - Allow newlines in calendar event descriptions (Ross)
+ - All HTTP requests should now honor SSL verify setting, debug, etc
+
+GAM 4.1
+ - Fix "gam create project"
+ - project create cleanups by Ross
+ - Various fixes by Ross
+ - Improved batch command performance. Some commands will see 2-3x speedups.
+ - Faster "gam info user" commands via batch license retrieval
+
+GAM 4.03
+ - Minor fixes by Jay and Ross. Mostly to new install process.
+
+GAM 4.02
+ - "gam create project" command simplifies creation of client_secrets.json and oauth2service.json project files.
+ - Automated wizard simplifies GAM setup on Linux and MacOS (coming soon to Windows MSI).
+ - "gam calendar <email> deleteevent" deletes events by ID or query
+
 GAM 3.8
 - Old GData APIs removed from GAM. Admin Settings and Email Audit commands are no longer included, keep a copy of GAM 3.72 around if you use them. All Email Settings commands now use new Gmail API.
 - Updated httplib2, google-api-client, uritemplate libraries

src/windows-build.bat

@@ -5,20 +5,25 @@ rmdir /q /s dist
 del /q /f gam-%1-windows.zip
 del /q /f gam-%1-windows-x64.zip
 del /q /f gam-%1-windows-x64.msi
-del /q /f gam.wixobj
-del /q /f gam.wixpdb
+del /q /f *.wixobj
+del /q /f *.wixpdb
 
 c:\python27-32\scripts\pyinstaller --clean -F --distpath=gam windows-gam.spec
 xcopy LICENSE gam\
 xcopy whatsnew.txt gam\
+xcopy gam-setup.bat gam\
+xcopy GamCommands.txt gam\
 del gam\w9xpopen.exe
 "%ProgramFiles%\7-Zip\7z.exe" a -tzip gam-%1-windows.zip gam\ -xr!.svn
 
 c:\python27-64\scripts\pyinstaller --clean -F --distpath=gam-64 windows-gam.spec
 xcopy LICENSE gam-64\
 xcopy whatsnew.txt gam-64\
+xcopy gam-setup.bat gam-64\
+xcopy GamCommands.txt gam-64\
 "%ProgramFiles%\7-Zip\7z.exe" a -tzip gam-%1-windows-x64.zip gam-64\ -xr!.svn
 
 set GAMVERSION=%1
 "%ProgramFiles(x86)%\WiX Toolset v3.10\bin\candle.exe" -arch x64 gam.wxs
-"%ProgramFiles(x86)%\WiX Toolset v3.10\bin\light.exe" -ext "%ProgramFiles(x86)%\WiX Toolset v3.10\bin\WixUIExtension.dll" gam.wixobj -o gam-%1-windows-x64.msi
+"%ProgramFiles(x86)%\WiX Toolset v3.10\bin\light.exe" -ext "%ProgramFiles(x86)%\WiX Toolset v3.10\bin\WixUIExtension.dll" gam.wixobj -o gam-%1-windows-x64.msi
+del /q /f gam-%1-windows-x64.wixpdb

src/windows-gam.spec

@@ -11,7 +11,6 @@ for d in a.datas:
         break
 a.datas += [('httplib2/cacerts.txt', 'httplib2\cacerts.txt', 'DATA')]
 a.datas += [('cloudprint-v2.json', 'cloudprint-v2.json', 'DATA')]
-a.datas += [('email-audit-v1.json', 'email-audit-v1.json', 'DATA')]
 a.datas += [('email-settings-v2.json', 'email-settings-v2.json', 'DATA')]
 pyz = PYZ(a.pure)
 exe = EXE(pyz,