Python

0.4.0

Datastore#

Cloud Datastore in 10 seconds#

Install the library#

The source code for the library (and demo code) lives on GitHub, You can install the library quickly with pip:

$ pip install gcloud

Run the demo#

In order to run the demo, you need to have registred an actual gcloud project and so you’ll need to provide some environment variables to facilitate authentication to your project:

  • GCLOUD_TESTS_PROJECT_ID: Developers Console project ID (e.g. bamboo-shift-455).
  • GCLOUD_TESTS_DATASET_ID: The name of the dataset your tests connect to. This is typically the same as GCLOUD_TESTS_PROJECT_ID.
  • GOOGLE_APPLICATION_CREDENTIALS: The path to a JSON key file; see regression/app_credentials.json.sample as an example. Such a file can be downloaded directly from the developer’s console by clicking “Generate new JSON key”. See private key docs for more details.

Run the example script included in the package:

$ python -m gcloud.datastore.demo

And that’s it! You just read and wrote a bunch of data to the Cloud Datastore.

Try it yourself#

You can interact with a demo dataset in a Python interactive shell.

Start by importing the demo module and initializing the demo settings:

>>> from gcloud.datastore import demo
>>> demo.initialize()

Once you have initialized, you can create entities and save them:

>>> from gcloud import datastore
>>> entity = datastore.Entity(key=datastore.Key('Person'))
>>> entity['name'] = 'Your name'
>>> entity['age'] = 25
>>> datastore.put([entity])
>>> list(datastore.Query(kind='Person').fetch())
[<Entity{...} {'name': 'Your name', 'age': 25}>]

gcloud.datastore#

Shortcut methods for getting set up with Google Cloud Datastore.

You’ll typically use these to get started with the API:

>>> from gcloud import datastore

>>> datastore.set_defaults()

>>> key = datastore.Key('EntityKind', 1234)
>>> entity = datastore.Entity(key)
>>> query = datastore.Query(kind='EntityKind')

The main concepts with this API are:

gcloud.datastore.__init__.SCOPE = ('https://www.googleapis.com/auth/datastore', 'https://www.googleapis.com/auth/userinfo.email')#

The scopes required for authenticating as a Cloud Datastore consumer.

gcloud.datastore.__init__.get_connection()[source]#

Shortcut method to establish a connection to the Cloud Datastore.

Use this if you are going to access several datasets with the same set of credentials (unlikely):

>>> from gcloud import datastore

>>> connection = datastore.get_connection()
>>> key1 = datastore.Key('Kind', 1234, dataset_id='dataset1')
>>> key2 = datastore.Key('Kind', 1234, dataset_id='dataset2')
>>> entity1 = datastore.get(key1, connection=connection)
>>> entity2 = datastore.get(key2, connection=connection)
Return type:gcloud.datastore.connection.Connection
Returns:A connection defined with the proper credentials.
gcloud.datastore.__init__.set_default_connection(connection=None)[source]#

Set default connection either explicitly or implicitly as fall-back.

Parameters:connection (gcloud.datastore.connection.Connection) – A connection provided to be the default.
gcloud.datastore.__init__.set_default_dataset_id(dataset_id=None)[source]#

Set default dataset ID either explicitly or implicitly as fall-back.

In implicit case, supports three cases. In order of precedence, the implicit cases are: - GCLOUD_DATASET_ID environment variable - Google App Engine application ID - Google Compute Engine project ID (from metadata server)

Parameters:dataset_id (string) – Optional. The dataset ID to use as default.
gcloud.datastore.__init__.set_defaults(dataset_id=None, connection=None)[source]#

Set defaults either explicitly or implicitly as fall-back.

Uses the arguments to call the individual default methods - set_default_dataset_id - set_default_connection

In the future we will likely enable methods like - set_default_namespace

Parameters:

Connections#

Connections to gcloud datastore API servers.

class gcloud.datastore.connection.Connection(credentials=None)[source]#

Bases: gcloud.connection.Connection

A connection to the Google Cloud Datastore via the Protobuf API.

This class should understand only the basic types (and protobufs) in method arguments, however should be capable of returning advanced types.

Parameters:credentials (oauth2client.client.OAuth2Credentials) – The OAuth2 Credentials to use for this connection.

Constructor for Connection.

Parameters:credentials (oauth2client.client.OAuth2Credentials or NoneType) – The OAuth2 Credentials to use for this connection.
API_URL_TEMPLATE = '{api_base}/datastore/{api_version}/datasets/{dataset_id}/{method}'#

A template for the URL of a particular API call.

API_VERSION = 'v1beta2'#

The version of the API, used in building the API call’s URL.

allocate_ids(dataset_id, key_pbs)[source]#

Obtain backend-generated IDs for a set of keys.

Maps the DatastoreService.AllocateIds protobuf RPC.

Parameters:
  • dataset_id (string) – The ID of the dataset to which the transaction belongs.
  • key_pbs (list of gcloud.datastore._datastore_v1_pb2.Key) – The keys for which the backend should allocate IDs.
Return type:

list of gcloud.datastore._datastore_v1_pb2.Key
Returns:

An equal number of keys, with IDs filled in by the backend.
begin_transaction(dataset_id, serializable=False)[source]#

Begin a transaction.

Maps the DatastoreService.BeginTransaction protobuf RPC.

Parameters:
  • dataset_id (string) – The ID dataset to which the transaction applies.
  • serializable (boolean) – Boolean indicating if the isolation level of the transaction should be SERIALIZABLE (True) or SNAPSHOT (False).
Return type:

_datastore_v1_pb2.BeginTransactionResponse
Returns’:

the result protobuf for the begin transaction request.
classmethod build_api_url(dataset_id, method, base_url=None, api_version=None)[source]#

Construct the URL for a particular API call.

This method is used internally to come up with the URL to use when making RPCs to the Cloud Datastore API.

Parameters:
  • dataset_id (string) – The ID of the dataset to connect to. This is usually your project name in the cloud console.
  • method (string) – The API method to call (ie, runQuery, lookup, ...).
  • base_url (string) – The base URL where the API lives. You shouldn’t have to provide this.
  • api_version (string) – The version of the API to connect to. You shouldn’t have to provide this.
commit(dataset_id, mutation_pb, transaction_id=None)[source]#

Commit dataset mutations in context of current transation (if any).

Maps the DatastoreService.Commit protobuf RPC.

Parameters:
  • dataset_id (string) – The ID dataset to which the transaction applies.
  • mutation_pb (datastore_pb.Mutation.) – The protobuf for the mutations being saved.
  • transaction_id (string) – The transaction ID returned from begin_transaction(). If not passed, the commit will be non-transactional.
Return type:

gcloud.datastore._datastore_v1_pb2.MutationResult.
Returns’:

the result protobuf for the mutation.
lookup(dataset_id, key_pbs, eventual=False, transaction_id=None)[source]#

Lookup keys from a dataset in the Cloud Datastore.

Maps the DatastoreService.Lookup protobuf RPC.

This method deals only with protobufs (gcloud.datastore._datastore_v1_pb2.Key and gcloud.datastore._datastore_v1_pb2.Entity) and is used under the hood in gcloud.datastore.get():

>>> from gcloud import datastore
>>> datastore.set_defaults()
>>> key = datastore.Key('MyKind', 1234, dataset_id='dataset-id')
>>> datastore.get([key])
[<Entity object>]

Using the connection class directly:

>>> connection.lookup('dataset-id', [key.to_protobuf()])
[<Entity protobuf>]
Parameters:
  • dataset_id (string) – The ID of the dataset to look up the keys.
  • key_pbs (list of gcloud.datastore._datastore_v1_pb2.Key) – The keys to retrieve from the datastore.
  • eventual (boolean) – If False (the default), request STRONG read consistency. If True, request EVENTUAL read consistency.
  • transaction_id (string) – If passed, make the request in the scope of the given transaction. Incompatible with eventual==True.
Return type:

tuple
Returns:

A triple of (results, missing, deferred) where both results and missing are lists of gcloud.datastore._datastore_v1_pb2.Entity and deferred is a list of gcloud.datastore._datastore_v1_pb2.Key.
rollback(dataset_id, transaction_id)[source]#

Rollback the connection’s existing transaction.

Maps the DatastoreService.Rollback protobuf RPC.

Parameters:
  • dataset_id (string) – The ID of the dataset to which the transaction belongs.
  • transaction_id (string) – The transaction ID returned from begin_transaction().
run_query(dataset_id, query_pb, namespace=None, eventual=False, transaction_id=None)[source]#

Run a query on the Cloud Datastore.

Maps the DatastoreService.RunQuery protobuf RPC.

Given a Query protobuf, sends a runQuery request to the Cloud Datastore API and returns a list of entity protobufs matching the query.

You typically wouldn’t use this method directly, in favor of the gcloud.datastore.query.Query.fetch() method.

Under the hood, the gcloud.datastore.query.Query class uses this method to fetch data:

>>> from gcloud import datastore

>>> datastore.set_defaults()

>>> query = datastore.Query(kind='MyKind')
>>> query.add_filter('property', '=', 'val')

Using the query’s fetch_page method...

>>> entities, cursor, more_results = query.fetch_page()
>>> entities
[<list of Entity unmarshalled from protobuf>]
>>> cursor
<string containing cursor where fetch stopped>
>>> more_results
<boolean of more results>

Under the hood this is doing...

>>> connection.run_query('dataset-id', query.to_protobuf())
[<list of Entity Protobufs>], cursor, more_results, skipped_results
Parameters:
  • dataset_id (string) – The ID of the dataset over which to run the query.
  • query_pb (gcloud.datastore._datastore_v1_pb2.Query) – The Protobuf representing the query to run.
  • namespace (string) – The namespace over which to run the query.
  • eventual (boolean) – If False (the default), request STRONG read consistency. If True, request EVENTUAL read consistency.
  • transaction_id (string) – If passed, make the request in the scope of the given transaction. Incompatible with eventual==True.

Interacting with the API#

Methods for interacting with Google Cloud Datastore.

Allows interacting with the datastore via user-friendly Key, Entity and Query objects rather than via protobufs.

gcloud.datastore.api.allocate_ids(incomplete_key, num_ids, connection=None)[source]#

Allocates a list of IDs from a partial key.

Parameters:
Return type:

list of gcloud.datastore.key.Key
Returns:

The (complete) keys allocated with incomplete_key as root.
Raises:

ValueError if incomplete_key is not a partial key.
gcloud.datastore.api.delete(keys, connection=None, dataset_id=None)[source]#

Delete the keys in the Cloud Datastore.

Parameters:
Raises:

EnvironmentError if connection or dataset_id not passed, and cannot be inferred from the environment. ValueError if one or more keys has a dataset ID not matching the passed / inferred dataset ID.
gcloud.datastore.api.get(keys, missing=None, deferred=None, connection=None, dataset_id=None)[source]#

Retrieves entities, along with their attributes.

Parameters:
  • keys (list of gcloud.datastore.key.Key) – The keys to be retrieved from the datastore.
  • missing (an empty list or None.) – If a list is passed, the key-only entities returned by the backend as “missing” will be copied into it. Use only as a keyword param.
  • deferred (an empty list or None.) – If a list is passed, the keys returned by the backend as “deferred” will be copied into it. Use only as a keyword param.
  • connection (gcloud.datastore.connection.Connection) – Optional. The connection used to connect to datastore. If not passed, inferred from the environment.
  • dataset_id (gcloud.datastore.connection.Connection) – Optional. The dataset ID used to connect to datastore. If not passed, inferred from the environment.
Return type:

list of gcloud.datastore.entity.Entity
Returns:

The requested entities.
Raises:

EnvironmentError if connection or dataset_id not passed, and cannot be inferred from the environment. ValueError if one or more of keys has a dataset ID which does not match the passed / inferred dataset ID.
gcloud.datastore.api.put(entities, connection=None, dataset_id=None)[source]#

Save the entities in the Cloud Datastore.

Parameters:
Raises:

EnvironmentError if connection or dataset_id not passed, and cannot be inferred from the environment. ValueError if one or more entities has a key with a dataset ID not matching the passed / inferred dataset ID.

Helper functions#

Helper functions for dealing with Cloud Datastore’s Protobuf API.

The non-private functions are part of the API.

gcloud.datastore.helpers.entity_from_protobuf(pb)[source]#

Factory method for creating an entity based on a protobuf.

The protobuf should be one returned from the Cloud Datastore Protobuf API.

Parameters:pb (gcloud.datastore._datastore_v1_pb2.Entity) – The Protobuf representing the entity.
Return type:gcloud.datastore.entity.Entity
Returns:The entity derived from the protobuf.
gcloud.datastore.helpers.key_from_protobuf(pb)[source]#

Factory method for creating a key based on a protobuf.

The protobuf should be one returned from the Cloud Datastore Protobuf API.

Parameters:pb (gcloud.datastore._datastore_v1_pb2.Key) – The Protobuf representing the key.
Return type:gcloud.datastore.key.Key
Returns:a new Key instance