分散カウンタ

多くのリアルタイムアプリにはカウンタとして働くドキュメントがあります。たとえば、投稿の「いいね」や特定のアイテムの「お気に入り」をカウントしたい場合があります。

Cloud Firestore では、1 つのドキュメントを無制限の頻度で更新することはできません。単一ドキュメントに基づくカウンタがあり、高い頻度でインクリメントを行うと、最終的にドキュメントの更新で競合が発生します。1 つのドキュメントの更新をご覧ください。

解決策: 分散カウンタ

カウンタをもっと頻繁に更新できるようにするには、分散カウンタを作成します。各分散カウンタは「シャード」のサブコレクションを持つドキュメントであり、そのカウンタの値はシャードの値の合計です。

書き込みスループットはシャードの数に比例して増加するため、10 個のシャードを持つ分散カウンタは従来のカウンタの 10 倍の書き込みを処理できます。

ウェブ

// counters/${ID}
{
  "num_shards": NUM_SHARDS,
  "shards": [subcollection]
}

// counters/${ID}/shards/${NUM}
{
  "count": 123
}

Swift

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

// counters/${ID}
struct Counter {
  let numShards: Int

  init(numShards: Int) {
    self.numShards = numShards
  }
}

// counters/${ID}/shards/${NUM}
struct Shard {
  let count: Int

  init(count: Int) {
    self.count = count
  }
}SolutionCountersViewController.swift

Objective-C

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

// counters/${ID}
@interface FIRCounter : NSObject
@property (nonatomic, readonly) NSInteger shardCount;
@end

@implementation FIRCounter
- (instancetype)initWithShardCount:(NSInteger)shardCount {
  self = [super init];
  if (self != nil) {
    _shardCount = shardCount;
  }
  return self;
}
@end

// counters/${ID}/shards/${NUM}
@interface FIRShard : NSObject
@property (nonatomic, readonly) NSInteger count;
@end

@implementation FIRShard
- (instancetype)initWithCount:(NSInteger)count {
  self = [super init];
  if (self != nil) {
    _count = count;
  }
  return self;
}
@endFIRSolutionCountersViewController.m

Kotlin+KTX

// counters/${ID}
data class Counter(var numShards: Int)

// counters/${ID}/shards/${NUM}
data class Shard(var count: Int)SolutionCounters.kt

Java

// counters/${ID}
public class Counter {
    int numShards;

    public Counter(int numShards) {
        this.numShards = numShards;
    }
}

// counters/${ID}/shards/${NUM}
public class Shard {
    int count;

    public Shard(int count) {
        this.count = count;
    }
}SolutionCounters.java

Python

import random

from google.cloud import firestore


class Shard:
    """
    A shard is a distributed counter. Each shard can support being incremented
    once per second. Multiple shards are needed within a Counter to allow
    more frequent incrementing.
    """

    def __init__(self):
        self._count = 0

    def to_dict(self):
        return {"count": self._count}


class Counter:
    """
    A counter stores a collection of shards which are
    summed to return a total count. This allows for more
    frequent incrementing than a single document.
    """

    def __init__(self, num_shards):
        self._num_shards = num_shards
distributed_counters.py

Python

import random

from google.cloud import firestore


class Shard:
    """
    A shard is a distributed counter. Each shard can support being incremented
    once per second. Multiple shards are needed within a Counter to allow
    more frequent incrementing.
    """

    def __init__(self):
        self._count = 0

    def to_dict(self):
        return {"count": self._count}


class Counter:
    """
    A counter stores a collection of shards which are
    summed to return a total count. This allows for more
    frequent incrementing than a single document.
    """

    def __init__(self, num_shards):
        self._num_shards = num_shards
distributed_counters.py

Node.js

該当しない場合は、下のカウンタの増分スニペットをご覧ください。

Go

import (
	"context"
	"fmt"
	"math/rand"
	"strconv"

	"cloud.google.com/go/firestore"
	"google.golang.org/api/iterator"
)

// Counter is a collection of documents (shards)
// to realize counter with high frequency.
type Counter struct {
	numShards int
}

// Shard is a single counter, which is used in a group
// of other shards within Counter.
type Shard struct {
	Count int
}
solution_counters.go

PHP

該当しない場合は、下のカウンタの初期化スニペットをご覧ください。

C#

/// <summary>
/// Shard is a document that contains the count.
/// </summary>
[FirestoreData]
public class Shard
{
    [FirestoreProperty(name: "count")]
    public int Count { get; set; }
}Program.cs

次のコードは、分散カウンタを初期化します。

ウェブ

function createCounter(ref, num_shards) {
    var batch = db.batch();

    // Initialize the counter document
    batch.set(ref, { num_shards: num_shards });

    // Initialize each shard with count=0
    for (let i = 0; i < num_shards; i++) {
        const shardRef = ref.collection('shards').doc(i.toString());
        batch.set(shardRef, { count: 0 });
    }

    // Commit the write batch
    return batch.commit();
}test.solution-counters.js

Swift

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

func createCounter(ref: DocumentReference, numShards: Int) async {
  do {
    try await ref.setData(["numShards": numShards])
    for i in 0...numShards {
      try await ref.collection("shards").document(String(i)).setData(["count": 0])
    }
  } catch {
    // ...
  }
}SolutionCountersViewController.swift

Objective-C

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

- (void)createCounterAtReference:(FIRDocumentReference *)reference
                      shardCount:(NSInteger)shardCount {
  [reference setData:@{ @"numShards": @(shardCount) } completion:^(NSError * _Nullable error) {
    for (NSInteger i = 0; i < shardCount; i++) {
      NSString *shardName = [NSString stringWithFormat:@"%ld", (long)shardCount];
      [[[reference collectionWithPath:@"shards"] documentWithPath:shardName]
          setData:@{ @"count": @(0) }];
    }
  }];
}FIRSolutionCountersViewController.m

Kotlin+KTX

fun createCounter(ref: DocumentReference, numShards: Int): Task<Void> {
    // Initialize the counter document, then initialize each shard.
    return ref.set(Counter(numShards))
        .continueWithTask { task ->
            if (!task.isSuccessful) {
                throw task.exception!!
            }

            val tasks = arrayListOf<Task<Void>>()

            // Initialize each shard with count=0
            for (i in 0 until numShards) {
                val makeShard = ref.collection("shards")
                    .document(i.toString())
                    .set(Shard(0))

                tasks.add(makeShard)
            }

            Tasks.whenAll(tasks)
        }
}SolutionCounters.kt

Java

public Task<Void> createCounter(final DocumentReference ref, final int numShards) {
    // Initialize the counter document, then initialize each shard.
    return ref.set(new Counter(numShards))
            .continueWithTask(new Continuation<Void, Task<Void>>() {
                @Override
                public Task<Void> then(@NonNull Task<Void> task) throws Exception {
                    if (!task.isSuccessful()) {
                        throw task.getException();
                    }

                    List<Task<Void>> tasks = new ArrayList<>();

                    // Initialize each shard with count=0
                    for (int i = 0; i < numShards; i++) {
                        Task<Void> makeShard = ref.collection("shards")
                                .document(String.valueOf(i))
                                .set(new Shard(0));

                        tasks.add(makeShard);
                    }

                    return Tasks.whenAll(tasks);
                }
            });
}SolutionCounters.java

Python

def init_counter(self, doc_ref):
    """
    Create a given number of shards as
    subcollection of specified document.
    """
    col_ref = doc_ref.collection("shards")

    # Initialize each shard with count=0
    for num in range(self._num_shards):
        shard = Shard()
        col_ref.document(str(num)).set(shard.to_dict())
distributed_counters.py

Python

async def init_counter(self, doc_ref):
    """
    Create a given number of shards as
    subcollection of specified document.
    """
    col_ref = doc_ref.collection("shards")

    # Initialize each shard with count=0
    for num in range(self._num_shards):
        shard = Shard()
        await col_ref.document(str(num)).set(shard.to_dict())
distributed_counters.py

Node.js

該当しない場合は、下のカウンタの増分スニペットをご覧ください。

Go


// initCounter creates a given number of shards as
// subcollection of specified document.
func (c *Counter) initCounter(ctx context.Context, docRef *firestore.DocumentRef) error {
	colRef := docRef.Collection("shards")

	// Initialize each shard with count=0
	for num := 0; num < c.numShards; num++ {
		shard := Shard{0}

		if _, err := colRef.Doc(strconv.Itoa(num)).Set(ctx, shard); err != nil {
			return fmt.Errorf("Set: %w", err)
		}
	}
	return nil
}
solution_counters.go

PHP

$numShards = 10;
$ref = $db->collection('samples/php/distributedCounters');
for ($i = 0; $i < $numShards; $i++) {
    $doc = $ref->document((string) $i);
    $doc->set(['Cnt' => 0]);
}solution_sharded_counter_create.php

C#

/// <summary>
/// Create a given number of shards as a
/// subcollection of specified document.
/// </summary>
/// <param name="docRef">The document reference <see cref="DocumentReference"/></param>
private static async Task CreateCounterAsync(DocumentReference docRef, int numOfShards)
{
    CollectionReference colRef = docRef.Collection("shards");
    var tasks = new List<Task>();
    // Initialize each shard with Count=0
    for (var i = 0; i < numOfShards; i++)
    {
        tasks.Add(colRef.Document(i.ToString()).SetAsync(new Shard() { Count = 0 }));
    }
    await Task.WhenAll(tasks);
}Program.cs

Ruby

# project_id = "Your Google Cloud Project ID"
# num_shards = "Number of shards for distributed counter"
# collection_path = "shards"

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new project_id: project_id

shards_ref = firestore.col collection_path

# Initialize each shard with count=0
num_shards.times do |i|
  shards_ref.doc(i).set({ count: 0 })
end

puts "Distributed counter shards collection created."distributed_counters.rb

カウンタをインクリメントするには、シャードをランダムに選択します。

ウェブ

function incrementCounter(ref, num_shards) {
    // Select a shard of the counter at random
    const shard_id = Math.floor(Math.random() * num_shards).toString();
    const shard_ref = ref.collection('shards').doc(shard_id);

    // Update count
    return shard_ref.update("count", firebase.firestore.FieldValue.increment(1));
}test.solution-counters.js

Swift

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

func incrementCounter(ref: DocumentReference, numShards: Int) {
  // Select a shard of the counter at random
  let shardId = Int(arc4random_uniform(UInt32(numShards)))
  let shardRef = ref.collection("shards").document(String(shardId))

  shardRef.updateData([
    "count": FieldValue.increment(Int64(1))
  ])
}SolutionCountersViewController.swift

Objective-C

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

- (void)incrementCounterAtReference:(FIRDocumentReference *)reference
                         shardCount:(NSInteger)shardCount {
  // Select a shard of the counter at random
  NSInteger shardID = (NSInteger)arc4random_uniform((uint32_t)shardCount);
  NSString *shardName = [NSString stringWithFormat:@"%ld", (long)shardID];
  FIRDocumentReference *shardReference =
      [[reference collectionWithPath:@"shards"] documentWithPath:shardName];

  [shardReference updateData:@{
    @"count": [FIRFieldValue fieldValueForIntegerIncrement:1]
  }];
}FIRSolutionCountersViewController.m

Kotlin+KTX

fun incrementCounter(ref: DocumentReference, numShards: Int): Task<Void> {
    val shardId = Math.floor(Math.random() * numShards).toInt()
    val shardRef = ref.collection("shards").document(shardId.toString())

    return shardRef.update("count", FieldValue.increment(1))
}SolutionCounters.kt

Java

public Task<Void> incrementCounter(final DocumentReference ref, final int numShards) {
    int shardId = (int) Math.floor(Math.random() * numShards);
    DocumentReference shardRef = ref.collection("shards").document(String.valueOf(shardId));

    return shardRef.update("count", FieldValue.increment(1));
}SolutionCounters.java

Python

def increment_counter(self, doc_ref):
    """Increment a randomly picked shard."""
    doc_id = random.randint(0, self._num_shards - 1)

    shard_ref = doc_ref.collection("shards").document(str(doc_id))
    return shard_ref.update({"count": firestore.Increment(1)})
distributed_counters.py

Python

async def increment_counter(self, doc_ref):
    """Increment a randomly picked shard."""
    doc_id = random.randint(0, self._num_shards - 1)

    shard_ref = doc_ref.collection("shards").document(str(doc_id))
    return await shard_ref.update({"count": firestore.Increment(1)})
distributed_counters.py

Node.js

function incrementCounter(docRef, numShards) {
  const shardId = Math.floor(Math.random() * numShards);
  const shardRef = docRef.collection('shards').doc(shardId.toString());
  return shardRef.set({count: FieldValue.increment(1)}, {merge: true});
}solution-counters.js

Go


// incrementCounter increments a randomly picked shard.
func (c *Counter) incrementCounter(ctx context.Context, docRef *firestore.DocumentRef) (*firestore.WriteResult, error) {
	docID := strconv.Itoa(rand.Intn(c.numShards))

	shardRef := docRef.Collection("shards").Doc(docID)
	return shardRef.Update(ctx, []firestore.Update{
		{Path: "Count", Value: firestore.Increment(1)},
	})
}
solution_counters.go

PHP

$ref = $db->collection('samples/php/distributedCounters');
$numShards = 0;
$docCollection = $ref->documents();
foreach ($docCollection as $doc) {
    $numShards++;
}
$shardIdx = random_int(0, max(1, $numShards) - 1);
$doc = $ref->document((string) $shardIdx);
$doc->update([
    ['path' => 'Cnt', 'value' => FieldValue::increment(1)]
]);solution_sharded_counter_increment.php

C#

/// <summary>
/// Increment a randomly picked shard by 1.
/// </summary>
/// <param name="docRef">The document reference <see cref="DocumentReference"/></param>
/// <returns>The <see cref="Task"/></returns>
private static async Task IncrementCounterAsync(DocumentReference docRef, int numOfShards)
{
    int documentId;
    lock (s_randLock)
    {
        documentId = s_rand.Next(numOfShards);
    }
    var shardRef = docRef.Collection("shards").Document(documentId.ToString());
    await shardRef.UpdateAsync("count", FieldValue.Increment(1));
}Program.cs

Ruby

# project_id = "Your Google Cloud Project ID"
# num_shards = "Number of shards for distributed counter"
# collection_path = "shards"

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new project_id: project_id

# Select a shard of the counter at random
shard_id = rand 0...num_shards
shard_ref = firestore.doc "#{collection_path}/#{shard_id}"

# increment counter
shard_ref.update({ count: firestore.field_increment(1) })

puts "Counter incremented."distributed_counters.rb

カウントの合計を取得するには、すべてのシャードに対してクエリを実行して count フィールドを合計します。

ウェブ

function getCount(ref) {
    // Sum the count of each shard in the subcollection
    return ref.collection('shards').get().then((snapshot) => {
        let total_count = 0;
        snapshot.forEach((doc) => {
            total_count += doc.data().count;
        });

        return total_count;
    });
}test.solution-counters.js

Swift

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

func getCount(ref: DocumentReference) async {
  do {
    let querySnapshot = try await ref.collection("shards").getDocuments()
    var totalCount = 0
    for document in querySnapshot.documents {
      let count = document.data()["count"] as! Int
      totalCount += count
    }

    print("Total count is \(totalCount)")
  } catch {
    // handle error
  }
}SolutionCountersViewController.swift

Objective-C

注: このプロダクトは、watchOS と App Clip の各ターゲットでは利用できません。

- (void)getCountWithReference:(FIRDocumentReference *)reference {
  [[reference collectionWithPath:@"shards"]
      getDocumentsWithCompletion:^(FIRQuerySnapshot *snapshot,
                                   NSError *error) {
        NSInteger totalCount = 0;
        if (error != nil) {
          // Error getting shards
          // ...
        } else {
          for (FIRDocumentSnapshot *document in snapshot.documents) {
            NSInteger count = [document[@"count"] integerValue];
            totalCount += count;
          }

          NSLog(@"Total count is %ld", (long)totalCount);
        }
  }];
}FIRSolutionCountersViewController.m

Kotlin+KTX

fun getCount(ref: DocumentReference): Task<Int> {
    // Sum the count of each shard in the subcollection
    return ref.collection("shards").get()
        .continueWith { task ->
            var count = 0
            for (snap in task.result!!) {
                val shard = snap.toObject<Shard>()
                count += shard.count
            }
            count
        }
}SolutionCounters.kt

Java

public Task<Integer> getCount(final DocumentReference ref) {
    // Sum the count of each shard in the subcollection
    return ref.collection("shards").get()
            .continueWith(new Continuation<QuerySnapshot, Integer>() {
                @Override
                public Integer then(@NonNull Task<QuerySnapshot> task) throws Exception {
                    int count = 0;
                    for (DocumentSnapshot snap : task.getResult()) {
                        Shard shard = snap.toObject(Shard.class);
                        count += shard.count;
                    }
                    return count;
                }
            });
}SolutionCounters.java

Python

def get_count(self, doc_ref):
    """Return a total count across all shards."""
    total = 0
    shards = doc_ref.collection("shards").list_documents()
    for shard in shards:
        total += shard.get().to_dict().get("count", 0)
    return total
distributed_counters.py

Python

async def get_count(self, doc_ref):
    """Return a total count across all shards."""
    total = 0
    shards = doc_ref.collection("shards").list_documents()
    async for shard in shards:
        total += (await shard.get()).to_dict().get("count", 0)
    return total
distributed_counters.py

Node.js

async function getCount(docRef) {
  const querySnapshot = await docRef.collection('shards').get();
  const documents = querySnapshot.docs;

  let count = 0;
  for (const doc of documents) {
    count += doc.get('count');
  }
  return count;
}solution-counters.js

Go


// getCount returns a total count across all shards.
func (c *Counter) getCount(ctx context.Context, docRef *firestore.DocumentRef) (int64, error) {
	var total int64
	shards := docRef.Collection("shards").Documents(ctx)
	for {
		doc, err := shards.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return 0, fmt.Errorf("Next: %w", err)
		}

		vTotal := doc.Data()["Count"]
		shardCount, ok := vTotal.(int64)
		if !ok {
			return 0, fmt.Errorf("firestore: invalid dataType %T, want int64", vTotal)
		}
		total += shardCount
	}
	return total, nil
}
solution_counters.go

PHP

$result = 0;
$docCollection = $db->collection('samples/php/distributedCounters')->documents();
foreach ($docCollection as $doc) {
    $result += $doc->data()['Cnt'];
}solution_sharded_counter_get.php

C#

/// <summary>
/// Get total count across all shards.
/// </summary>
/// <param name="docRef">The document reference <see cref="DocumentReference"/></param>
/// <returns>The <see cref="int"/></returns>
private static async Task<int> GetCountAsync(DocumentReference docRef)
{
    var snapshotList = await docRef.Collection("shards").GetSnapshotAsync();
    return snapshotList.Sum(shard => shard.GetValue<int>("count"));
}Program.cs

Ruby

# project_id = "Your Google Cloud Project ID"
# collection_path = "shards"

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new project_id: project_id

shards_ref = firestore.col_group collection_path

count = 0
shards_ref.get do |doc_ref|
  count += doc_ref[:count]
end

puts "Count value is #{count}."distributed_counters.rb

制限事項

上記の解決策は Cloud Firestore で共有カウンタを作成するスケーラブルな方法ですが、次の制限に注意する必要があります。

シャードの数 - シャードの数は分散カウンタのパフォーマンスに影響を与えます。シャードが少なすぎると、再試行が必要となるトランザクションが発生し、結果的に書き込みが遅くなります。シャードが多すぎると、読み取り速度が遅くなり、コストがかかります。少ない頻度で更新される別のロールアップドキュメントでカウンタの合計を維持し、合計を取得する際はクライアントにそのドキュメントから読み取らせるようにすることで、読み取りコストを相殺できます。ただし、その反面、クライアントは更新直後にすべてのシャードを読み取って合計を計算することはできなくなります。代わりに、ロールアップドキュメントが更新されるまで待機しなければなりません。
コスト - シャードのサブコレクション全体を読み込まなければならないため、カウンタ値を読み取るコストはシャードの数に比例して増加します。