Logs generated by applications and services can provide an immense amount of information about how your deployment is running and the experiences your users are having as they interact with the products and services. But as deployments grow more complex, gleaning insights from this data becomes more challenging. Logs come from an increasing number of sources, so they can be hard to collate and query for useful information. And building, operating and maintaining your own infrastructure to analyze log data at scale requires extensive expertise in running distributed systems and storage. Today, we’re introducing a new solution paper and reference implementation that will show how you can process logs from multiple sources and extract meaningful information by using Google Cloud Platform and Google Cloud Dataflow.

Log processing typically involves some combination of the following activities:

• Configuring applications and services
• Collecting and capturing log files
• Storing and managing log data
• Processing and extracting data
• Persisting insights

Each of those components has it’s own scaling and management challenges, often using different approaches at different times. These sorts of challenges can slow down the generation of meaningful, actionable information from your log data.

Cloud Platform provides a number of services that can help you to address these challenges. You can use Cloud Logging to collect logs from applications and services, and then store them in Google Cloud Storage buckets or stream them to Pub/Sub topics. Dataflow can read from Cloud Storage or Pub/Sub (and many more), process log data, extract and transform metadata and compute aggregations. You can persist the output from Dataflow in BigQuery, where it can be analyzed or reviewed anytime. These mechanisms are offered as managed services—meaning they can scale when needed. That also means that you don't need to worry about provisioning resources up front.

The solution paper and reference implementation describe how you can use Dataflow to process log data from multiple sources and persist findings directly in BigQuery. You’ll learn how to configure Cloud Logging to collect logs from applications running in Container Engine, how to export those logs to Cloud Storage, and how to execute the Dataflow processing job. In addition, the solution shows you how to reconfigure Cloud Logging to use Pub/Sub to stream data directly to Dataflow, so you can process logs in real-time.

Check out the Processing Logs at Scale using Cloud Dataflow solution to learn how to combine logging, storage, processing and persistence into a scalable log processing approach. Then take a look at the reference implementation tutorial on Github to deploy a complete end-to-end working example. Feedback is welcome and appreciated; comment here, submit a pull request, create an issue, or find me on Twitter @crcsmnky and let me know how I can help.

- Posted by Sandeep Parikh, Google Solutions Architect

Firebase is a platform for building Android, iOS and web-based mobile apps, offering real-time data storage and automatic synchronization across devices. But what about when you need to run backend processes on the data?

By connecting an App Engine application to your Firebase database, you can perform complex logic on the data without having to manage synchronization and updates; Firebase handles that for you.

Updates in the Android client release of Firebase 2.4.0 make it easy to access a Firebase database from an App Engine application.

The tutorial, Use Firebase and Google App Engine in an Android App, walks you through the steps to create an Android app that stores a to-do list in Firebase, and uses backend logic running on App Engine to send daily reminder emails.

In the process of working through the tutorial, you’ll learn how to use the following technologies:

• Firebase — a platform for building mobile apps, offering realtime data storage and synchronization, user authentication, and more.
• Android Studio — an Android development environment based on IntelliJ IDEA.
• Cloud Tools for Android Studio — a set of tools included with Android Studio, that integrate with Google Cloud Platform services.
• Google App Engine — an environment for running application code in the cloud; Platform as a service (PaaS).

- Posted by Benjamin Wulfe, Firebase

“Finally she sat in the wee little chair and it was just right. But Goldilocks rocked so hard in the chair that it broke into pieces!” The Story of the Three Bears by Robert Southey (1837)

Getting auto scaling just right is one of the hardest things about cloud apps. Scale too slowly and you can't meet customer demand. Scale too quickly and you pay for more than you need. For new applications in particular, you might need to have strong prediction skills to provision the correct number of virtual machine (VM) instances to meet customer demand.

We don’t want your Google Compute Engine app to break when it faces increased demand but, just as importantly, we’d like to ensure you don’t pay for excess capacity when you don't need it.

Sure, you could estimate peak load and provision for it. However, if your application’s peak load only occurs during the holiday period, your infrastructure will be largely unused for the majority of the year. Alternatively, you could provision for estimated average use. But then, by definition, half the time your infrastructure will be over-provisioned and half the time it will be under-provisioned, so you end up disappointing either your customers or your managers all the time.

To learn how to automatically scale Compute Engine instances to meet demand, read the newly published article Auto Scaling on the Google Cloud Platform. In this paper, we explore a framework for a Google App Engine application that scales Compute Engine instances up or down as demand increases or decreases. We designed the framework to be extremely flexible, so that you can adjust it to meet your business needs.

To get started, download the sample apps:

• Orchestrator and Status Publisher, a Java autoscaling framework for App Engine and Compute Engine.
• Smashpix, an Android and App Engine application for uploading and processing images.

Orchestrating an App Engine+ Compute Engine application

Unlike Goldilock’s adventures, this isn’t a fairy tale. You can download the apps and start implementing today.

-Posted by Kathrin Probst, Solutions Architect

Once you get your data into Google BigQuery, you don’t have to worry about running out of machine capacity, because you use Google’s machines as if they were your own. But what if you want to transform your source data before putting it into BigQuery and you don’t have the server capacity to handle the transformation? In this case, how about using Google Compute Engine to run your Extract, Transform and Load (ETL) processing? To learn how, read our paper Getting Started With Google BigQuery. To get started, download the sample ETL tool for Google Compute Engine from GitHub.

The sample ETL tool is an application that automates the steps of getting the Google Compute Engine instance up and running, and installing the software you need to rapidly design, create and execute the ETL workflow. The application includes a sample workflow that uses KNIME to help you understand the entire process, as shown here:

If you already have an established process for performing the ETL process to prepare the data and load it into Google Cloud Storage, but need a reliable way to load the data from there into BigQuery, we have a another sample application to help you. The Automated File Loader for BigQuery sample app demonstrates how to automate data loading from Google Cloud Storage to BigQuery.

This application uses the Cloud Storage Object Change Notification API to receive notifications that files have been uploaded to a bucket in Google Cloud Storage, then uses the BigQuery API to load the data from the bucket into BigQuery. Download it now from GitHub.

Both these sample applications accompany the article Getting Started With Google BigQuery, which provides an overview of the end-to-end process from loading data into BigQuery to visualization, and design practices that should be considered when using BigQuery.

-Posted by Wally Yau, Solutions Architect

Do you need an environment that seamlessly scales to meet demand? Do multiple teams develop your app using a variety of programming languages? Do you need to mix managed applications and self-serve virtual machines running a variety of separate services? Google Cloud Platform has you covered on all fronts.

Take a look at our latest paper to learn how to overcome the challenges of deploying large applications in Cloud Platform. Our paper discusses a solution that shows how to use the right tools for:

• Different developer expertise
• Different data storage requirements
• Interactive and batch processing requirements
• Custom analysis engines
• Logically separate business components

Our paper walks through a real-world scenario of building a video sharing community. In this scenario, users upload videos from a mobile phone, the content is transcoded for playback in multiple formats and is presented on a web community site for watching and commenting.

Our sample solution showcases Google App Engine applications written in PHP, Java and Python. We use Google Cloud SQL for storing video metadata, to capitalize on the popularity and reduced learning curve of MySQL, and we store the videos in Google Cloud Storage for speed and scalability. The mobile client is written in Java and runs on Android. Hadoop clusters running on Google Compute Engine analyze and add sentiments to users' comments. And the whole shebang is orchestrated by another App Engine application. To learn how to build a solution like this yourself, get started now by reading all about it.

-Posted by Brian Lynch, Solutions Architect

Whether you are a startup considering Google Cloud Platform for a new venture or an existing company who wants to breathe new life into the way you do business, we’re here to help. We have experienced solutions architects and software engineers developing architecture frameworks and adaptable solutions to help you get started on Cloud Platform.

Based on the strong interests we've heard from customers, we've created resources for you that focus on mobile, gaming, big data, big science, and complex web apps to start. We are providing sample applications, tools and architecture frameworks to make it as easy as possible for you to build apps and run your business on Cloud Platform.

Check out the solution scenarios and sample applications we have already published on the Cloud Platform resources page.

Here are just some of the articles we’ve already published:

Dedicated server gaming solution – In depth gaming solution, providing guidance on how to create a scalable and reliable online gaming solution

Getting started with Google BigQuery – Technical article providing a basic understanding of how Google BigQuery works, and an overview of the end-to-end process from loading data into BigQuery to visualization

Digital media asset management and sharing - Comprehensive example scenario that provides a technical deep-dive on how to use Cloud Platform to build a digital media asset management and sharing system

Also, take a look at some of the sample apps (with working source code) that we have already made available for you to download and extend:

Load balanced gaming server
Mobile shopping assistant

We have more technical articles, case studies and sample apps in the works, and we'll let you know about them as soon as they are ready.

Your first stop for all this great technical content is cloud.google.com/resources. Enjoy!

-Posted by Dora Hsu, Solutions Director