Google adds Cloud Logging for BigQuery Python UDFs

Google has added a logging feature for Python User-Defined Functions in BigQuery, sending structured logs from Python UDFs to Cloud Logging.

Developers can use Python's standard logging library inside BigQuery UDFs without external plugins or custom connectors. Google framed the addition around operational monitoring and debugging, including data pipeline checks that can continue running when individual records fail validation.

It illustrated the feature with an eCommerce event ingestion workflow in which a Python UDF parses incoming JSON payloads and checks for required fields. Valid records move to a destination table, while malformed or incomplete rows are logged and return null values so the wider query does not fail.

In the example, a function called safe_parse_json uses Python code within BigQuery to inspect each payload. If the payload is empty, syntactically invalid or missing a required key, the function writes a structured error entry through the logging library and returns no result for that row.

This allows a query to complete while excluding problematic records from the clean output table. In Google's sample dataset, valid events were written for records containing user identifiers and event names, while corrupted JSON and a payload missing the user_id field were sent to logging output.

How it works

Google described the workflow in three stages. First, the UDF performs inline validation. Second, Cloud Monitoring provides real-time alerting through a log-based metric that notifies teams when error levels pass a set threshold. Third, engineers use SQL-based triage and replay by querying logged records through BigQuery Log Analytics and reprocessing data after fixing the underlying issue.

To enable logging in a session, users set UDF properties to turn on debug output before running the query. Once enabled, logs from managed routine invocations can be filtered in Logs Explorer by resource type and severity, allowing teams to create alerts for error conditions.

Google also explained how logs can be queried through a linked _AllLogs view in BigQuery. In the example query, engineers select the timestamp, severity and fields such as error type, missing key and raw payload from JSON log entries, then sort them to inspect the latest failures.

Recovery path

The addition is partly aimed at cases where bad data should be isolated rather than allowed to stop an entire workload. By preserving raw payloads in Cloud Logging, teams can use SQL to recover quarantined records and insert corrected versions into production tables after upstream fixes are applied.

Google's recovery example shows a row with a missing required field being replayed into the clean events table with a fallback user identifier and a recovered status. The final result combines original valid records with repaired entries in a single destination table.

Beyond ingestion checks, the same pattern can be used for side-by-side comparisons between legacy and new business logic, auditing personal data redaction and monitoring model inference drift. In each case, the UDF can return a production result while writing diagnostic information to Cloud Logging.

Google also cautioned users on cost and access control. Logs should be written selectively in high-throughput pipelines to limit ingestion volume, and credentials or secrets should not be placed in log entries because Cloud Logging is visible to users with Logging Viewer permissions.

The feature extends BigQuery's Python UDF model by giving developers a native way to capture row-level diagnostics inside SQL-driven data processing. In Google's example, the query completed successfully while malformed records were isolated through log entries rather than stopping the workload.