New Audit Log

This feature is currently available in Beta version

📘

The new version of Audit log that is described in this article is active only on the private and exclusive instances. If you are using the shared instance, please refer to the old Audit log article.

As security is a crucial concern to Bloomreach, we consider it necessary to allow our clients to have an overview of all activities performed in the Bloomreach Engagement app. Especially when an activity is suspicious, we want you to be able to detect and investigate it. Audit logs, capturing and storing records of user activity within Bloomreach Engagement, enable you to do just that.

What is Audit log?

Audit logs are chronological records of user activity within the Bloomreach Engagement application. They capture audit trails that help in investigations of unauthorized access, data leak, or GDPR incidents, and they help with understanding alerts of repeated incidents and whether company-specific policies are met.

In simple terms, Audit Logs do that by answering the questions of Who? Did What? When? And Where?

  • Who? - The actor that performed an action. This can be public access, authenticated user, API key or Bloomreach Engagement's internal process.
  • Did What? - What action was performed can be identified by a unique combination of a request method and request path. Some records contain references to a specific resource and additional information within a service data.
  • When? - Each action has a precise timestamp. Time within the Bloomreach Engagement platform is synchronized using Network Time Protocol (NTP).
  • Where? - Actions are logged from different parts and scope of the application. The scope is one of: instance, account, or a project. Service name refers to one of many internal services. The screen where an action was performed by users from a browser can be accessed by the referrer URL.

Enabling and accessing the Audit log

To enable Audit log, you have to contact your CSM or Bloomreach Engagement support and, unless previously included within the original order form, a Letter of instruction must be signed. Enabling the Audit log usually takes up to 3 business days.

Access is managed by a CSM utilizing GSuite Google Groups (separate from the Access management within the Bloomreach Engagement application). Only the members of this group will be able to access the data. You will need Google Identities to use the Google Groups. These identities are either Google's user accounts or service accounts.

For security reasons (as a protection from any kind of manipulation or tampering with the audit log), there is no direct access from the Application, and users will be granted read-only access to Google Cloud Storage bucket containing exported Audit log data, which means the data itself cannot be directly edited.

Different ways of accessing the Audit log

Accessing with Google Cloud SDK tools: gsutil

  1. You need to install gsutil first. gsutil performs all operations, including uploads and downloads, using HTTPS and transport-layer security (TLS).
  2. Then authenticate with a google service account. You need a service account private key JSON file.

Authentication

$ gcloud auth activate-service-account --key-file service_account_key.json

Listing files

$ gsutil ls  gs://iid-auditlog-storage/iid/2021/10/12

Downloading files

$ gsutil -m rsync gs://iid-auditlog-storage/iid/2021/10/12 /data/local/storage/10/12

If you would like to learn more about accessing Audit log through Google Drive SDKs tools, please refer to the article.

Accessing with direct API calls

The second option is to access the audit log through direct API calls. If you want to learn more about it, read the article. You can see the example for using Google Cloud SDK python libraries down below:

from google.cloud import storage
from google.oauth2 import service_account

key_path = "service_account_key.json"

credentials = service_account.Credentials.from_service_account_file(key_path, scopes=["https://www.googleapis.com/auth/cloud-platform"],)
client = storage.Client(credentials=credentials, project=credentials.project_id,)

# list files

blobs = storage_client.list_blobs('iid-auditlog-storage')
for blob in blobs:
    print(blob.name)  

# download and print in console

bucket = client.get_bucket('iid-auditlog-storage')
blob = bucket.get_blob('iid/2021/02/23/20210223T080000.000Z-0.jsonl.gz')

print(blob.download_as_string())

🚧

Unique Bucket name

Your bucket name included in the code is unique. You need to replace "iid" in the above examples with your unique instance identifier instance_id (three alphanumeric lowercase symbols). Bucket naming convention is instance_id + "-auditlog-storage" followed by year/ month/ day (path formatting).

How does Audit log work

Audit log tracks requests sent by application users. Results are stored in a separate GCS bucket and can be downloaded and processed by an automated system.

The Bloomreach Engagement app contains a component that verifies user access for all parts of the application. Every time a logged user performs an action, this component creates an event, which is then stored in specific storage. Some events are considered to be sensitive and are therefore supplemented with additional information (e.g. how many rows have been exported/imported etc.).

Buckets are named as [instance-auditlog-storage], where “instance” is the name of the instance in which the bucket is stored. The data in these buckets is organized in folders according to time in the following way: Year > Month > Day.

Files with audit log events are JSON one line. This means every event is noted on one line. Files are created every hour and each file contains new logs. Files are not updated after creation. Delayed records are written in a file with an incremental number suffix.

🚧

Files are available for download for at least 60 days. After this time period, they can be moved into our archive storage and are not accessible for download.

Understanding the Audit log

Correlating multiple records

Some actions may generate more records on different levels of the application. Different components add component-specific information to the logs as service data. The unique request ID is generated on the first touchpoint and then propagated to subsequent events.

Scope identification

All events contain a field that defines on which layer the event was created - connecting it with a specific project, account, or instance.

  • An action performed on a project level has the Scope type value “PROJECT” and Scope ID value is the project token
  • An action performed on an account level has the Scope type value “ACCOUNT” and Scope ID value is the account ID
  • An action performed on the instance level has the Scope type value “INSTANCE” and no Scope ID value

Correlating user activity

Use session ID to correlate and analyze the flow of user activity during a session. A new unique session ID is created for each login session and discarded after logout or session expiration. Methods that do not require authentication do not have a session ID.

Audit log record schema

Log records have a predefined schema and come in JSON format. Top-level schema is fixed which makes audit logs to be easily imported into all commonly used SIEMs (e.g. Splunk, Graylog).

Fixed schema fields are reliable, can be used as indexed columns, and shall be used for SIEM alerts. Drilling down, records may contain not fixed (untyped) fields that are meant to provide additional information, are useful during investigations, and may be used for less critical SIEM alerts.

Schema: Fixed fields

Fixed fields are always present. These include:

Field nameDescription
timestampThe timestamp of the operation, ISO 8601 datetime in UTC, e.g. 2020-07-09T11:30:57.239442245Z
requestmethod or path
statusHTTP status code, e.g. 403 for Forbidden actions
serviceNameThe internal name of the service within Bloomreach Engagement architecture
scopeTypeenum: INSTANCE, ACCOUNT, PROJECT
scopeIDA unique project or account identifier
requestIDA unique ID, useful for correlating individual records connected with single user action

📘

Scope-related fields (scopeType, scopeID) are present only for methods requiring authorization. Fields are not present for other methods which are public methods or require only authentication.

Schema: Optional fields

Optional fields are present only for some operations.

Field nameDescription
authenticationInfoidentity: operations performed by an authenticated actor, e.g. read/update user profile, two-factor activation, logout

type: e.g. USER, SYSTEM_SERVICE_ACCOUNT, BASIC_AUTH
authorizationInfoallowed: whether the action was allowed by an authorization check

permission: operations that require permission, e.g. customer data read
metadataclientIP: a remote IP address from where request was called

host: hostname (domain) for external requests

referrer: browser reported document referrer, useful to find the origin screen

session: session ID for requests within authenticated user session

userAgent: rowser reported User-Agent
resource_idEvery operation on a specific resource with internal ID, e.g. customer, scenario, banner, experiment.

Useful for, e.g. filtering, alerts

Schema: Service Data

ServiceData types:

  • GenericServiceData
  • PolicyDelta
  • AnonymizationServiceData

GenericServiceData
Generic service data provide some additional information only for informational use cases. This information may be useful during investigations. Info field is not fixed and should not be used for critical use cases.

Field nameDescription
@typeauditlog.GenericServiceData
info (untyped)JSON encoded string, informational field, schema is not fixed and it's subject to change
version_idEvery operation on a specific resource with enabled versioning history, e.g. reports create/update, scenario create/update/start
{"@type":"auditlog.GenericServiceData","info":"{\"action_types\":[\"ab-split\",\"batch-webhook-action\",\"condition\",\"limit\",\"now-trigger\",\"on-event-trigger\",\"planned-trigger\",\"repeated-trigger\",\"send-email-action\",\"set-property-action\",\"wait-action\"],\"webhooks\":{\"endpoints\":[\"https://aud.app.exponea.dev/\"]},\"webhook_has_general_consent\":false,\"email_has_general_consent\":false}","versionID":"5f3b83344785e844fa76b0fd"}

PolicyChange

Field nameDescription
@typeiam.PolicyUpdate
policyChangesA list of policy changes.

member: user ID of affected user

roleId: role ID that was changed; e.g. "LegacyProjectAdmin"

type: change type, enum:
- ADD: role has been granted
- UPDATE: role has been updated
- REMOVE: role has been revoked

oldExpireTIme: (optional) previously set role expiration timestamp

newExpireTime: (optional) newly set role expiration timestamp
updateTimeThe timestamp of the operation
userIDUser ID of the subject performing the operation
{"@type":"iam.PolicyUpdate","id":"5f3cdcc931b9028e985237c1","policyChanges":[{"member":"[email protected]","roleId":"PII","type":"ADD"}],"updateTime":"2020-08-19T08:03:21.529134238Z","userId":"[email protected]"}
{"@type":"iam.PolicyUpdate","id":"5f3633bc699b028f7121560b","policyChanges":[{"member":"[email protected]","roleId":"LegacyProjectAdmin","type":"REMOVE"},{"member":"[email protected]","roleId":"PII","type":"REMOVE"},{"member":"[email protected]","roleId":"LegacyAccountAdmin","type":"REMOVE"}],"updateTime":"2020-08-14T06:48:28.870266563Z","userId":"[email protected]"}
{"@type":"iam.PolicyUpdate","id":"5f3cf286a011be198cde4905","policyChanges":[{"member":"[email protected]","newExpireTime":"0001-01-01T00:00:00Z","oldExpireTime":"2020-08-22T22:00:00Z","roleId":"LegacyProjectAdmin","type":"UPDATE"}],"updateTime":"2020-08-19T09:36:06.174072194Z","userId":"[email protected]"}

AnonymizationServiceData

Field nameDescription
@typeauditlog.AnonymizationServiceData
requestsA list of the requested IDs.

ids: A list of the external customer IDs.
- name: ID name, e.g. "registered"
- value: ID value, e.g. "[email protected]"

ID name = "_id" for Bloomreach Engagement's internal customer ID
{"authenticationInfo":{"identity":"44zrbjouu8jdio64zv6hli4ax0hklk0lkbn5gn4bd151epdffj2mhwj1lteta4tr","type":"BASIC_AUTH"},"authorizationInfo":{"allowed":true},"instanceName":"aud","logName":"audit.v1.Log","metadata":{"clientIP":"34.76.115.106","host":"aud.api.exponea.dev","userAgent":"PostmanRuntime/7.26.2"},"request":{"@type":"http","method":"POST","path":"/data/v2/projects/a25641b8-dd39-11ea-b199-ae02a0152881/customers/anonymize_bulk"},"serviceData":{"@type":"auditlog.AnonymizationServiceData","requests":[{"ids":[{"name":"registered","value":"1"}]},{"ids":[{"name":"_id","value":"5f3ea5a70d61525115e31e51"}]}]},"serviceName":"ext-api","status":200,"timestamp":"2020-08-20T16:50:41.801251Z"}

Examples of audit logs visualized in Splunk

Audit log files are best visualized and evaluated in a dedicated security system, such as Splunk, Greylog, Kibana, etc.

Here are some self-explanatory samples of the visualization:

14401440 15521552 15851585