Audit Log

As security is a crucial concern to Bloomreach, we consider it necessary to allow our clients to have an overview of all activities performed in the Bloomreach Engagement app. Especially when an activity is suspicious, we want you to be able to detect and investigate it. Audit logs, which capture and store records of user activity within Bloomreach Engagement, enable you to do just that.

What is audit log?

Audit logs are chronological records of user activity within the Bloomreach Engagement application. They capture audit trails that help in investigations of unauthorized access, data leak, or GDPR incidents, and they help with understanding alerts of repeated incidents and whether company-specific policies are met.

In simple terms, audit logs do that by answering the question of Who? Did What? When? And Where?

• Who? - The actor that performed an action. This can be public access, authenticated user, API key, or Bloomreach Engagement's internal process.
• Did What? - What action was performed can be identified by a unique combination of a request method and request path. Some records contain references to a specific resource and additional information within a service data.
• When? - Each action has a precise timestamp. Time within the Bloomreach Engagement platform is synchronized using Network Time Protocol (NTP).
• Where? - Actions are logged from different parts and scopes of the application. The scope is one of: an instance, account, or project. Service name refers to one of many internal services. The screen where an action was performed by users from a browser can be accessed by the referrer URL.

Enabling and accessing the audit log

To enable audit log, you have to contact your CSM, and unless previously included within the original order form, a Letter of instruction must be signed. Enabling the audit log usually takes up to 3 business days.

Access is managed by a CSM utilizing GSuite Google Groups (separate from the Access management within the Bloomreach Engagement application). Only the members of this group will be able to access the data. You will need Google Identities to use Google Groups. These identities are either Google's user accounts or service accounts.

For security reasons (as a protection from any kind of manipulation or tampering with the audit log), there is no direct access from the Application, and users will be granted read-only access to the Google Cloud Storage bucket containing exported audit log data, which means the data itself cannot be directly edited.

Different ways of accessing the audit log

Accessing with Google Cloud SDK tools: gsutil

1. You need to install gsutil first. gsutil performs all operations, including uploads and downloads, using HTTPS and transport-layer security (TLS).
2. Then authenticate with a google service account. You need a service account private key JSON file.

Authentication

$ gcloud auth activate-service-account --key-file service_account_key.json

Listing files

$ gsutil ls  gs://iid-auditlog-storage/iid/2021/10/12

Downloading files

$ gsutil -m rsync gs://iid-auditlog-storage/iid/2021/10/12 /data/local/storage/10/12

If you would like to learn more about accessing audit logs through Google Drive SDKs tools, please refer to the article.

Accessing with direct API calls

The second option is to access the audit log through direct API calls. If you want to learn more about it, read the article. You can see the example for using Google Cloud SDK python libraries down below:

from google.cloud import storage
from google.oauth2 import service_account

key_path = "service_account_key.json"

credentials = service_account.Credentials.from_service_account_file(key_path, scopes=["https://www.googleapis.com/auth/cloud-platform"],)
client = storage.Client(credentials=credentials, project=credentials.project_id,)

# list files

blobs = storage_client.list_blobs('iid-auditlog-storage')
for blob in blobs:
    print(blob.name)  

# download and print in console

bucket = client.bucket('iid-auditlog-storage')
blob = bucket.get_blob('iid/2021/02/23/20210223T080000.000Z-0.jsonl.gz')

print(blob.download_as_string())

🚧

Unique Bucket name

Your bucket name included in the code is unique. You need to replace "iid" in the above examples with your unique instance identifier instance_id (three alphanumeric lowercase symbols). Bucket naming convention is instance_id + "-auditlog-storage" followed by year/ month/ day (path formatting).

How audit log works

The audit log tracks requests sent by application users. Results are stored in a separate GCS bucket and can be downloaded and processed by an automated system.

The Bloomreach Engagement app contains a component that verifies user access for all parts of the application. Every time a logged user performs an action, this component creates an event, which is then stored in specific storage. Some events are considered to be sensitive and are therefore supplemented with additional information (e.g. how many rows have been exported/imported etc.).

Buckets are named[instance-auditlog-storage], where “instance” is the name of the instance in which the bucket is stored. The data in these buckets are organized in folders according to time in the following way: Year > Month > Day.

Files with audit log events are JSON one line. This means every event is noted on one line. Files are created every hour and each file contains new logs. Files are not updated after creation. Delayed records are written in a file with an incremental number suffix.

Data Retention

Files are available for download for at least 60 days. After this time period, they can be moved into our archive storage and are not accessible for download.

Understanding the audit log

Correlating multiple records

Some actions may generate more records on different levels of the application. Different components add component-specific information to the logs as service data. The unique request ID is generated on the first touchpoint and then propagated to subsequent events.

Scope identification

All events contain a field that defines on which layer the event was created - connecting it with a specific project, account, or instance.

• An action performed on a project level has the Scope type value “PROJECT” and_ Scope ID _value is the project token
• An action performed on an account level has the Scope type value “ACCOUNT” and Scope ID value is the account ID
• An action performed on the instance level has the Scope type value “INSTANCE” and no Scope ID value

Correlating user activity

Use session ID to correlate and analyze the flow of user activity during a session. A new unique session ID is created for each login session and discarded after logout or session expiration. Methods that do not require authentication do not have a session ID.

Audit log record schema

Log records have a predefined schema and come in JSON format. The top-level schema is fixed which makes audit logs to be easily imported into all commonly used SIEMs (e.g. Splunk, Graylog).

Fixed schema fields are reliable, can be used as indexed columns, and shall be used for SIEM alerts. Drilling down, records may contain not fixed (untyped) fields that are meant to provide additional information, are useful during investigations, and may be used for less critical SIEM alerts.

Schema: Fixed fields

Fixed fields are always present. These include:

📘

Scope-related fields (scopeType, scopeID) are present only for methods requiring authorization. Fields are not present for other methods which are public methods or require only authentication.

Schema: Optional fields

Optional fields are present only for some operations.

Schema: Service Data

ServiceData types:

• GenericServiceData
• PolicyDelta
• AnonymizationServiceData

GenericServiceData
Generic service data provide some additional information only for informational use cases. This information may be useful during investigations. The info field is not fixed and should not be used for critical use cases.

{"@type":"auditlog.GenericServiceData","info":"{\"action_types\":[\"ab-split\",\"batch-webhook-action\",\"condition\",\"limit\",\"now-trigger\",\"on-event-trigger\",\"planned-trigger\",\"repeated-trigger\",\"send-email-action\",\"set-property-action\",\"wait-action\"],\"webhooks\":{\"endpoints\":[\"https://aud.app.exponea.dev/\"]},\"webhook_has_general_consent\":false,\"email_has_general_consent\":false}","versionID":"5f3b83344785e844fa76b0fd"}

PolicyChange

{"@type":"iam.PolicyUpdate","id":"5f3cdcc931b9028e985237c1","policyChanges":[{"member":"[email protected]","roleId":"PII","type":"ADD"}],"updateTime":"2020-08-19T08:03:21.529134238Z","userId":"[email protected]"}

{"@type":"iam.PolicyUpdate","id":"5f3633bc699b028f7121560b","policyChanges":[{"member":"[email protected]","roleId":"LegacyProjectAdmin","type":"REMOVE"},{"member":"[email protected]","roleId":"PII","type":"REMOVE"},{"member":"[email protected]","roleId":"LegacyAccountAdmin","type":"REMOVE"}],"updateTime":"2020-08-14T06:48:28.870266563Z","userId":"[email protected]"}

{"@type":"iam.PolicyUpdate","id":"5f3cf286a011be198cde4905","policyChanges":[{"member":"[email protected]","newExpireTime":"0001-01-01T00:00:00Z","oldExpireTime":"2020-08-22T22:00:00Z","roleId":"LegacyProjectAdmin","type":"UPDATE"}],"updateTime":"2020-08-19T09:36:06.174072194Z","userId":"[email protected]"}

AnonymizationServiceData

{"authenticationInfo":{"identity":"44zrbjouu8jdio64zv6hli4ax0hklk0lkbn5gn4bd151epdffj2mhwj1lteta4tr","type":"BASIC_AUTH"},"authorizationInfo":{"allowed":true},"instanceName":"aud","logName":"audit.v1.Log","metadata":{"clientIP":"34.76.115.106","host":"aud.api.exponea.dev","userAgent":"PostmanRuntime/7.26.2"},"request":{"@type":"http","method":"POST","path":"/data/v2/projects/a25641b8-dd39-11ea-b199-ae02a0152881/customers/anonymize_bulk"},"serviceData":{"@type":"auditlog.AnonymizationServiceData","requests":[{"ids":[{"name":"registered","value":"1"}]},{"ids":[{"name":"_id","value":"5f3ea5a70d61525115e31e51"}]}]},"serviceName":"ext-api","status":200,"timestamp":"2020-08-20T16:50:41.801251Z"}

Examples of audit logs visualized in Splunk

Audit log files are best visualized and evaluated in a dedicated security system, such as Splunk, Graylog, Kibana, etc.

Here are some self-explanatory samples of the visualization: