Audit Log
As security is a crucial concern to Bloomreach, we consider it necessary to allow our clients to have an overview of all activities performed in the Bloomreach Engagement app. Especially when an activity is suspicious, we want you to be able to detect and investigate it. Audit logs, which capture and store records of user activity within Bloomreach Engagement, enable you to do just that.
What is audit log?
Audit logs are chronological records of user activity within the Bloomreach Engagement application. They capture audit trails that help in investigations of unauthorized access, data leak, or GDPR incidents, and they help with understanding alerts of repeated incidents and whether company-specific policies are met.
In simple terms, audit logs do that by answering the question of Who? Did What? When? And Where?
- Who? - The actor that performed an action. This can be public access, authenticated user, API key, or Bloomreach Engagement's internal process.
- Did What? - What action was performed can be identified by a unique combination of a request method and request path. Some records contain references to a specific resource and additional information within a service data.
- When? - Each action has a precise timestamp. Time within the Bloomreach Engagement platform is synchronized using Network Time Protocol (NTP).
- Where? - Actions are logged from different parts and scopes of the application. The scope is one of: an instance, account, or project. Service name refers to one of many internal services. The screen where an action was performed by users from a browser can be accessed by the referrer URL.
Enabling and accessing the audit log
To enable audit log, you have to contact your CSM, and unless previously included within the original order form, a Letter of instruction must be signed. Enabling the audit log usually takes up to 3 business days.
Access is managed by a CSM utilizing GSuite Google Groups (separate from the Access management within the Bloomreach Engagement application). Only the members of this group will be able to access the data. You will need Google Identities to use Google Groups. These identities are either Google's user accounts or service accounts.
For security reasons (as a protection from any kind of manipulation or tampering with the audit log), there is no direct access from the Application, and users will be granted read-only access to the Google Cloud Storage bucket containing exported audit log data, which means the data itself cannot be directly edited.
Different ways of accessing the audit log
Accessing with Google Cloud SDK tools: gsutil
- You need to install gsutil first. gsutil performs all operations, including uploads and downloads, using HTTPS and transport-layer security (TLS).
- Then authenticate with a google service account. You need a service account private key JSON file.
Authentication
$ gcloud auth activate-service-account --key-file service_account_key.json
Listing files
$ gsutil ls gs://iid-auditlog-storage/iid/2021/10/12
Downloading files
$ gsutil -m rsync gs://iid-auditlog-storage/iid/2021/10/12 /data/local/storage/10/12
If you would like to learn more about accessing audit logs through Google Drive SDKs tools, please refer to the article.
Accessing with direct API calls
The second option is to access the audit log through direct API calls. If you want to learn more about it, read the article. You can see the example for using Google Cloud SDK python libraries down below:
from google.cloud import storage
from google.oauth2 import service_account
key_path = "service_account_key.json"
credentials = service_account.Credentials.from_service_account_file(key_path, scopes=["https://www.googleapis.com/auth/cloud-platform"],)
client = storage.Client(credentials=credentials, project=credentials.project_id,)
# list files
blobs = storage_client.list_blobs('iid-auditlog-storage')
for blob in blobs:
print(blob.name)
# download and print in console
bucket = client.bucket('iid-auditlog-storage')
blob = bucket.get_blob('iid/2021/02/23/20210223T080000.000Z-0.jsonl.gz')
print(blob.download_as_string())
Unique Bucket name
Your bucket name included in the code is unique. You need to replace "iid" in the above examples with your unique instance identifier
instance_id
(three alphanumeric lowercase symbols). Bucket naming convention isinstance_id + "-auditlog-storage"
followed byyear
/month
/day
(path formatting).
How audit log works
The audit log tracks requests sent by application users. Results are stored in a separate GCS bucket and can be downloaded and processed by an automated system.
The Bloomreach Engagement app contains a component that verifies user access for all parts of the application. Every time a logged user performs an action, this component creates an event, which is then stored in specific storage. Some events are considered to be sensitive and are therefore supplemented with additional information (e.g. how many rows have been exported/imported etc.).
Buckets are named[instance-auditlog-storage]
, where “instance” is the name of the instance in which the bucket is stored. The data in these buckets are organized in folders according to time in the following way: Year > Month > Day.
Files with audit log events are JSON one line. This means every event is noted on one line. Files are created every hour and each file contains new logs. Files are not updated after creation. Delayed records are written in a file with an incremental number suffix.
Files are available for download for at least 60 days. After this time period, they can be moved into our archive storage and are not accessible for download.
Understanding the audit log
Correlating multiple records
Some actions may generate more records on different levels of the application. Different components add component-specific information to the logs as service data. The unique request ID is generated on the first touchpoint and then propagated to subsequent events.
Scope identification
All events contain a field that defines on which layer the event was created - connecting it with a specific project, account, or instance.
- An action performed on a project level has the Scope type value “PROJECT” and_ Scope ID _value is the project token
- An action performed on an account level has the Scope type value “ACCOUNT” and Scope ID value is the account ID
- An action performed on the instance level has the Scope type value “INSTANCE” and no Scope ID value
Correlating user activity
Use session ID to correlate and analyze the flow of user activity during a session. A new unique session ID is created for each login session and discarded after logout or session expiration. Methods that do not require authentication do not have a session ID.
Audit log record schema
Log records have a predefined schema and come in JSON format. The top-level schema is fixed which makes audit logs to be easily imported into all commonly used SIEMs (e.g. Splunk, Graylog).
Fixed schema fields are reliable, can be used as indexed columns, and shall be used for SIEM alerts. Drilling down, records may contain not fixed (untyped) fields that are meant to provide additional information, are useful during investigations, and may be used for less critical SIEM alerts.
Schema: Fixed fields
Fixed fields are always present. These include:
Field name | Description |
---|---|
timestamp | The timestamp of the operation, ISO 8601 datetime in UTC, e.g. 2020-07-09T11:30:57.239442245Z |
request | method or path |
status | HTTP status code, e.g. 403 for Forbidden actions |
serviceName | The internal name of the service within Bloomreach Engagement architecture |
scopeType | enum: INSTANCE, ACCOUNT, PROJECT |
scopeID | A unique project or account identifier |
requestID | A unique ID, useful for correlating individual records connected with single user action |
Scope-related fields (scopeType, scopeID) are present only for methods requiring authorization. Fields are not present for other methods which are public methods or require only authentication.
Schema: Optional fields
Optional fields are present only for some operations.
Field name | Description |
---|---|
authenticationInfo | identity : operations performed by an authenticated actor, e.g. read/update a user profile, two-factor activation, logouttype : e.g. USER, SYSTEM_SERVICE_ACCOUNT, BASIC_AUTH |
authorizationInfo | allowed : whether the action was allowed by an authorization checkpermission : operations that require permission, e.g. customer data read |
metadata | clientIP : a remote IP address from where the request was calledhost : hostname (domain) for external requestsreferrer : browser reported document referrer, useful to find the origin screensession : session ID for requests within an authenticated user sessionuserAgent : browser reported User-Agent |
resource_id | Every operation on a specific resource with internal ID, e.g. customer, scenario, banner, experiment. Useful for, e.g. filtering, alerts |
Schema: Service Data
ServiceData types:
- GenericServiceData
- PolicyDelta
- AnonymizationServiceData
GenericServiceData
Generic service data provide some additional information only for informational use cases. This information may be useful during investigations. The info field is not fixed and should not be used for critical use cases.
Field name | Description |
---|---|
@type | auditlog.GenericServiceData |
info (untyped) | JSON encoded string, informational field, and schema is not fixed and it's subject to change |
version_id | Every operation on a specific resource with enabled versioning history, e.g. reports create/update, scenario create/update/start |
{"@type":"auditlog.GenericServiceData","info":"{\"action_types\":[\"ab-split\",\"batch-webhook-action\",\"condition\",\"limit\",\"now-trigger\",\"on-event-trigger\",\"planned-trigger\",\"repeated-trigger\",\"send-email-action\",\"set-property-action\",\"wait-action\"],\"webhooks\":{\"endpoints\":[\"https://aud.app.exponea.dev/\"]},\"webhook_has_general_consent\":false,\"email_has_general_consent\":false}","versionID":"5f3b83344785e844fa76b0fd"}
PolicyChange
Field name | Description |
---|---|
@type | iam.PolicyUpdate |
policyChanges | A list of policy changes.member : the user ID of the affected userroleId : role ID that was changed; e.g. "LegacyProjectAdmin"type : change type, enum:- ADD: role has been granted - UPDATE: role has been updated - REMOVE: role has been revoked oldExpireTIme : (optional) previously set role expiration timestampnewExpireTime : (optional) newly set role expiration timestamp |
updateTime | The timestamp of the operation |
userID | User ID of the subject performing the operation |
{"@type":"iam.PolicyUpdate","id":"5f3cdcc931b9028e985237c1","policyChanges":[{"member":"[email protected]","roleId":"PII","type":"ADD"}],"updateTime":"2020-08-19T08:03:21.529134238Z","userId":"[email protected]"}
{"@type":"iam.PolicyUpdate","id":"5f3633bc699b028f7121560b","policyChanges":[{"member":"[email protected]","roleId":"LegacyProjectAdmin","type":"REMOVE"},{"member":"[email protected]","roleId":"PII","type":"REMOVE"},{"member":"[email protected]","roleId":"LegacyAccountAdmin","type":"REMOVE"}],"updateTime":"2020-08-14T06:48:28.870266563Z","userId":"[email protected]"}
{"@type":"iam.PolicyUpdate","id":"5f3cf286a011be198cde4905","policyChanges":[{"member":"[email protected]","newExpireTime":"0001-01-01T00:00:00Z","oldExpireTime":"2020-08-22T22:00:00Z","roleId":"LegacyProjectAdmin","type":"UPDATE"}],"updateTime":"2020-08-19T09:36:06.174072194Z","userId":"[email protected]"}
AnonymizationServiceData
Field name | Description |
---|---|
@type | auditlog.AnonymizationServiceData |
requests | A list of the requested IDs.ids : A list of the external customer IDs.- name: ID name_, e.g. "registered" - value: ID value, e.g. "[email protected]"_ID name = "_id" for Bloomreach Engagement's internal customer ID |
{"authenticationInfo":{"identity":"44zrbjouu8jdio64zv6hli4ax0hklk0lkbn5gn4bd151epdffj2mhwj1lteta4tr","type":"BASIC_AUTH"},"authorizationInfo":{"allowed":true},"instanceName":"aud","logName":"audit.v1.Log","metadata":{"clientIP":"34.76.115.106","host":"aud.api.exponea.dev","userAgent":"PostmanRuntime/7.26.2"},"request":{"@type":"http","method":"POST","path":"/data/v2/projects/a25641b8-dd39-11ea-b199-ae02a0152881/customers/anonymize_bulk"},"serviceData":{"@type":"auditlog.AnonymizationServiceData","requests":[{"ids":[{"name":"registered","value":"1"}]},{"ids":[{"name":"_id","value":"5f3ea5a70d61525115e31e51"}]}]},"serviceName":"ext-api","status":200,"timestamp":"2020-08-20T16:50:41.801251Z"}
Examples of audit logs visualized in Splunk
Audit log files are best visualized and evaluated in a dedicated security system, such as Splunk, Graylog, Kibana, etc.
Here are some self-explanatory samples of the visualization:
Updated about 13 hours ago