Building a Custom Analytics Platform

Refer to Building a Custom Tracking and Analytics Platform: Information is Power

Why

exclusive access to our data
flexibility in data collection
able to build custom tracking and analytics website that would allow us to record and interpret information

Design Decisions

What to track (front-end)

A user's activities

user's ID (for a login user)
user's IP address
user's location (using navigator.geolocation.getCurrentPosition if not through proxy)
visited page URL
time stamp
browser (used for an unidentified user)
OS
if using a mobile device

Sample data:

{
  userAgent: window.navigator.userAgent,
  browser: getBrowserFamily(),
  browserVersion: getBrowserVersion(),
  os: getOS(),
  device: getDevice(),
  customerId: getConsumerId(),
  loggedIn: isLoggedIn(),
  browserId: getBrowserId()
  // ... Any other metadata ...
}

user's session (distinguish between an individual customer’s visits to the site)
- session ID
- referer (optional, not very useful in our case)

Sample data:

{
  id: getSessionId(),
  referral: {
    referrer: document.referrer,
    referralParams: parseReferralParams()
  }
}

An event

clicks on specific HTML elements

Sample data:

{
  event_name: "tracking-init",
  customer: // customer data from above
  session: // session data from above
  page: // page data from above
}

Store Tracking Data (back-end)

Simply make a POST request for each tracking event, and store data in the Oracle database

Database Design

LOG_NODE_ANALYTICS

Column Name	Description	Required?	Type	How to retrieve?

Column Name	Description	Required?	Type	How to retrieve?
ID	primary key	Y	String	session ID + time stamp
USERID		N	String
IP	the requester's ip address	N	String	`reqIpAddress = (req.headers['x-forwarded-for'] \|\| req.connection.remoteAddress \|\| req.socket.remoteAddress \|\| req.connection.socket.remoteAddress \|\| req.ip).split(",")[0];`
LOCATION		N	String	2 options: client-side: navigator.geolocation.getCurrentPosition server-side: https://dev.maxmind.com/geoip/geoip2/web-services/ or https://github.com/ip2location-nodejs/IP2Location
ORI_URL	the request URL, e.g. /cip/login or http://.../cip/login	Y	String	req.originalUrl
APP_HOST	the host name, e.g. cpms.sanjoseca.gov the localhost ip 127.0.0.1:<port> if proxy is used	Y	String	req.headers.host
REFERER	e.g. /auth/login	N	String	req.headers.referer
RAW_HEADERS		Y	Clob	JSON.stringify(req.headers)
TIME_STAMP		Y	Date
BROWSER	the browser which the requester is using, e.g. Chrome 60.0.3112	N	String	using https://www.npmjs.com/package/useragent agent.toAgent()
OS	the OS which the requester is using, e.g. Windows 10.0.0	N	String	agent.os.toString()
DEVICE	the device which the requester is using, e.g. Asus A100	N	String	agent.device.toString()
SESSION_ID		Y	String	req.sessionID

API

The best solution is to collect data on the server-side.

An alternative client-side solution refers to Backend Collector for Client-Side Data