AI-Based Behavioural Risk Modelling and Predictive Threat Anticipation

Produced by: Ave Bilişim – Research & Data Analytics Unit Reference: ATR-2024-07-04 Type: Internal Technical Report Completion Date: July 2024 Confidentiality: Internal – Abstract may be public

1. Abstract

This internal technical report documents a multi-phase research programme conducted during 2023–2024 by Ave Bilişim’s Research & Data Analytics Unit. The aim was to design, implement, and evaluate an AI-based framework for behavioural risk modelling and predictive threat anticipation in large, heterogeneous data ecosystems. The proposed pipeline fuses multi-source streams, engineers relational and temporal features, and applies a calibrated ensemble of graph representation learning, sequence models, and probabilistic inference.

Extensive simulations on synthetic yet statistically realistic datasets show that the framework improves precision-recall balance, reduces false positives, accelerates time-to-alert, and preserves analyst interpretability compared with baseline rule-based methods. The approach is domain-agnostic and portable to safety-critical and operational-risk contexts (e.g., infrastructure reliability, logistics integrity, industrial operations). This report was completed and archived in July 2024 as part of Ave Bilişim’s independent research portfolio.

2. Executive Summary

Organisations increasingly rely on data generated by communications platforms, logistics systems, transaction back-ends, and sensor networks. While each source is informative, weak signals of risk are often distributed across sources and evolve over time. Traditional monitoring focused on isolated anomalies misses these cross-modal, sequential patterns. Ave Bilişim set out to build a framework that:

• Learns behavioural motifs spanning multiple systems and time scales.

• Produces calibrated risk scores that reflect uncertainty.

• Provides explanations suitable for analyst triage and audit.

• Scales to high event rates with robust governance and reproducibility.

We built a five-layer architecture (ingestion → processing → modelling → scoring → visualisation) and validated an ensemble combining heterogeneous GNNs, autoencoders, LSTMs, and Bayesian calibration. The system achieved higher F1, lower false-positive rates, and earlier alerts than baselines, while maintaining traceable decision paths. The design emphasises methodological portability and responsible AI practices.

3. Background and Rationale

3.1 Operational context

Data volume, variety, and velocity have outpaced conventional analytics. Events that matter rarely appear as single outliers; they emerge from interactions across entities (actors, assets, processes) and temporal dynamics (burstiness, synchrony, lead-lag effects). A siloed approach creates blind spots.

3.2 Problem statement

Rule-based detectors require constant maintenance, degrade under concept drift, and provide little rationale beyond threshold breaches. Machine-learning classifiers improve accuracy but often lack interpretability and calibration, limiting trust in operations.

3.3 Research questions

1. Can multi-source fusion and relational features improve early-warning power?

2. Do graph and temporal models remain robust under drift compared with static baselines?

3. How can we deliver explanations and maintain probability calibration acceptable to human analysts?

4. What MLOps and governance mechanisms ensure reliable lifecycle management?

3.4 Objectives

• Design a modular, reproducible pipeline.

• Evaluate complementary model families and their ensemble.

• Quantify performance, latency, and calibration quality.

• Document governance, audit, and ethical safeguards.

• Demonstrate portability to varied operational-risk contexts.

4. Scope and Definitions

• Entity: an actor, asset, or process node (e.g., device, vehicle, account).

• Event: a time-stamped action (message, movement, transaction, sensor reading).

• Behavioural motif: a recurring sub-pattern of interactions over time.

• Risk score: a calibrated probability proxy for ranking and triage.

• Early warning: an alert raised prior to threshold breach or manifest incident.

This study evaluates methodology, not a domain-specific deployment; no personal data were used.

5. Related Work and Conceptual Foundations

5.1 Behavioural risk modelling

Risk is treated as an emergent property of networked interactions and timelines, not merely of individual outliers. By learning trajectories and motifs—e.g., densification of contacts plus route detours—models anticipate elevated risk before traditional thresholds trip.

5.2 Graph representation learning

Graph Neural Networks propagate messages along edges so each node’s embedding encodes its neighbourhood and higher-order structure (community, centrality). For heterogeneous graphs, relation types receive separate parameters before aggregation, enabling context-aware anomaly scoring.

5.3 Temporal modelling

Recurrent architectures (LSTM/GRU) and temporal convolutions capture sequential dependencies such as burstiness, seasonality, and cross-modal synchrony. Forecast residuals become deviation signals.

5.4 Probabilistic inference and calibration

Bayesian Belief Networks (BBNs) encode conditional dependencies (e.g., timing irregularity implies elevated risk only when relational density is high). Post-hoc temperature scaling aligns predicted probabilities with observed frequencies, improving threshold reliability.

5.5 Human-centred AI

Operational use demands explainability. We employ feature attributions (SHAP-style), nearest-prototype retrieval (“this resembles motif M”), compact textual rationales, and counterfactuals (“if synchrony index were 30% lower, risk < threshold”) to support analyst judgment.

6. Data, Assumptions, and Pre-processing

6.1 Sources and volumes

Synthetic datasets were generated to mirror enterprise telemetry statistics:

After cleansing (outlier clipping, schema harmonisation, time consistency), ~26.4M records remained.

6.2 Canonical schema and integrity

All sources were mapped to a canonical schema (entity_id, counterparty, timestamp, geohash, modality, attributes). Integrity checks enforced referential completeness, monotonic time, and consistent time zones.

6.3 Privacy and safety

Identifiers were tokenised; quasi-identifiers were generalised (reduced geohash precision). Differential noise was added to aggregated counts to prevent re-identification while preserving distributional properties needed for modelling.

6.4 Data quality management

• Completeness: missingness thresholds per field; imputation rules documented.

• Consistency: cross-source timestamp reconciliation; duplicate suppression.

• Traceability: data lineage recorded from ingestion to feature store.

7. Feature Engineering

7.1 Temporal features

• Inter-event intervals; burstiness; circadian deviation.

• Rolling-window counts/ratios (5m/1h/8h/24h).

• Change-point flags (CUSUM on activity rates).

7.2 Spatial/trajectory features

• Average displacement; detour index; route entropy.

• Co-location density with peers (k-NN in space-time).

• Dwell-time distributions at stay-points.

7.3 Graph/relational features

• Degree, weighted degree; betweenness & eigenvector centrality.

• Clustering coefficients; community membership & boundary scores.

• Edge-type asymmetries and role-based fingerprints.

7.4 Cross-modal composites

• Coupling index: alignment between movement and communications.

• Synchrony score: multi-entity phase alignment within sliding windows.

• Rare-motif flags: frequency of infrequent sub-graphs (mined via gSpan).

7.5 Selection and stability

SHAP-based importance ranking on validation folds; features with unstable ranks (>25% variance across folds) removed, yielding ~120 robust features.

8. Modelling Approach

8.1 Model families

1. Heterogeneous GNN – 3 convolution layers (64–64–32), relational attention, residual connections, layer norm, dropout 0.2.

2. Autoencoder – 5-layer dense (512→64→16→64→512), sparsity penalty; anomaly via reconstruction error.

3. BBN – ~45 nodes; structure from heuristics + score-based refinement; parameters via EM.

4. LSTM – 2 layers, 128 hidden units; horizon h=12 for short-term behaviour forecasting.

8.2 Ensemble and calibration

Outputs feed a logistic meta-learner with learned weights; temperature scaling on validation lowers ECE (Expected Calibration Error), stabilising thresholds under drift.

8.3 Training regime

• 80/20 time-ordered split; fivefold time-series CV.

• Class imbalance via focal loss (γ=2) + minority up-weighting.

• Mixed-precision training on 4×GPU; deterministic ops and fixed seeds.

8.4 Robustness tests

• k-bootstrapping for confidence intervals.

• Synthetic covariate shift (α∈[0.05, 0.2]).

• Ablation removing feature families to measure marginal gain.

9. Evaluation Protocol

9.1 Metrics

We report Accuracy, Precision, Recall, F1, AUC, FPR, Latency, ECE; and operational measures time-to-alert and analyst triage time.

9.2 Baselines

• Logistic regression (engineered features).

• Random forest (500 trees).

• Rule-based detector (expert thresholds).

9.3 Holdout strategy

Four time blocks for training and one for validation, rotating across folds; final performance reported on a chronological holdout (last 10% of timeline).

10. Results

10.1 Aggregate performance

The ensemble improves F1 by +37 points vs the rule-based baseline and halves FPR. Calibration (ECE) improves notably, enabling safer thresholds.

10.2 Ablation insights (F1 on holdout)

• Without GNN: −0.07; without LSTM: −0.04; without BBN: −0.05; without Autoencoder: −0.03; without graph features: −0.09. Conclusion: relational structure contributes the largest marginal gain.

10.3 Error analysis

False positives cluster in low-density regions (sparse interactions) and new entity cold-starts; mitigated by BBN priors and prototype-similarity fallbacks. False negatives relate to short spikes without context; mitigated by shorter windows and adaptive thresholds.

10.4 Robustness under drift

Under synthetic shift α=0.10, F1 falls 0.81→0.77; refreshing calibration every 24h recovers F1 to 0.79. Drift monitoring plus periodic rescaling is sufficient at moderate shift levels.

10.5 Time-to-alert and analyst workload

Median time from precursor onset to alert: 6 minutes (IQR 4–10). In pilot analyst sessions, triage time fell ~40% thanks to explanations and prototype retrieval.

11. Scalability and Systems Engineering

11.1 Architecture and throughput

Microservices isolate ingestion (Kafka), processing (Flink/Beam), feature store (point-in-time), modelling (TF/PyTorch/Neo4j), scoring (gRPC), and dashboards (Grafana/Plotly). Stress tests: 150k events/sec with <9 s end-to-end latency and 99.9% availability over 8-hour runs.

11.2 Backpressure and resilience

Bounded queues, priority channels, and adaptive sampling maintain stability under bursts. Canary scoring precedes full rollout; failures auto-rollback via versioned registries.

11.3 Observability and security

Prometheus metrics, OpenTelemetry traces, and structured logs enable deep audits. Images are scanned; secrets managed in KMS; mTLS and least-privilege IAM enforced.

12. Interpretability and Analyst Workflow

12.1 Explanations

Each alert ships with SHAP summaries, top contributing features, nearest historical prototypes, and a counterfactual suggestion.

12.2 Triage loop

Alerts enter a ranked queue. Analyst decisions (confirm / dismiss / watch) are written to a feedback store; periodic re-training improves thresholds and class balance.

12.3 Documentation

Model cards and data sheets accompany every release: scope, assumptions, limitations, metrics, failure modes, and slice analyses.

13. Governance, Ethics, and Risk Management

13.1 Principles and oversight

The project followed European good practices on transparency, accountability, fairness, and human oversight. An internal committee reviewed milestones; minutes and approvals are archived.

13.2 Data protection

Only synthetic/anonymised data were used. Tokenisation, aggregation, and controlled noise reduced re-identification risk. Access to datasets followed least-privilege.

13.3 Fairness and bias

Simulated sub-populations tested parity of error rates; disparities >5% triggered feature review and recalibration. Residual risks are documented with mitigation plans.

13.4 Auditability

Experiments are reproducible via versioned code, data snapshots, seeds, and environment manifests. Each alert carries a decision trace (features, model id, commit hash).

14. Applications and Transferability

The pipeline is readily adaptable to:

• Logistics & supply-chain integrity (route irregularities, anomalous handoffs),

• Industrial safety & predictive maintenance (sensor precursors to equipment failure),

• Financial compliance & fraud (coordinated behavioural patterns),

• Public safety & operations (network dynamics and early signals).

Adaptation requires schema mapping, minor feature re-engineering, and threshold re-calibration—not a full redesign.

15. Limitations

• Synthetic realism: Field noise and rare edge cases may reduce precision; plan controlled pilots with anonymised operational feeds.

• Cold-start entities: Limited history reduces reliability; use prior-based BBN nodes and prototype similarity.

• Concept drift: Gradual shifts harm calibration; schedule drift detection and temperature rescaling.

• Human workload: Rich explanations can still overwhelm; apply progressive disclosure and tuned alert budgets.

Early motif mining with k=3 raised FP clusters in low-density regions; we reverted to gSpan with min support α=0.02.

16. Future Work and Roadmap

1. Field validation: controlled integrations with anonymised, real-world feeds.

2. Domain adaptation: transfer/meta-learning for rapid re-tuning.

3. Edge inference: lightweight models near sources for latency-sensitive scenarios.

4. Active learning: selective labelling to maximise information gain.

5. Public dissemination: a short abstract on Ave Bilişim’s website as part of knowledge-sharing efforts.

17. Conclusions

We present a reproducible, explainable, and scalable framework for behavioural risk modelling and predictive threat anticipation. The ensemble of graph, sequence, and probabilistic models outperforms traditional baselines, remains robust under moderate drift, and integrates with analyst workflows through meaningful explanations and strong governance.

The work was completed and archived in July 2024 as part of Ave Bilişim’s independent research portfolio. It constitutes prior evidence of methodological capability in large-scale, trustworthy AI for safety-critical and operational-risk domains and can serve as a template for future evaluations and deployments.

Appendices

Appendix A – Extended Metric Definitions

• Accuracy: Correct classifications / total observations; insensitive to class imbalance, used only for coarse comparison.

• Precision: True positives / predicted positives; reflects alert quality.

• Recall (Sensitivity): True positives / actual positives; reflects missed-risk tolerance.

• F1 Score: Harmonic mean of precision and recall; balances false-positive/negative trade-offs.

• AUC: Probability that a random positive ranks above a random negative; threshold-free performance.

• FPR: False positives / actual negatives; operationally tied to analyst workload.

• ECE: Expected Calibration Error; absolute difference between predicted and empirical probabilities across bins.

• Latency: End-to-end time from event ingestion to risk score delivery.

• Time-to-alert: Duration from precursor onset to first alert; key for anticipatory value.

• Analyst triage time: Median human review time per alert; proxy for practical usability.

Appendix B – Model Parameter Tables (Illustrative)

B.1 Heterogeneous GNN

• Layers: 3 conv (64-64-32), relational attention=On, residual=On, dropout=0.2

• Optimiser: Adam, lr=1e-3, weight decay=1e-5

• Epochs: 60 (early stop patience=8 on AUC)

B.2 Autoencoder

• Encoder: 512-256-64-16; Decoder mirrored; sparsity λ=1e-4; dropout=0.2

• Criterion: MSE + sparsity; threshold via Youden’s J on validation

B.3 LSTM

• Layers: 2; hidden=128; horizon h=12; teacher forcing=0.7

• Loss: MAE on forecast residuals; anomaly if residual z-score>τ

B.4 BBN

• Nodes: ~45; structure init via heuristics; EM for parameters; calibration via Platt/temperature scaling

Appendix C – Data Schema and Quality Rules

• Core fields: entity_id, counterpart_id, timestamp_utc, geohash, modality, attrs_json.

• Quality rules: timestamp monotonicity; null ratio <5% per critical field; deduplicate by (entity_id, timestamp, modality).

• Lineage: dataset_id, ingestion_job_id, transform_hash, feature_store_version.

Appendix D – Synthetic Data Generator (Specification)

• Communication process: inhomogeneous Poisson with circadian baseline + burst processes.

• Mobility: Levy-flight with dwell-time heavy tails; route perturbations add detours.

• Transactions: Hawkes process (self-exciting) to mimic rapid sequences.

• Sensors: finite-state machine with stochastic transitions for operational states.

• Coupling: cross-modal synchrony controlled by parameter ρ∈[0,1]; motifs injected at low frequency for controlled positives.

Appendix E – Governance Artefacts (Templates)

• Model Card: scope, assumptions, limitations, metrics, slices.

• Data Sheet: provenance, license, collection protocol, QA, known issues.

• Risk Log: identified risks, likelihood, impact, mitigations, status.

• Approval Record: reviewer role, decision, date, version hashes.

Appendix F – Risk Register (Excerpt)

Appendix G – SOP: Model Change & Incident Handling (Summary)

1. Propose change (model or threshold) → open change request.

2. Offline evaluation → benchmarks, fairness slices, calibration.

3. Risk review → governance committee sign-off.

4. Canary scoring → shadow mode; compare KPIs.

5. Promote or rollback based on acceptance gates.

6. Incident response → freeze, revert, root-cause analysis, corrective action.

Appendix H – Glossary

• Calibration: alignment between predicted risk and observed frequency.

• Concept drift: change in data distribution over time affecting model performance.

• Counterfactual explanation: minimal input change that flips a prediction.

• Motif: recurring sub-graph or temporal pattern associated with elevated risk.

• PSI (Population Stability Index): drift measure between distributions.